2023-11-28T16:51:55.973 INFO:root:teuthology version: 0.0.1.dev209+g5ae09a5 2023-11-28T16:51:55.974 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} --owner scheduled_vshankar@teuthology --name vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi --verbose -- /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360/orig.config.yaml 2023-11-28T16:51:56.020 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T16:51:56.108 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360 branch: wip-vshankar-testing-20231127.102654 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} email: vshankar@redhat.com first_in_suite: false job_id: '7469360' kernel: flavor: default kdb: true sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20231127.102654 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - client session with non-allowable root sha1: 47c1c547372ed5bff7984647dc200bac0e019caf ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 47c1c547372ed5bff7984647dc200bac0e019caf thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20231127.102654 sha1: 47c1c547372ed5bff7984647dc200bac0e019caf owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1098 sha1: 47c1c547372ed5bff7984647dc200bac0e019caf sleep_before_teardown: 0 subset: 854/1024 suite: fs suite_branch: wip-vshankar-testing-20231127.102654 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 47c1c547372ed5bff7984647dc200bac0e019caf syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAd/T2uVnzOHp9OpssfoR7Mv6f0LDVHPMou5adf1YT1rkfXwdtt2VxAB+RNPeCVBdlQuqDkrd4w7MPavNdXVc+w= smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHTMAA4YyLkkHqT3TQSZosSRhTfgBHoAWQmyxP5/lAyfF2GZ1OX97dufJBP9jzaMDtkuTglviynajI16ajnPc+k= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_sessionmap teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 5ae09a5cc7064ca2bf994df19f3006b79794789a timestamp: 2023-11-28_06:23:42 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-28T16:51:56.111 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa; will attempt to use it 2023-11-28T16:51:56.112 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks 2023-11-28T16:51:56.112 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-28T16:51:56.113 INFO:teuthology.task.internal:Checking packages... 2023-11-28T16:51:56.138 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '47c1c547372ed5bff7984647dc200bac0e019caf' 2023-11-28T16:51:56.138 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-28T16:51:56.138 INFO:teuthology.packaging:ref: None 2023-11-28T16:51:56.138 INFO:teuthology.packaging:tag: None 2023-11-28T16:51:56.138 INFO:teuthology.packaging:branch: wip-vshankar-testing-20231127.102654 2023-11-28T16:51:56.138 INFO:teuthology.packaging:sha1: 47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T16:51:56.139 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20231127.102654 2023-11-28T16:51:56.364 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7531.g47c1c547 2023-11-28T16:51:56.364 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-28T16:51:56.371 INFO:teuthology.task.internal:no buildpackages task found 2023-11-28T16:51:56.371 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-28T16:51:56.390 INFO:teuthology.task.internal:Saving configuration 2023-11-28T16:51:56.407 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-28T16:51:56.413 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T16:51:56.435 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 16:46:47.808555', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAd/T2uVnzOHp9OpssfoR7Mv6f0LDVHPMou5adf1YT1rkfXwdtt2VxAB+RNPeCVBdlQuqDkrd4w7MPavNdXVc+w='} 2023-11-28T16:51:56.454 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 16:46:47.806967', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHTMAA4YyLkkHqT3TQSZosSRhTfgBHoAWQmyxP5/lAyfF2GZ1OX97dufJBP9jzaMDtkuTglviynajI16ajnPc+k='} 2023-11-28T16:51:56.454 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-28T16:51:56.461 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-11-28T16:51:56.461 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-11-28T16:51:56.461 INFO:teuthology.run_tasks:Running task console_log... 2023-11-28T16:51:56.532 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7e6d96da60>, signals=[15]) 2023-11-28T16:51:56.533 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-28T16:51:56.540 INFO:teuthology.task.internal:Opening connections... 2023-11-28T16:51:56.540 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T16:51:56.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T16:51:56.667 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T16:51:56.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T16:51:56.754 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-28T16:51:56.761 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2023-11-28T16:51:56.779 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2023-11-28T16:51:56.780 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2023-11-28T16:51:56.835 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2023-11-28T16:51:56.836 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2023-11-28T16:51:56.836 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2023-11-28T16:51:56.836 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2023-11-28T16:51:56.836 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2023-11-28T16:51:56.836 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2023-11-28T16:51:56.836 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-28T16:51:56.836 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2023-11-28T16:51:56.836 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2023-11-28T16:51:56.836 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-28T16:51:56.836 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2023-11-28T16:51:56.836 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-28T16:51:56.837 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-28T16:51:56.837 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-28T16:51:56.837 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2023-11-28T16:51:56.857 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2023-11-28T16:51:56.874 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2023-11-28T16:51:56.874 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2023-11-28T16:51:56.931 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Stream" 2023-11-28T16:51:56.931 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="9" 2023-11-28T16:51:56.931 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2023-11-28T16:51:56.931 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2023-11-28T16:51:56.931 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="9" 2023-11-28T16:51:56.932 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el9" 2023-11-28T16:51:56.932 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-28T16:51:56.932 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2023-11-28T16:51:56.932 INFO:teuthology.orchestra.run.smithi156.stdout:LOGO="fedora-logo-icon" 2023-11-28T16:51:56.932 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-28T16:51:56.932 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://centos.org/" 2023-11-28T16:51:56.932 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-28T16:51:56.933 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-28T16:51:56.933 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-28T16:51:56.933 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2023-11-28T16:51:56.948 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-28T16:51:56.958 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-28T16:51:56.964 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-28T16:51:56.965 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2023-11-28T16:51:56.967 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2023-11-28T16:51:56.988 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-28T16:51:56.997 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-28T16:51:56.998 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2023-11-28T16:51:57.023 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2023-11-28T16:51:57.126 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-28T16:51:57.177 INFO:teuthology.run_tasks:Running task kernel... 2023-11-28T16:51:57.194 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'} 2023-11-28T16:51:57.195 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}}, timeout 300 2023-11-28T16:51:57.195 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T16:51:57.196 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T16:51:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T16:51:57.212 DEBUG:teuthology.task.kernel:sha1 for mon.a is 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T16:51:57.212 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "8247bbb5b470f7e2a1f3eefcc16e24cf1338f606"... 2023-11-28T16:51:57.213 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2023-11-28T16:51:57.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T16:51:57.214 DEBUG:teuthology.task.kernel:sha1 for mon.b is 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T16:51:57.214 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "8247bbb5b470f7e2a1f3eefcc16e24cf1338f606"... 2023-11-28T16:51:57.215 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2023-11-28T16:51:57.269 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-361.el9.x86_64 2023-11-28T16:51:57.270 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T16:51:57.270 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-11-28T16:51:57.271 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T16:51:57.275 INFO:teuthology.orchestra.run.smithi156.stdout:5.14.0-361.el9.x86_64 2023-11-28T16:51:57.276 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T16:51:57.276 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-11-28T16:51:57.276 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T16:51:57.427 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum upgrade -y linux-firmware 2023-11-28T16:51:57.497 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum upgrade -y linux-firmware 2023-11-28T16:51:59.504 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-11-28T16:51:59.629 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-11-28T16:52:00.862 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2023-11-28T16:52:01.230 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2023-11-28T16:52:02.622 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 7.9 MB 00:01 2023-11-28T16:52:03.204 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 7.9 MB 00:01 2023-11-28T16:52:06.304 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 18 MB 00:01 2023-11-28T16:52:07.870 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 18 MB 00:04 2023-11-28T16:52:10.988 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 5.8 MB 00:01 2023-11-28T16:52:11.888 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 5.8 MB 00:00 2023-11-28T16:52:13.197 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 15 kB 00:01 2023-11-28T16:52:13.673 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2023-11-28T16:52:15.093 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2023-11-28T16:52:15.209 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2023-11-28T16:52:19.832 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-11-28T16:52:20.031 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2023-11-28T16:52:22.241 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-28T16:52:22.252 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-28T16:52:22.253 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-11-28T16:52:22.253 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-28T16:52:22.253 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2023-11-28T16:52:22.253 INFO:teuthology.orchestra.run.smithi116.stdout: linux-firmware noarch 20231030-141.el9 CentOS-BaseOS 328 M 2023-11-28T16:52:22.253 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T16:52:22.253 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-28T16:52:22.253 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2023-11-28T16:52:22.253 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 1 Package 2023-11-28T16:52:22.253 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T16:52:22.253 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 328 M 2023-11-28T16:52:22.254 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-28T16:52:22.448 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-11-28T16:52:22.463 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-11-28T16:52:22.463 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-11-28T16:52:22.463 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-11-28T16:52:22.464 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2023-11-28T16:52:22.464 INFO:teuthology.orchestra.run.smithi156.stdout: linux-firmware noarch 20231030-141.el9 CentOS-BaseOS 328 M 2023-11-28T16:52:22.464 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T16:52:22.464 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-11-28T16:52:22.464 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-11-28T16:52:22.464 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 1 Package 2023-11-28T16:52:22.464 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T16:52:22.464 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 328 M 2023-11-28T16:52:22.464 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-11-28T16:52:30.340 INFO:teuthology.orchestra.run.smithi156.stdout:linux-firmware-20231030-141.el9.noarch.rpm 42 MB/s | 328 MB 00:07 2023-11-28T16:52:30.341 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-11-28T16:52:30.341 INFO:teuthology.orchestra.run.smithi156.stdout:Total 42 MB/s | 328 MB 00:07 2023-11-28T16:52:30.342 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-11-28T16:52:30.427 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-11-28T16:52:30.428 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-11-28T16:52:32.607 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-11-28T16:52:32.608 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-11-28T16:52:33.173 INFO:teuthology.orchestra.run.smithi116.stdout:linux-firmware-20231030-141.el9.noarch.rpm 30 MB/s | 328 MB 00:10 2023-11-28T16:52:33.174 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-28T16:52:33.175 INFO:teuthology.orchestra.run.smithi116.stdout:Total 30 MB/s | 328 MB 00:10 2023-11-28T16:52:33.175 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-28T16:52:33.255 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-28T16:52:33.255 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-28T16:52:35.413 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-28T16:52:35.414 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-28T16:52:38.549 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-11-28T16:52:38.900 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-28T16:52:38.987 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T16:52:39.967 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T16:52:39.967 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-28T16:52:40.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T16:52:40.337 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T16:52:40.337 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2023-11-28T16:52:40.337 INFO:teuthology.orchestra.run.smithi156.stdout: linux-firmware-20231030-141.el9.noarch 2023-11-28T16:52:40.337 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T16:52:40.337 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-11-28T16:52:40.614 INFO:teuthology.task.kernel:Downloading kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on mon.b... 2023-11-28T16:52:40.616 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T16:52:40.843 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ 2023-11-28T16:52:40.844 DEBUG:teuthology.orchestra.run.smithi156:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ --input-file=- 2023-11-28T16:52:40.847 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi156.front.sepia.ceph.com to complete... 2023-11-28T16:52:41.130 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-28T16:52:41.513 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-28T16:52:41.612 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T16:52:42.582 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T16:52:42.583 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : linux-firmware-20231030-141.el9.noarch 1/2 2023-11-28T16:52:43.012 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2023-11-28T16:52:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T16:52:43.013 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2023-11-28T16:52:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: linux-firmware-20231030-141.el9.noarch 2023-11-28T16:52:43.013 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T16:52:43.013 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-28T16:52:43.313 INFO:teuthology.task.kernel:Downloading kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on mon.a... 2023-11-28T16:52:43.314 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 2023-11-28T16:52:43.459 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ 2023-11-28T16:52:43.459 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/ --input-file=- 2023-11-28T16:52:43.462 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi116.front.sepia.ceph.com to complete... 2023-11-28T16:52:56.247 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28 16:52:56 URL:https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm [320493731/320493731] -> "/tmp/kernel.x86_64.rpm" [1] 2023-11-28T16:52:56.247 INFO:teuthology.orchestra.run.smithi156.stderr:FINISHED --2023-11-28 16:52:56-- 2023-11-28T16:52:56.247 INFO:teuthology.orchestra.run.smithi156.stderr:Total wall clock time: 15s 2023-11-28T16:52:56.248 INFO:teuthology.orchestra.run.smithi156.stderr:Downloaded: 1 files, 306M in 15s (20.4 MB/s) 2023-11-28T16:52:56.249 INFO:teuthology.task.kernel:Installing kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on mon.b... 2023-11-28T16:52:56.249 DEBUG:teuthology.orchestra.run.smithi156:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-11-28T16:52:58.092 INFO:teuthology.orchestra.run.smithi156.stdout:Verifying... ######################################## 2023-11-28T16:52:58.176 INFO:teuthology.orchestra.run.smithi156.stdout:Preparing... ######################################## 2023-11-28T16:52:58.256 INFO:teuthology.orchestra.run.smithi156.stdout:Updating / installing... 2023-11-28T16:53:05.357 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-6.6.0_g8247bbb5b470-1 ######################################## 2023-11-28T16:53:06.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28 16:53:06 URL:https://3.chacra.ceph.com/r/kernel/testing/8247bbb5b470f7e2a1f3eefcc16e24cf1338f606/centos/9/flavors/default/x86_64/kernel-6.6.0_g8247bbb5b470-1.x86_64.rpm [320493731/320493731] -> "/tmp/kernel.x86_64.rpm" [1] 2023-11-28T16:53:06.099 INFO:teuthology.orchestra.run.smithi116.stderr:FINISHED --2023-11-28 16:53:06-- 2023-11-28T16:53:06.099 INFO:teuthology.orchestra.run.smithi116.stderr:Total wall clock time: 23s 2023-11-28T16:53:06.099 INFO:teuthology.orchestra.run.smithi116.stderr:Downloaded: 1 files, 306M in 22s (13.8 MB/s) 2023-11-28T16:53:06.099 INFO:teuthology.task.kernel:Installing kernel 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 on mon.a... 2023-11-28T16:53:06.099 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-11-28T16:53:07.948 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying... ######################################## 2023-11-28T16:53:08.045 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing... ######################################## 2023-11-28T16:53:08.125 INFO:teuthology.orchestra.run.smithi116.stdout:Updating / installing... 2023-11-28T16:53:15.170 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-6.6.0_g8247bbb5b470-1 ######################################## 2023-11-28T16:53:20.793 INFO:teuthology.orchestra.run.smithi156.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-11-28T16:53:30.003 INFO:teuthology.orchestra.run.smithi116.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-11-28T16:53:42.055 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi156.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-11-28T16:53:42.055 DEBUG:teuthology.orchestra.run.smithi156:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-11-28T16:53:42.136 INFO:teuthology.orchestra.run.smithi156.stdout:/boot/System.map-6.6.0-g8247bbb5b470 2023-11-28T16:53:42.137 INFO:teuthology.orchestra.run.smithi156.stdout:/boot/config-6.6.0-g8247bbb5b470 2023-11-28T16:53:42.137 INFO:teuthology.orchestra.run.smithi156.stdout:/boot/vmlinuz-6.6.0-g8247bbb5b470 2023-11-28T16:53:42.137 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470 2023-11-28T16:53:42.137 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel 2023-11-28T16:53:42.137 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch 2023-11-28T16:53:42.137 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86 2023-11-28T16:53:42.137 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto 2023-11-28T16:53:42.137 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-11-28T16:53:42.137 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-11-28T16:53:42.137 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-11-28T16:53:42.138 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-11-28T16:53:42.138 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-11-28T16:53:42.138 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-11-28T16:53:42.138 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-11-28T16:53:42.138 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32c-intel.ko 2023-11-28T16:53:42.138 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-11-28T16:53:42.138 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-11-28T16:53:42.138 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-11-28T16:53:42.138 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-11-28T16:53:42.138 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx2.ko 2023-11-28T16:53:42.138 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-11-28T16:53:42.138 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-11-28T16:53:42.138 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-11-28T16:53:42.139 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-11-28T16:53:42.139 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-11-28T16:53:42.139 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-11-28T16:53:42.139 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-11-28T16:53:42.139 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel 2023-11-28T16:53:42.139 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu 2023-11-28T16:53:42.139 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce 2023-11-28T16:53:42.139 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-11-28T16:53:42.139 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm 2023-11-28T16:53:42.139 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-amd.ko 2023-11-28T16:53:42.139 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-intel.ko 2023-11-28T16:53:42.139 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm.ko 2023-11-28T16:53:42.140 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto 2023-11-28T16:53:42.140 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ansi_cprng.ko 2023-11-28T16:53:42.140 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/anubis.ko 2023-11-28T16:53:42.140 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/arc4.ko 2023-11-28T16:53:42.140 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx 2023-11-28T16:53:42.140 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_memcpy.ko 2023-11-28T16:53:42.140 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_pq.ko 2023-11-28T16:53:42.140 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_raid6_recov.ko 2023-11-28T16:53:42.140 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_tx.ko 2023-11-28T16:53:42.140 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_xor.ko 2023-11-28T16:53:42.140 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/raid6test.ko 2023-11-28T16:53:42.140 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authenc.ko 2023-11-28T16:53:42.141 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authencesn.ko 2023-11-28T16:53:42.141 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blake2b_generic.ko 2023-11-28T16:53:42.141 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_common.ko 2023-11-28T16:53:42.141 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_generic.ko 2023-11-28T16:53:42.141 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/camellia_generic.ko 2023-11-28T16:53:42.141 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast5_generic.ko 2023-11-28T16:53:42.141 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast6_generic.ko 2023-11-28T16:53:42.141 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast_common.ko 2023-11-28T16:53:42.141 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ccm.ko 2023-11-28T16:53:42.141 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cmac.ko 2023-11-28T16:53:42.141 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crc32_generic.ko 2023-11-28T16:53:42.141 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_engine.ko 2023-11-28T16:53:42.141 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_user.ko 2023-11-28T16:53:42.142 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cts.ko 2023-11-28T16:53:42.142 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/des_generic.ko 2023-11-28T16:53:42.142 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/dh_generic.ko 2023-11-28T16:53:42.142 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/echainiv.ko 2023-11-28T16:53:42.142 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/essiv.ko 2023-11-28T16:53:42.142 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/fcrypt.ko 2023-11-28T16:53:42.142 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/khazad.ko 2023-11-28T16:53:42.142 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4.ko 2023-11-28T16:53:42.142 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4hc.ko 2023-11-28T16:53:42.142 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/md4.ko 2023-11-28T16:53:42.142 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/michael_mic.ko 2023-11-28T16:53:42.142 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcbc.ko 2023-11-28T16:53:42.143 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcrypt.ko 2023-11-28T16:53:42.143 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/rmd160.ko 2023-11-28T16:53:42.143 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/seed.ko 2023-11-28T16:53:42.143 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/serpent_generic.ko 2023-11-28T16:53:42.143 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tcrypt.ko 2023-11-28T16:53:42.143 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tea.ko 2023-11-28T16:53:42.143 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_common.ko 2023-11-28T16:53:42.143 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_generic.ko 2023-11-28T16:53:42.143 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/vmac.ko 2023-11-28T16:53:42.143 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/wp512.ko 2023-11-28T16:53:42.143 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xcbc.ko 2023-11-28T16:53:42.143 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xor.ko 2023-11-28T16:53:42.144 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xxhash_generic.ko 2023-11-28T16:53:42.144 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers 2023-11-28T16:53:42.144 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi 2023-11-28T16:53:42.144 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_ipmi.ko 2023-11-28T16:53:42.144 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_pad.ko 2023-11-28T16:53:42.144 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/custom_method.ko 2023-11-28T16:53:42.144 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/ec_sys.ko 2023-11-28T16:53:42.144 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/platform_profile.ko 2023-11-28T16:53:42.144 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbs.ko 2023-11-28T16:53:42.144 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbshc.ko 2023-11-28T16:53:42.144 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/video.ko 2023-11-28T16:53:42.144 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata 2023-11-28T16:53:42.144 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/acard-ahci.ko 2023-11-28T16:53:42.145 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ahci_platform.ko 2023-11-28T16:53:42.145 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ata_generic.ko 2023-11-28T16:53:42.145 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/libahci_platform.ko 2023-11-28T16:53:42.145 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_acpi.ko 2023-11-28T16:53:42.145 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ali.ko 2023-11-28T16:53:42.145 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_amd.ko 2023-11-28T16:53:42.145 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_artop.ko 2023-11-28T16:53:42.145 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atiixp.ko 2023-11-28T16:53:42.145 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atp867x.ko 2023-11-28T16:53:42.145 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd640.ko 2023-11-28T16:53:42.145 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd64x.ko 2023-11-28T16:53:42.145 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cypress.ko 2023-11-28T16:53:42.145 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_efar.ko 2023-11-28T16:53:42.146 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt366.ko 2023-11-28T16:53:42.146 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt37x.ko 2023-11-28T16:53:42.146 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x2n.ko 2023-11-28T16:53:42.146 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x3.ko 2023-11-28T16:53:42.146 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it8213.ko 2023-11-28T16:53:42.146 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it821x.ko 2023-11-28T16:53:42.146 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_jmicron.ko 2023-11-28T16:53:42.146 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_marvell.ko 2023-11-28T16:53:42.146 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_mpiix.ko 2023-11-28T16:53:42.146 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_netcell.ko 2023-11-28T16:53:42.146 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ninja32.ko 2023-11-28T16:53:42.146 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87410.ko 2023-11-28T16:53:42.146 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87415.ko 2023-11-28T16:53:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_oldpiix.ko 2023-11-28T16:53:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_opti.ko 2023-11-28T16:53:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_optidma.ko 2023-11-28T16:53:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pcmcia.ko 2023-11-28T16:53:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc2027x.ko 2023-11-28T16:53:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-11-28T16:53:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_piccolo.ko 2023-11-28T16:53:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_rdc.ko 2023-11-28T16:53:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sch.ko 2023-11-28T16:53:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_serverworks.ko 2023-11-28T16:53:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sil680.ko 2023-11-28T16:53:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sis.ko 2023-11-28T16:53:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sl82c105.ko 2023-11-28T16:53:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_triflex.ko 2023-11-28T16:53:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_via.ko 2023-11-28T16:53:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pdc_adma.ko 2023-11-28T16:53:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_inic162x.ko 2023-11-28T16:53:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_mv.ko 2023-11-28T16:53:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_nv.ko 2023-11-28T16:53:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_promise.ko 2023-11-28T16:53:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_qstor.ko 2023-11-28T16:53:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil.ko 2023-11-28T16:53:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil24.ko 2023-11-28T16:53:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sis.ko 2023-11-28T16:53:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_svw.ko 2023-11-28T16:53:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sx4.ko 2023-11-28T16:53:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_uli.ko 2023-11-28T16:53:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_via.ko 2023-11-28T16:53:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_vsc.ko 2023-11-28T16:53:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay 2023-11-28T16:53:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay/ks0108.ko 2023-11-28T16:53:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base 2023-11-28T16:53:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap 2023-11-28T16:53:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-i2c.ko 2023-11-28T16:53:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-mmio.ko 2023-11-28T16:53:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma 2023-11-28T16:53:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma/bcma.ko 2023-11-28T16:53:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block 2023-11-28T16:53:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe 2023-11-28T16:53:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe/aoe.ko 2023-11-28T16:53:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/brd.ko 2023-11-28T16:53:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd 2023-11-28T16:53:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd/drbd.ko 2023-11-28T16:53:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/floppy.ko 2023-11-28T16:53:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/loop.ko 2023-11-28T16:53:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx 2023-11-28T16:53:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-11-28T16:53:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/nbd.ko 2023-11-28T16:53:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk 2023-11-28T16:53:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk/null_blk.ko 2023-11-28T16:53:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/pktcdvd.ko 2023-11-28T16:53:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/rbd.ko 2023-11-28T16:53:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/virtio_blk.ko 2023-11-28T16:53:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback 2023-11-28T16:53:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-11-28T16:53:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkfront.ko 2023-11-28T16:53:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram 2023-11-28T16:53:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram/zram.ko 2023-11-28T16:53:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char 2023-11-28T16:53:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hangcheck-timer.ko 2023-11-28T16:53:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random 2023-11-28T16:53:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/amd-rng.ko 2023-11-28T16:53:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/intel-rng.ko 2023-11-28T16:53:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-11-28T16:53:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/via-rng.ko 2023-11-28T16:53:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/virtio-rng.ko 2023-11-28T16:53:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi 2023-11-28T16:53:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-11-28T16:53:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-11-28T16:53:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-11-28T16:53:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_si.ko 2023-11-28T16:53:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-11-28T16:53:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-11-28T16:53:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/lp.ko 2023-11-28T16:53:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave 2023-11-28T16:53:42.153 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave/mwave.ko 2023-11-28T16:53:42.153 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ppdev.ko 2023-11-28T16:53:42.153 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tlclk.ko 2023-11-28T16:53:42.153 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm 2023-11-28T16:53:42.153 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm.ko 2023-11-28T16:53:42.153 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_atmel.ko 2023-11-28T16:53:42.153 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_infineon.ko 2023-11-28T16:53:42.153 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_nsc.ko 2023-11-28T16:53:42.153 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis.ko 2023-11-28T16:53:42.153 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-11-28T16:53:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/uv_mmtimer.ko 2023-11-28T16:53:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/virtio_console.ko 2023-11-28T16:53:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq 2023-11-28T16:53:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-11-28T16:53:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-11-28T16:53:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/p4-clockmod.ko 2023-11-28T16:53:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-11-28T16:53:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/powernow-k8.ko 2023-11-28T16:53:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/speedstep-lib.ko 2023-11-28T16:53:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto 2023-11-28T16:53:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp 2023-11-28T16:53:42.154 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-11-28T16:53:42.155 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp.ko 2023-11-28T16:53:42.155 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel 2023-11-28T16:53:42.155 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat 2023-11-28T16:53:42.155 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common 2023-11-28T16:53:42.155 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-11-28T16:53:42.155 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-11-28T16:53:42.155 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-11-28T16:53:42.155 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-aes.ko 2023-11-28T16:53:42.155 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-sha.ko 2023-11-28T16:53:42.155 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio 2023-11-28T16:53:42.155 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-11-28T16:53:42.155 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca 2023-11-28T16:53:42.156 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca/dca.ko 2023-11-28T16:53:42.156 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq 2023-11-28T16:53:42.156 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-11-28T16:53:42.156 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma 2023-11-28T16:53:42.156 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw 2023-11-28T16:53:42.156 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw/dw_dmac.ko 2023-11-28T16:53:42.156 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat 2023-11-28T16:53:42.156 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat/ioatdma.ko 2023-11-28T16:53:42.156 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac 2023-11-28T16:53:42.156 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/amd64_edac.ko 2023-11-28T16:53:42.156 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/e752x_edac.ko 2023-11-28T16:53:42.156 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/edac_mce_amd.ko 2023-11-28T16:53:42.156 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3000_edac.ko 2023-11-28T16:53:42.157 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3200_edac.ko 2023-11-28T16:53:42.157 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5100_edac.ko 2023-11-28T16:53:42.157 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5400_edac.ko 2023-11-28T16:53:42.157 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7300_edac.ko 2023-11-28T16:53:42.157 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7core_edac.ko 2023-11-28T16:53:42.157 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i82975x_edac.ko 2023-11-28T16:53:42.157 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/ie31200_edac.ko 2023-11-28T16:53:42.157 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/sb_edac.ko 2023-11-28T16:53:42.157 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/x38_edac.ko 2023-11-28T16:53:42.157 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware 2023-11-28T16:53:42.157 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/edd.ko 2023-11-28T16:53:42.158 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/iscsi_ibft.ko 2023-11-28T16:53:42.158 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio 2023-11-28T16:53:42.158 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-ich.ko 2023-11-28T16:53:42.158 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-viperboard.ko 2023-11-28T16:53:42.158 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid 2023-11-28T16:53:42.158 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-appleir.ko 2023-11-28T16:53:42.158 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-aureal.ko 2023-11-28T16:53:42.158 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-axff.ko 2023-11-28T16:53:42.158 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-dr.ko 2023-11-28T16:53:42.158 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elecom.ko 2023-11-28T16:53:42.158 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elo.ko 2023-11-28T16:53:42.158 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-emsff.ko 2023-11-28T16:53:42.158 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gaff.ko 2023-11-28T16:53:42.159 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gt683r.ko 2023-11-28T16:53:42.159 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gyration.ko 2023-11-28T16:53:42.159 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-kbd.ko 2023-11-28T16:53:42.159 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-mouse.ko 2023-11-28T16:53:42.159 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtekff.ko 2023-11-28T16:53:42.159 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-hyperv.ko 2023-11-28T16:53:42.159 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-icade.ko 2023-11-28T16:53:42.159 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-keytouch.ko 2023-11-28T16:53:42.159 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-kye.ko 2023-11-28T16:53:42.159 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lcpower.ko 2023-11-28T16:53:42.159 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-led.ko 2023-11-28T16:53:42.159 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lenovo.ko 2023-11-28T16:53:42.160 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-dj.ko 2023-11-28T16:53:42.160 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-11-28T16:53:42.160 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-multitouch.ko 2023-11-28T16:53:42.160 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-ortek.ko 2023-11-28T16:53:42.160 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-penmount.ko 2023-11-28T16:53:42.160 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-petalynx.ko 2023-11-28T16:53:42.160 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-picolcd.ko 2023-11-28T16:53:42.160 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-pl.ko 2023-11-28T16:53:42.160 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-plantronics.ko 2023-11-28T16:53:42.160 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-primax.ko 2023-11-28T16:53:42.160 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-rmi.ko 2023-11-28T16:53:42.160 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-arvo.ko 2023-11-28T16:53:42.160 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-common.ko 2023-11-28T16:53:42.161 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-isku.ko 2023-11-28T16:53:42.161 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kone.ko 2023-11-28T16:53:42.161 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-11-28T16:53:42.161 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-konepure.ko 2023-11-28T16:53:42.161 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-11-28T16:53:42.161 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-lua.ko 2023-11-28T16:53:42.161 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-pyra.ko 2023-11-28T16:53:42.161 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-ryos.ko 2023-11-28T16:53:42.161 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-savu.ko 2023-11-28T16:53:42.161 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat.ko 2023-11-28T16:53:42.161 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-saitek.ko 2023-11-28T16:53:42.161 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-samsung.ko 2023-11-28T16:53:42.162 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sensor-hub.ko 2023-11-28T16:53:42.162 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sjoy.ko 2023-11-28T16:53:42.162 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sony.ko 2023-11-28T16:53:42.162 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-speedlink.ko 2023-11-28T16:53:42.162 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-steelseries.ko 2023-11-28T16:53:42.162 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sunplus.ko 2023-11-28T16:53:42.162 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-thrustmaster.ko 2023-11-28T16:53:42.162 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tivo.ko 2023-11-28T16:53:42.162 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tmff.ko 2023-11-28T16:53:42.162 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-topseed.ko 2023-11-28T16:53:42.162 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-twinhan.ko 2023-11-28T16:53:42.162 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-uclogic.ko 2023-11-28T16:53:42.163 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-waltop.ko 2023-11-28T16:53:42.163 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-wiimote.ko 2023-11-28T16:53:42.163 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-xinmo.ko 2023-11-28T16:53:42.163 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zpff.ko 2023-11-28T16:53:42.163 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zydacron.ko 2023-11-28T16:53:42.163 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/uhid.ko 2023-11-28T16:53:42.163 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/wacom.ko 2023-11-28T16:53:42.163 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv 2023-11-28T16:53:42.163 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_balloon.ko 2023-11-28T16:53:42.163 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_utils.ko 2023-11-28T16:53:42.163 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_vmbus.ko 2023-11-28T16:53:42.163 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon 2023-11-28T16:53:42.163 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru.ko 2023-11-28T16:53:42.164 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru3.ko 2023-11-28T16:53:42.164 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/acpi_power_meter.ko 2023-11-28T16:53:42.164 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7414.ko 2023-11-28T16:53:42.164 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7418.ko 2023-11-28T16:53:42.164 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adc128d818.ko 2023-11-28T16:53:42.164 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1025.ko 2023-11-28T16:53:42.164 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1026.ko 2023-11-28T16:53:42.164 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1029.ko 2023-11-28T16:53:42.164 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1031.ko 2023-11-28T16:53:42.164 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm9240.ko 2023-11-28T16:53:42.164 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ads7828.ko 2023-11-28T16:53:42.164 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7410.ko 2023-11-28T16:53:42.165 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7411.ko 2023-11-28T16:53:42.165 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7462.ko 2023-11-28T16:53:42.165 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7470.ko 2023-11-28T16:53:42.165 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7475.ko 2023-11-28T16:53:42.165 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7x10.ko 2023-11-28T16:53:42.165 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/amc6821.ko 2023-11-28T16:53:42.165 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/applesmc.ko 2023-11-28T16:53:42.165 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asb100.ko 2023-11-28T16:53:42.165 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asc7621.ko 2023-11-28T16:53:42.165 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asus_atk0110.ko 2023-11-28T16:53:42.165 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/atxp1.ko 2023-11-28T16:53:42.165 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/coretemp.ko 2023-11-28T16:53:42.165 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-11-28T16:53:42.166 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dme1737.ko 2023-11-28T16:53:42.166 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds1621.ko 2023-11-28T16:53:42.166 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds620.ko 2023-11-28T16:53:42.166 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc1403.ko 2023-11-28T16:53:42.166 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc6w201.ko 2023-11-28T16:53:42.166 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71805f.ko 2023-11-28T16:53:42.166 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71882fg.ko 2023-11-28T16:53:42.166 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f75375s.ko 2023-11-28T16:53:42.166 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fam15h_power.ko 2023-11-28T16:53:42.166 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fschmd.ko 2023-11-28T16:53:42.166 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g760a.ko 2023-11-28T16:53:42.166 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g762.ko 2023-11-28T16:53:42.167 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl518sm.ko 2023-11-28T16:53:42.167 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl520sm.ko 2023-11-28T16:53:42.167 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/hwmon-vid.ko 2023-11-28T16:53:42.167 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5500_temp.ko 2023-11-28T16:53:42.167 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5k_amb.ko 2023-11-28T16:53:42.167 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmaem.ko 2023-11-28T16:53:42.167 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmpex.ko 2023-11-28T16:53:42.167 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina209.ko 2023-11-28T16:53:42.167 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina2xx.ko 2023-11-28T16:53:42.167 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/it87.ko 2023-11-28T16:53:42.167 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k10temp.ko 2023-11-28T16:53:42.167 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k8temp.ko 2023-11-28T16:53:42.168 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lineage-pem.ko 2023-11-28T16:53:42.168 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm63.ko 2023-11-28T16:53:42.168 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm73.ko 2023-11-28T16:53:42.168 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm75.ko 2023-11-28T16:53:42.168 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm77.ko 2023-11-28T16:53:42.168 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm78.ko 2023-11-28T16:53:42.168 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm80.ko 2023-11-28T16:53:42.168 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm83.ko 2023-11-28T16:53:42.168 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm85.ko 2023-11-28T16:53:42.168 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm87.ko 2023-11-28T16:53:42.168 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm90.ko 2023-11-28T16:53:42.168 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm92.ko 2023-11-28T16:53:42.169 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm93.ko 2023-11-28T16:53:42.169 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95234.ko 2023-11-28T16:53:42.169 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95241.ko 2023-11-28T16:53:42.169 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95245.ko 2023-11-28T16:53:42.169 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc2945.ko 2023-11-28T16:53:42.169 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4151.ko 2023-11-28T16:53:42.169 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4215.ko 2023-11-28T16:53:42.169 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4222.ko 2023-11-28T16:53:42.169 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4245.ko 2023-11-28T16:53:42.169 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4260.ko 2023-11-28T16:53:42.169 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4261.ko 2023-11-28T16:53:42.169 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max16065.ko 2023-11-28T16:53:42.170 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1619.ko 2023-11-28T16:53:42.170 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1668.ko 2023-11-28T16:53:42.170 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max197.ko 2023-11-28T16:53:42.170 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6639.ko 2023-11-28T16:53:42.170 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6650.ko 2023-11-28T16:53:42.170 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6697.ko 2023-11-28T16:53:42.170 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/mcp3021.ko 2023-11-28T16:53:42.170 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6683.ko 2023-11-28T16:53:42.170 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775-core.ko 2023-11-28T16:53:42.170 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775.ko 2023-11-28T16:53:42.170 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct7802.ko 2023-11-28T16:53:42.170 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87360.ko 2023-11-28T16:53:42.171 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87427.ko 2023-11-28T16:53:42.171 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pcf8591.ko 2023-11-28T16:53:42.171 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus 2023-11-28T16:53:42.171 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-11-28T16:53:42.171 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-11-28T16:53:42.171 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-11-28T16:53:42.171 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max16064.ko 2023-11-28T16:53:42.171 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max34440.ko 2023-11-28T16:53:42.171 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max8688.ko 2023-11-28T16:53:42.171 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-11-28T16:53:42.171 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-11-28T16:53:42.171 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-11-28T16:53:42.172 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-11-28T16:53:42.172 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-11-28T16:53:42.172 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-11-28T16:53:42.172 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/powr1220.ko 2023-11-28T16:53:42.172 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5627.ko 2023-11-28T16:53:42.172 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5636.ko 2023-11-28T16:53:42.172 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch56xx-common.ko 2023-11-28T16:53:42.172 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht15.ko 2023-11-28T16:53:42.172 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht21.ko 2023-11-28T16:53:42.172 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/shtc1.ko 2023-11-28T16:53:42.172 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sis5595.ko 2023-11-28T16:53:42.172 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47b397.ko 2023-11-28T16:53:42.173 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m1.ko 2023-11-28T16:53:42.173 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m192.ko 2023-11-28T16:53:42.173 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/thmc50.ko 2023-11-28T16:53:42.173 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp102.ko 2023-11-28T16:53:42.173 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp103.ko 2023-11-28T16:53:42.173 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp401.ko 2023-11-28T16:53:42.173 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp421.ko 2023-11-28T16:53:42.173 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via-cputemp.ko 2023-11-28T16:53:42.173 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via686a.ko 2023-11-28T16:53:42.173 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt1211.ko 2023-11-28T16:53:42.173 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt8231.ko 2023-11-28T16:53:42.173 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627ehf.ko 2023-11-28T16:53:42.173 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627hf.ko 2023-11-28T16:53:42.174 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83781d.ko 2023-11-28T16:53:42.174 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83791d.ko 2023-11-28T16:53:42.174 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83792d.ko 2023-11-28T16:53:42.174 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83793.ko 2023-11-28T16:53:42.174 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83795.ko 2023-11-28T16:53:42.174 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l785ts.ko 2023-11-28T16:53:42.174 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l786ng.ko 2023-11-28T16:53:42.174 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c 2023-11-28T16:53:42.174 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos 2023-11-28T16:53:42.174 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-11-28T16:53:42.174 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-11-28T16:53:42.174 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses 2023-11-28T16:53:42.174 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-11-28T16:53:42.175 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-11-28T16:53:42.175 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-11-28T16:53:42.175 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-11-28T16:53:42.175 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-11-28T16:53:42.175 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-11-28T16:53:42.175 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-11-28T16:53:42.175 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-11-28T16:53:42.175 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-i801.ko 2023-11-28T16:53:42.175 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-isch.ko 2023-11-28T16:53:42.175 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-11-28T16:53:42.175 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-11-28T16:53:42.175 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-11-28T16:53:42.175 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-parport.ko 2023-11-28T16:53:42.176 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-11-28T16:53:42.176 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-11-28T16:53:42.176 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-11-28T16:53:42.176 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-11-28T16:53:42.176 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-11-28T16:53:42.176 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-11-28T16:53:42.176 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-via.ko 2023-11-28T16:53:42.176 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-11-28T16:53:42.176 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-11-28T16:53:42.176 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-dev.ko 2023-11-28T16:53:42.176 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-mux.ko 2023-11-28T16:53:42.176 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-11-28T16:53:42.177 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-smbus.ko 2023-11-28T16:53:42.177 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-stub.ko 2023-11-28T16:53:42.177 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband 2023-11-28T16:53:42.177 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core 2023-11-28T16:53:42.177 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_cm.ko 2023-11-28T16:53:42.177 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_core.ko 2023-11-28T16:53:42.177 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_umad.ko 2023-11-28T16:53:42.177 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-11-28T16:53:42.177 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/iw_cm.ko 2023-11-28T16:53:42.177 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_cm.ko 2023-11-28T16:53:42.177 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-11-28T16:53:42.177 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw 2023-11-28T16:53:42.178 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4 2023-11-28T16:53:42.178 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-11-28T16:53:42.178 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4 2023-11-28T16:53:42.178 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-11-28T16:53:42.178 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5 2023-11-28T16:53:42.178 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-11-28T16:53:42.178 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca 2023-11-28T16:53:42.178 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-11-28T16:53:42.178 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp 2023-11-28T16:53:42.178 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib 2023-11-28T16:53:42.178 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-11-28T16:53:42.178 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser 2023-11-28T16:53:42.178 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-11-28T16:53:42.179 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert 2023-11-28T16:53:42.179 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-11-28T16:53:42.179 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp 2023-11-28T16:53:42.179 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-11-28T16:53:42.179 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt 2023-11-28T16:53:42.179 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-11-28T16:53:42.179 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input 2023-11-28T16:53:42.179 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport 2023-11-28T16:53:42.179 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-11-28T16:53:42.179 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/fm801-gp.ko 2023-11-28T16:53:42.179 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/gameport.ko 2023-11-28T16:53:42.179 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/joydev.ko 2023-11-28T16:53:42.180 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard 2023-11-28T16:53:42.180 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/gpio_keys.ko 2023-11-28T16:53:42.180 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse 2023-11-28T16:53:42.180 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/appletouch.ko 2023-11-28T16:53:42.180 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/bcm5974.ko 2023-11-28T16:53:42.180 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/cyapatp.ko 2023-11-28T16:53:42.180 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/elan_i2c.ko 2023-11-28T16:53:42.180 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/sermouse.ko 2023-11-28T16:53:42.180 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-11-28T16:53:42.180 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_usb.ko 2023-11-28T16:53:42.180 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/vsxxxaa.ko 2023-11-28T16:53:42.180 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4 2023-11-28T16:53:42.180 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4/rmi_core.ko 2023-11-28T16:53:42.181 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio 2023-11-28T16:53:42.181 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/altera_ps2.ko 2023-11-28T16:53:42.181 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/arc_ps2.ko 2023-11-28T16:53:42.181 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-11-28T16:53:42.181 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/serio_raw.ko 2023-11-28T16:53:42.181 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/sparse-keymap.ko 2023-11-28T16:53:42.181 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu 2023-11-28T16:53:42.181 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd 2023-11-28T16:53:42.181 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd/iommu_v2.ko 2023-11-28T16:53:42.181 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds 2023-11-28T16:53:42.181 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-blinkm.ko 2023-11-28T16:53:42.181 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm3530.ko 2023-11-28T16:53:42.182 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lp3944.ko 2023-11-28T16:53:42.182 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-ss4200.ko 2023-11-28T16:53:42.182 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger 2023-11-28T16:53:42.182 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-11-28T16:53:42.182 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-11-28T16:53:42.182 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-11-28T16:53:42.182 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-11-28T16:53:42.182 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-11-28T16:53:42.182 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-11-28T16:53:42.182 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-11-28T16:53:42.182 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-11-28T16:53:42.182 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md 2023-11-28T16:53:42.182 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache 2023-11-28T16:53:42.183 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache/bcache.ko 2023-11-28T16:53:42.183 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-bio-prison.ko 2023-11-28T16:53:42.183 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache-smq.ko 2023-11-28T16:53:42.183 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache.ko 2023-11-28T16:53:42.183 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-crypt.ko 2023-11-28T16:53:42.183 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-delay.ko 2023-11-28T16:53:42.183 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-flakey.ko 2023-11-28T16:53:42.183 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-log-userspace.ko 2023-11-28T16:53:42.183 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-multipath.ko 2023-11-28T16:53:42.183 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-queue-length.ko 2023-11-28T16:53:42.183 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-raid.ko 2023-11-28T16:53:42.183 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-round-robin.ko 2023-11-28T16:53:42.184 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-service-time.ko 2023-11-28T16:53:42.184 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-switch.ko 2023-11-28T16:53:42.184 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-thin-pool.ko 2023-11-28T16:53:42.184 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-verity.ko 2023-11-28T16:53:42.184 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/faulty.ko 2023-11-28T16:53:42.184 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/linear.ko 2023-11-28T16:53:42.184 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/multipath.ko 2023-11-28T16:53:42.184 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data 2023-11-28T16:53:42.184 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-11-28T16:53:42.184 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid0.ko 2023-11-28T16:53:42.184 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid1.ko 2023-11-28T16:53:42.184 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid10.ko 2023-11-28T16:53:42.185 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid456.ko 2023-11-28T16:53:42.185 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message 2023-11-28T16:53:42.185 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion 2023-11-28T16:53:42.185 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptbase.ko 2023-11-28T16:53:42.185 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptctl.ko 2023-11-28T16:53:42.185 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptfc.ko 2023-11-28T16:53:42.185 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptlan.ko 2023-11-28T16:53:42.185 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptsas.ko 2023-11-28T16:53:42.185 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptscsih.ko 2023-11-28T16:53:42.185 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptspi.ko 2023-11-28T16:53:42.185 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd 2023-11-28T16:53:42.185 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_ich.ko 2023-11-28T16:53:42.186 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_sch.ko 2023-11-28T16:53:42.186 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/mfd-core.ko 2023-11-28T16:53:42.186 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/sm501.ko 2023-11-28T16:53:42.186 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/viperboard.ko 2023-11-28T16:53:42.186 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/vx855.ko 2023-11-28T16:53:42.186 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/wl1273-core.ko 2023-11-28T16:53:42.186 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc 2023-11-28T16:53:42.186 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl 2023-11-28T16:53:42.186 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-11-28T16:53:42.186 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds9802als.ko 2023-11-28T16:53:42.186 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds990x.ko 2023-11-28T16:53:42.186 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/bh1770glc.ko 2023-11-28T16:53:42.186 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710 2023-11-28T16:53:42.187 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710/cb710.ko 2023-11-28T16:53:42.187 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom 2023-11-28T16:53:42.187 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/at24.ko 2023-11-28T16:53:42.187 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom.ko 2023-11-28T16:53:42.187 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-11-28T16:53:42.187 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/max6875.ko 2023-11-28T16:53:42.187 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/enclosure.ko 2023-11-28T16:53:42.187 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/hpilo.ko 2023-11-28T16:53:42.187 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29003.ko 2023-11-28T16:53:42.187 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29020.ko 2023-11-28T16:53:42.187 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d 2023-11-28T16:53:42.188 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-11-28T16:53:42.188 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-11-28T16:53:42.188 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei 2023-11-28T16:53:42.188 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-me.ko 2023-11-28T16:53:42.188 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-txe.ko 2023-11-28T16:53:42.188 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei.ko 2023-11-28T16:53:42.188 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru 2023-11-28T16:53:42.188 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru/gru.ko 2023-11-28T16:53:42.188 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp 2023-11-28T16:53:42.188 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xp.ko 2023-11-28T16:53:42.188 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpc.ko 2023-11-28T16:53:42.188 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-11-28T16:53:42.189 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_7xx1.ko 2023-11-28T16:53:42.189 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_core.ko 2023-11-28T16:53:42.189 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tsl2550.ko 2023-11-28T16:53:42.189 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_balloon.ko 2023-11-28T16:53:42.189 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci 2023-11-28T16:53:42.189 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-11-28T16:53:42.189 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net 2023-11-28T16:53:42.189 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding 2023-11-28T16:53:42.189 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding/bonding.ko 2023-11-28T16:53:42.189 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa 2023-11-28T16:53:42.189 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53 2023-11-28T16:53:42.189 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53/b53_common.ko 2023-11-28T16:53:42.189 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/bcm-sf2.ko 2023-11-28T16:53:42.190 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6060.ko 2023-11-28T16:53:42.190 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx 2023-11-28T16:53:42.190 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-11-28T16:53:42.190 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dummy.ko 2023-11-28T16:53:42.190 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/eql.ko 2023-11-28T16:53:42.190 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet 2023-11-28T16:53:42.190 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com 2023-11-28T16:53:42.190 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-11-28T16:53:42.190 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-11-28T16:53:42.190 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-11-28T16:53:42.190 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-11-28T16:53:42.190 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390 2023-11-28T16:53:42.191 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/8390.ko 2023-11-28T16:53:42.191 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-11-28T16:53:42.191 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-11-28T16:53:42.191 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-11-28T16:53:42.191 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec 2023-11-28T16:53:42.191 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-11-28T16:53:42.191 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere 2023-11-28T16:53:42.191 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere/et131x.ko 2023-11-28T16:53:42.191 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon 2023-11-28T16:53:42.191 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-11-28T16:53:42.191 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera 2023-11-28T16:53:42.191 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-11-28T16:53:42.191 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd 2023-11-28T16:53:42.192 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-11-28T16:53:42.192 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-11-28T16:53:42.192 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-11-28T16:53:42.192 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros 2023-11-28T16:53:42.192 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx 2023-11-28T16:53:42.192 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-11-28T16:53:42.192 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c 2023-11-28T16:53:42.192 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-11-28T16:53:42.192 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e 2023-11-28T16:53:42.192 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-11-28T16:53:42.192 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx 2023-11-28T16:53:42.192 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-11-28T16:53:42.192 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-11-28T16:53:42.193 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom 2023-11-28T16:53:42.193 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-11-28T16:53:42.193 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-11-28T16:53:42.193 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-11-28T16:53:42.193 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-11-28T16:53:42.193 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-11-28T16:53:42.193 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet 2023-11-28T16:53:42.193 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-11-28T16:53:42.193 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-11-28T16:53:42.193 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade 2023-11-28T16:53:42.193 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna 2023-11-28T16:53:42.193 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-11-28T16:53:42.194 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio 2023-11-28T16:53:42.194 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb 2023-11-28T16:53:42.194 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-11-28T16:53:42.194 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-11-28T16:53:42.194 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-11-28T16:53:42.194 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-11-28T16:53:42.194 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-11-28T16:53:42.194 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-11-28T16:53:42.194 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-11-28T16:53:42.194 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-11-28T16:53:42.194 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-11-28T16:53:42.194 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco 2023-11-28T16:53:42.194 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic 2023-11-28T16:53:42.195 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-11-28T16:53:42.195 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec 2023-11-28T16:53:42.195 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip 2023-11-28T16:53:42.195 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-11-28T16:53:42.195 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-11-28T16:53:42.195 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-11-28T16:53:42.195 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-11-28T16:53:42.195 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-11-28T16:53:42.195 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-11-28T16:53:42.195 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink 2023-11-28T16:53:42.195 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-11-28T16:53:42.195 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-11-28T16:53:42.195 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dnet.ko 2023-11-28T16:53:42.196 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex 2023-11-28T16:53:42.196 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet 2023-11-28T16:53:42.196 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-11-28T16:53:42.196 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ethoc.ko 2023-11-28T16:53:42.196 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/fealnx.ko 2023-11-28T16:53:42.196 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel 2023-11-28T16:53:42.196 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e100.ko 2023-11-28T16:53:42.196 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000 2023-11-28T16:53:42.196 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-11-28T16:53:42.196 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e 2023-11-28T16:53:42.196 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-11-28T16:53:42.196 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k 2023-11-28T16:53:42.197 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-11-28T16:53:42.197 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e 2023-11-28T16:53:42.197 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-11-28T16:53:42.197 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf 2023-11-28T16:53:42.197 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-11-28T16:53:42.197 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb 2023-11-28T16:53:42.197 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-11-28T16:53:42.197 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf 2023-11-28T16:53:42.197 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-11-28T16:53:42.197 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe 2023-11-28T16:53:42.197 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-11-28T16:53:42.197 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf 2023-11-28T16:53:42.198 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-11-28T16:53:42.198 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/jme.ko 2023-11-28T16:53:42.198 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell 2023-11-28T16:53:42.198 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-11-28T16:53:42.198 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/skge.ko 2023-11-28T16:53:42.198 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-11-28T16:53:42.198 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox 2023-11-28T16:53:42.198 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4 2023-11-28T16:53:42.198 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-11-28T16:53:42.198 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-11-28T16:53:42.198 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5 2023-11-28T16:53:42.198 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-11-28T16:53:42.198 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-11-28T16:53:42.199 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel 2023-11-28T16:53:42.199 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-11-28T16:53:42.199 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom 2023-11-28T16:53:42.199 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge 2023-11-28T16:53:42.199 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-11-28T16:53:42.199 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi 2023-11-28T16:53:42.199 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-11-28T16:53:42.199 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-11-28T16:53:42.199 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion 2023-11-28T16:53:42.199 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-11-28T16:53:42.199 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia 2023-11-28T16:53:42.199 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-11-28T16:53:42.200 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines 2023-11-28T16:53:42.200 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-11-28T16:53:42.200 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-11-28T16:53:42.200 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic 2023-11-28T16:53:42.200 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen 2023-11-28T16:53:42.200 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-11-28T16:53:42.200 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-11-28T16:53:42.200 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-11-28T16:53:42.200 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-11-28T16:53:42.200 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc 2023-11-28T16:53:42.200 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-11-28T16:53:42.200 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek 2023-11-28T16:53:42.201 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-11-28T16:53:42.201 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-11-28T16:53:42.201 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/atp.ko 2023-11-28T16:53:42.201 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-11-28T16:53:42.201 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker 2023-11-28T16:53:42.201 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-11-28T16:53:42.201 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc 2023-11-28T16:53:42.201 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-11-28T16:53:42.201 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan 2023-11-28T16:53:42.201 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-11-28T16:53:42.201 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis 2023-11-28T16:53:42.201 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis190.ko 2023-11-28T16:53:42.201 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis900.ko 2023-11-28T16:53:42.202 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc 2023-11-28T16:53:42.202 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-11-28T16:53:42.202 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-11-28T16:53:42.202 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-11-28T16:53:42.202 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-11-28T16:53:42.202 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro 2023-11-28T16:53:42.202 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac 2023-11-28T16:53:42.202 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-11-28T16:53:42.202 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-11-28T16:53:42.202 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun 2023-11-28T16:53:42.203 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/cassini.ko 2023-11-28T16:53:42.203 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/niu.ko 2023-11-28T16:53:42.203 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sungem.ko 2023-11-28T16:53:42.203 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-11-28T16:53:42.203 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti 2023-11-28T16:53:42.203 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-11-28T16:53:42.203 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti 2023-11-28T16:53:42.203 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti/tlan.ko 2023-11-28T16:53:42.203 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via 2023-11-28T16:53:42.203 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-11-28T16:53:42.203 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-11-28T16:53:42.203 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet 2023-11-28T16:53:42.204 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-11-28T16:53:42.204 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-11-28T16:53:42.204 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom 2023-11-28T16:53:42.204 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-11-28T16:53:42.204 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/geneve.ko 2023-11-28T16:53:42.204 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv 2023-11-28T16:53:42.204 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-11-28T16:53:42.204 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154 2023-11-28T16:53:42.204 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154/fakelb.ko 2023-11-28T16:53:42.204 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ifb.ko 2023-11-28T16:53:42.204 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan 2023-11-28T16:53:42.204 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan/ipvlan.ko 2023-11-28T16:53:42.204 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvlan.ko 2023-11-28T16:53:42.205 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvtap.ko 2023-11-28T16:53:42.205 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio 2023-11-28T16:53:42.205 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio.ko 2023-11-28T16:53:42.205 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-11-28T16:53:42.205 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-11-28T16:53:42.205 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-regmap.ko 2023-11-28T16:53:42.205 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mii.ko 2023-11-28T16:53:42.205 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/net_failover.ko 2023-11-28T16:53:42.205 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/netconsole.ko 2023-11-28T16:53:42.205 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/nlmon.ko 2023-11-28T16:53:42.205 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs 2023-11-28T16:53:42.205 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs-lynx.ko 2023-11-28T16:53:42.206 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-11-28T16:53:42.206 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy 2023-11-28T16:53:42.206 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/amd.ko 2023-11-28T16:53:42.206 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-11-28T16:53:42.206 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-11-28T16:53:42.206 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm7xxx.ko 2023-11-28T16:53:42.206 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm87xx.ko 2023-11-28T16:53:42.206 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/broadcom.ko 2023-11-28T16:53:42.206 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/cicada.ko 2023-11-28T16:53:42.206 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/davicom.ko 2023-11-28T16:53:42.206 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/dp83640.ko 2023-11-28T16:53:42.206 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/et1011c.ko 2023-11-28T16:53:42.207 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/icplus.ko 2023-11-28T16:53:42.207 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/lxt.ko 2023-11-28T16:53:42.207 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/marvell.ko 2023-11-28T16:53:42.207 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/micrel.ko 2023-11-28T16:53:42.207 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/national.ko 2023-11-28T16:53:42.207 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/phylink.ko 2023-11-28T16:53:42.207 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/qsemi.ko 2023-11-28T16:53:42.207 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/realtek.ko 2023-11-28T16:53:42.207 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/smsc.ko 2023-11-28T16:53:42.207 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/ste10Xp.ko 2023-11-28T16:53:42.207 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/vitesse.ko 2023-11-28T16:53:42.207 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp 2023-11-28T16:53:42.208 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/bsd_comp.ko 2023-11-28T16:53:42.208 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_async.ko 2023-11-28T16:53:42.208 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_deflate.ko 2023-11-28T16:53:42.208 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_generic.ko 2023-11-28T16:53:42.208 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_mppe.ko 2023-11-28T16:53:42.208 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_synctty.ko 2023-11-28T16:53:42.208 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppoe.ko 2023-11-28T16:53:42.208 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppox.ko 2023-11-28T16:53:42.208 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pptp.ko 2023-11-28T16:53:42.208 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip 2023-11-28T16:53:42.208 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slhc.ko 2023-11-28T16:53:42.208 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slip.ko 2023-11-28T16:53:42.208 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/sungem_phy.ko 2023-11-28T16:53:42.209 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tap.ko 2023-11-28T16:53:42.209 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team 2023-11-28T16:53:42.209 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team.ko 2023-11-28T16:53:42.209 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_activebackup.ko 2023-11-28T16:53:42.209 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_broadcast.ko 2023-11-28T16:53:42.209 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-11-28T16:53:42.209 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_random.ko 2023-11-28T16:53:42.209 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-11-28T16:53:42.209 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tun.ko 2023-11-28T16:53:42.209 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/veth.ko 2023-11-28T16:53:42.209 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/virtio_net.ko 2023-11-28T16:53:42.209 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3 2023-11-28T16:53:42.210 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-11-28T16:53:42.210 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan 2023-11-28T16:53:42.210 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan/vxlan.ko 2023-11-28T16:53:42.210 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback 2023-11-28T16:53:42.210 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback/xen-netback.ko 2023-11-28T16:53:42.210 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netfront.ko 2023-11-28T16:53:42.210 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb 2023-11-28T16:53:42.210 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb/ntb.ko 2023-11-28T16:53:42.210 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme 2023-11-28T16:53:42.210 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host 2023-11-28T16:53:42.210 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme-core.ko 2023-11-28T16:53:42.210 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme.ko 2023-11-28T16:53:42.210 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport 2023-11-28T16:53:42.211 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport.ko 2023-11-28T16:53:42.211 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_cs.ko 2023-11-28T16:53:42.211 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_pc.ko 2023-11-28T16:53:42.211 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_serial.ko 2023-11-28T16:53:42.211 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci 2023-11-28T16:53:42.211 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug 2023-11-28T16:53:42.211 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-11-28T16:53:42.211 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie 2023-11-28T16:53:42.211 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie/aer_inject.ko 2023-11-28T16:53:42.211 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/xen-pcifront.ko 2023-11-28T16:53:42.211 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia 2023-11-28T16:53:42.211 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/i82092.ko 2023-11-28T16:53:42.212 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pd6729.ko 2023-11-28T16:53:42.212 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/yenta_socket.ko 2023-11-28T16:53:42.212 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl 2023-11-28T16:53:42.212 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel 2023-11-28T16:53:42.212 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-11-28T16:53:42.212 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform 2023-11-28T16:53:42.212 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86 2023-11-28T16:53:42.212 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acer-wmi.ko 2023-11-28T16:53:42.212 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acerhdf.ko 2023-11-28T16:53:42.212 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/apple-gmux.ko 2023-11-28T16:53:42.212 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-laptop.ko 2023-11-28T16:53:42.212 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-11-28T16:53:42.212 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-wmi.ko 2023-11-28T16:53:42.213 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/classmate-laptop.ko 2023-11-28T16:53:42.213 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-11-28T16:53:42.213 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-11-28T16:53:42.213 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-11-28T16:53:42.213 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-11-28T16:53:42.213 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/hdaps.ko 2023-11-28T16:53:42.213 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel 2023-11-28T16:53:42.213 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-11-28T16:53:42.213 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel_ips.ko 2023-11-28T16:53:42.213 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/msi-wmi.ko 2023-11-28T16:53:42.213 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/mxm-wmi.ko 2023-11-28T16:53:42.213 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-11-28T16:53:42.213 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-laptop.ko 2023-11-28T16:53:42.214 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-q10.ko 2023-11-28T16:53:42.214 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/topstar-laptop.ko 2023-11-28T16:53:42.214 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-11-28T16:53:42.214 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_haps.ko 2023-11-28T16:53:42.214 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi-bmof.ko 2023-11-28T16:53:42.214 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi.ko 2023-11-28T16:53:42.214 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap 2023-11-28T16:53:42.214 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_common.ko 2023-11-28T16:53:42.214 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_msr.ko 2023-11-28T16:53:42.214 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps 2023-11-28T16:53:42.214 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients 2023-11-28T16:53:42.214 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-gpio.ko 2023-11-28T16:53:42.215 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-ldisc.ko 2023-11-28T16:53:42.215 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps_parport.ko 2023-11-28T16:53:42.215 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/pps_core.ko 2023-11-28T16:53:42.215 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp 2023-11-28T16:53:42.215 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp.ko 2023-11-28T16:53:42.215 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp_kvm.ko 2023-11-28T16:53:42.215 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc 2023-11-28T16:53:42.215 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-bq32k.ko 2023-11-28T16:53:42.215 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1286.ko 2023-11-28T16:53:42.215 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1307.ko 2023-11-28T16:53:42.215 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1374.ko 2023-11-28T16:53:42.215 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1511.ko 2023-11-28T16:53:42.215 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1553.ko 2023-11-28T16:53:42.216 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1672.ko 2023-11-28T16:53:42.216 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1742.ko 2023-11-28T16:53:42.216 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds2404.ko 2023-11-28T16:53:42.216 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds3232.ko 2023-11-28T16:53:42.216 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-em3027.ko 2023-11-28T16:53:42.216 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-fm3130.ko 2023-11-28T16:53:42.216 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl12022.ko 2023-11-28T16:53:42.216 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl1208.ko 2023-11-28T16:53:42.216 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m41t80.ko 2023-11-28T16:53:42.216 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t35.ko 2023-11-28T16:53:42.216 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t59.ko 2023-11-28T16:53:42.216 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max6900.ko 2023-11-28T16:53:42.216 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-msm6242.ko 2023-11-28T16:53:42.217 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf2127.ko 2023-11-28T16:53:42.217 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf85063.ko 2023-11-28T16:53:42.217 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8523.ko 2023-11-28T16:53:42.217 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8563.ko 2023-11-28T16:53:42.217 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8583.ko 2023-11-28T16:53:42.217 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rp5c01.ko 2023-11-28T16:53:42.217 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rs5c372.ko 2023-11-28T16:53:42.217 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-11-28T16:53:42.217 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8025.ko 2023-11-28T16:53:42.217 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8581.ko 2023-11-28T16:53:42.217 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-11-28T16:53:42.217 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-x1205.ko 2023-11-28T16:53:42.217 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi 2023-11-28T16:53:42.218 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-9xxx.ko 2023-11-28T16:53:42.218 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-sas.ko 2023-11-28T16:53:42.218 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-xxxx.ko 2023-11-28T16:53:42.218 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/BusLogic.ko 2023-11-28T16:53:42.218 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/a100u2w.ko 2023-11-28T16:53:42.218 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid 2023-11-28T16:53:42.218 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid/aacraid.ko 2023-11-28T16:53:42.218 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/advansys.ko 2023-11-28T16:53:42.218 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx 2023-11-28T16:53:42.218 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-11-28T16:53:42.218 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-11-28T16:53:42.218 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/am53c974.ko 2023-11-28T16:53:42.219 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr 2023-11-28T16:53:42.219 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-11-28T16:53:42.219 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/atp870u.ko 2023-11-28T16:53:42.219 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi 2023-11-28T16:53:42.219 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-11-28T16:53:42.219 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa 2023-11-28T16:53:42.219 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa/bfa.ko 2023-11-28T16:53:42.219 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc 2023-11-28T16:53:42.219 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-11-28T16:53:42.219 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i 2023-11-28T16:53:42.219 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-11-28T16:53:42.219 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ch.ko 2023-11-28T16:53:42.220 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor 2023-11-28T16:53:42.220 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor/csiostor.ko 2023-11-28T16:53:42.220 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi 2023-11-28T16:53:42.220 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i 2023-11-28T16:53:42.220 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-11-28T16:53:42.220 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i 2023-11-28T16:53:42.220 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-11-28T16:53:42.220 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-11-28T16:53:42.220 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dc395x.ko 2023-11-28T16:53:42.220 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler 2023-11-28T16:53:42.220 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-11-28T16:53:42.220 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-11-28T16:53:42.220 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-11-28T16:53:42.221 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-11-28T16:53:42.221 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dmx3191d.ko 2023-11-28T16:53:42.221 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r 2023-11-28T16:53:42.221 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r/esas2r.ko 2023-11-28T16:53:42.221 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esp_scsi.ko 2023-11-28T16:53:42.221 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe 2023-11-28T16:53:42.221 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/fcoe.ko 2023-11-28T16:53:42.221 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-11-28T16:53:42.221 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic 2023-11-28T16:53:42.221 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic/fnic.ko 2023-11-28T16:53:42.221 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hpsa.ko 2023-11-28T16:53:42.221 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hptiop.ko 2023-11-28T16:53:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hv_storvsc.ko 2023-11-28T16:53:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/initio.ko 2023-11-28T16:53:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ipr.ko 2023-11-28T16:53:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ips.ko 2023-11-28T16:53:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci 2023-11-28T16:53:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci/isci.ko 2023-11-28T16:53:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-11-28T16:53:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_tcp.ko 2023-11-28T16:53:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc 2023-11-28T16:53:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc/libfc.ko 2023-11-28T16:53:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi.ko 2023-11-28T16:53:42.222 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi_tcp.ko 2023-11-28T16:53:42.223 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas 2023-11-28T16:53:42.223 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas/libsas.ko 2023-11-28T16:53:42.223 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc 2023-11-28T16:53:42.223 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc/lpfc.ko 2023-11-28T16:53:42.223 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid 2023-11-28T16:53:42.223 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid.ko 2023-11-28T16:53:42.223 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-11-28T16:53:42.223 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-11-28T16:53:42.223 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-11-28T16:53:42.223 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas 2023-11-28T16:53:42.223 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-11-28T16:53:42.223 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas 2023-11-28T16:53:42.224 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas/mvsas.ko 2023-11-28T16:53:42.224 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvumi.ko 2023-11-28T16:53:42.224 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001 2023-11-28T16:53:42.224 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-11-28T16:53:42.224 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pmcraid.ko 2023-11-28T16:53:42.224 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla1280.ko 2023-11-28T16:53:42.224 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx 2023-11-28T16:53:42.224 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-11-28T16:53:42.224 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-11-28T16:53:42.224 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx 2023-11-28T16:53:42.224 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-11-28T16:53:42.224 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/raid_class.ko 2023-11-28T16:53:42.225 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_debug.ko 2023-11-28T16:53:42.225 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_fc.ko 2023-11-28T16:53:42.225 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-11-28T16:53:42.225 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_sas.ko 2023-11-28T16:53:42.225 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_spi.ko 2023-11-28T16:53:42.225 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_srp.ko 2023-11-28T16:53:42.225 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ses.ko 2023-11-28T16:53:42.225 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/st.ko 2023-11-28T16:53:42.225 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/stex.ko 2023-11-28T16:53:42.225 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2 2023-11-28T16:53:42.225 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-11-28T16:53:42.225 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/virtio_scsi.ko 2023-11-28T16:53:42.226 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/vmw_pvscsi.ko 2023-11-28T16:53:42.226 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/wd719x.ko 2023-11-28T16:53:42.226 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb 2023-11-28T16:53:42.226 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb/ssb.ko 2023-11-28T16:53:42.226 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target 2023-11-28T16:53:42.226 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi 2023-11-28T16:53:42.226 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-11-28T16:53:42.226 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback 2023-11-28T16:53:42.226 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback/tcm_loop.ko 2023-11-28T16:53:42.226 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_file.ko 2023-11-28T16:53:42.226 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_iblock.ko 2023-11-28T16:53:42.226 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_mod.ko 2023-11-28T16:53:42.227 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_pscsi.ko 2023-11-28T16:53:42.227 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_user.ko 2023-11-28T16:53:42.227 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc 2023-11-28T16:53:42.227 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-11-28T16:53:42.227 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal 2023-11-28T16:53:42.227 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel 2023-11-28T16:53:42.227 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal 2023-11-28T16:53:42.227 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-11-28T16:53:42.227 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-11-28T16:53:42.227 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-11-28T16:53:42.228 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-11-28T16:53:42.228 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-11-28T16:53:42.228 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-11-28T16:53:42.228 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-11-28T16:53:42.228 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-11-28T16:53:42.228 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-11-28T16:53:42.228 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-11-28T16:53:42.228 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-11-28T16:53:42.228 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-11-28T16:53:42.228 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-11-28T16:53:42.228 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-11-28T16:53:42.228 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-11-28T16:53:42.229 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty 2023-11-28T16:53:42.229 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless 2023-11-28T16:53:42.229 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-11-28T16:53:42.229 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_gsm.ko 2023-11-28T16:53:42.229 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_hdlc.ko 2023-11-28T16:53:42.229 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/nozomi.ko 2023-11-28T16:53:42.229 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial 2023-11-28T16:53:42.229 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250 2023-11-28T16:53:42.229 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-11-28T16:53:42.229 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/arc_uart.ko 2023-11-28T16:53:42.229 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm 2023-11-28T16:53:42.229 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm/jsm.ko 2023-11-28T16:53:42.229 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs 2023-11-28T16:53:42.230 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core 2023-11-28T16:53:42.230 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core/ufshcd-core.ko 2023-11-28T16:53:42.230 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host 2023-11-28T16:53:42.230 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-11-28T16:53:42.230 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio 2023-11-28T16:53:42.230 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio.ko 2023-11-28T16:53:42.230 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_aec.ko 2023-11-28T16:53:42.230 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_cif.ko 2023-11-28T16:53:42.230 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_pci_generic.ko 2023-11-28T16:53:42.230 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_sercos3.ko 2023-11-28T16:53:42.230 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb 2023-11-28T16:53:42.230 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class 2023-11-28T16:53:42.231 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-acm.ko 2023-11-28T16:53:42.231 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-wdm.ko 2023-11-28T16:53:42.231 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usblp.ko 2023-11-28T16:53:42.231 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usbtmc.ko 2023-11-28T16:53:42.231 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host 2023-11-28T16:53:42.231 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/sl811-hcd.ko 2023-11-28T16:53:42.231 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image 2023-11-28T16:53:42.231 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/mdc800.ko 2023-11-28T16:53:42.231 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/microtek.ko 2023-11-28T16:53:42.231 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc 2023-11-28T16:53:42.231 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/adutux.ko 2023-11-28T16:53:42.231 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/appledisplay.ko 2023-11-28T16:53:42.231 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi26.ko 2023-11-28T16:53:42.232 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi62.ko 2023-11-28T16:53:42.232 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ezusb.ko 2023-11-28T16:53:42.232 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/idmouse.ko 2023-11-28T16:53:42.232 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/iowarrior.ko 2023-11-28T16:53:42.232 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/isight_firmware.ko 2023-11-28T16:53:42.232 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ldusb.ko 2023-11-28T16:53:42.232 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/legousbtower.ko 2023-11-28T16:53:42.232 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga 2023-11-28T16:53:42.232 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-11-28T16:53:42.232 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/trancevibrator.ko 2023-11-28T16:53:42.232 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usb3503.ko 2023-11-28T16:53:42.232 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usblcd.ko 2023-11-28T16:53:42.233 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usbsevseg.ko 2023-11-28T16:53:42.233 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/uss720.ko 2023-11-28T16:53:42.233 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/yurex.ko 2023-11-28T16:53:42.233 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy 2023-11-28T16:53:42.233 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-generic.ko 2023-11-28T16:53:42.233 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial 2023-11-28T16:53:42.233 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/aircable.ko 2023-11-28T16:53:42.233 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ark3116.ko 2023-11-28T16:53:42.233 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/belkin_sa.ko 2023-11-28T16:53:42.233 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ch341.ko 2023-11-28T16:53:42.233 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cp210x.ko 2023-11-28T16:53:42.233 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cyberjack.ko 2023-11-28T16:53:42.233 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cypress_m8.ko 2023-11-28T16:53:42.234 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/digi_acceleport.ko 2023-11-28T16:53:42.234 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/empeg.ko 2023-11-28T16:53:42.234 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ftdi_sio.ko 2023-11-28T16:53:42.234 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/garmin_gps.ko 2023-11-28T16:53:42.234 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_edgeport.ko 2023-11-28T16:53:42.234 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_ti.ko 2023-11-28T16:53:42.234 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipaq.ko 2023-11-28T16:53:42.234 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipw.ko 2023-11-28T16:53:42.234 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ir-usb.ko 2023-11-28T16:53:42.234 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-11-28T16:53:42.234 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan.ko 2023-11-28T16:53:42.234 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan_pda.ko 2023-11-28T16:53:42.235 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kl5kusb105.ko 2023-11-28T16:53:42.235 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kobil_sct.ko 2023-11-28T16:53:42.235 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mct_u232.ko 2023-11-28T16:53:42.235 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7720.ko 2023-11-28T16:53:42.235 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7840.ko 2023-11-28T16:53:42.235 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/navman.ko 2023-11-28T16:53:42.235 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/omninet.ko 2023-11-28T16:53:42.235 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/opticon.ko 2023-11-28T16:53:42.235 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/option.ko 2023-11-28T16:53:42.235 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/oti6858.ko 2023-11-28T16:53:42.235 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/pl2303.ko 2023-11-28T16:53:42.235 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcaux.ko 2023-11-28T16:53:42.235 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcserial.ko 2023-11-28T16:53:42.236 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/quatech2.ko 2023-11-28T16:53:42.236 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/safe_serial.ko 2023-11-28T16:53:42.236 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/sierra.ko 2023-11-28T16:53:42.236 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/spcp8x5.ko 2023-11-28T16:53:42.236 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ssu100.ko 2023-11-28T16:53:42.236 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/symbolserial.ko 2023-11-28T16:53:42.236 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-11-28T16:53:42.236 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-11-28T16:53:42.236 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_debug.ko 2023-11-28T16:53:42.236 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_wwan.ko 2023-11-28T16:53:42.236 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/visor.ko 2023-11-28T16:53:42.236 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/whiteheat.ko 2023-11-28T16:53:42.237 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/xsens_mt.ko 2023-11-28T16:53:42.237 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage 2023-11-28T16:53:42.237 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/uas.ko 2023-11-28T16:53:42.237 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-alauda.ko 2023-11-28T16:53:42.237 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-cypress.ko 2023-11-28T16:53:42.237 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-datafab.ko 2023-11-28T16:53:42.237 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-11-28T16:53:42.237 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-freecom.ko 2023-11-28T16:53:42.237 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-isd200.ko 2023-11-28T16:53:42.237 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-11-28T16:53:42.237 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-karma.ko 2023-11-28T16:53:42.237 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-onetouch.ko 2023-11-28T16:53:42.237 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-realtek.ko 2023-11-28T16:53:42.238 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr09.ko 2023-11-28T16:53:42.238 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr55.ko 2023-11-28T16:53:42.238 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-usbat.ko 2023-11-28T16:53:42.238 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/usb-storage.ko 2023-11-28T16:53:42.238 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip 2023-11-28T16:53:42.238 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-core.ko 2023-11-28T16:53:42.238 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-host.ko 2023-11-28T16:53:42.238 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-11-28T16:53:42.238 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio 2023-11-28T16:53:42.238 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci 2023-11-28T16:53:42.238 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-11-28T16:53:42.238 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci.ko 2023-11-28T16:53:42.239 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio.ko 2023-11-28T16:53:42.239 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-11-28T16:53:42.239 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost 2023-11-28T16:53:42.239 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost.ko 2023-11-28T16:53:42.239 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_iotlb.ko 2023-11-28T16:53:42.239 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_net.ko 2023-11-28T16:53:42.239 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_scsi.ko 2023-11-28T16:53:42.239 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video 2023-11-28T16:53:42.239 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight 2023-11-28T16:53:42.239 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/apple_bl.ko 2023-11-28T16:53:42.239 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lcd.ko 2023-11-28T16:53:42.239 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/platform_lcd.ko 2023-11-28T16:53:42.239 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio 2023-11-28T16:53:42.240 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio.ko 2023-11-28T16:53:42.240 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_balloon.ko 2023-11-28T16:53:42.240 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_mmio.ko 2023-11-28T16:53:42.240 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci.ko 2023-11-28T16:53:42.240 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-11-28T16:53:42.240 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-11-28T16:53:42.240 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_ring.ko 2023-11-28T16:53:42.240 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1 2023-11-28T16:53:42.240 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters 2023-11-28T16:53:42.240 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2482.ko 2023-11-28T16:53:42.240 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2490.ko 2023-11-28T16:53:42.240 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves 2023-11-28T16:53:42.241 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-11-28T16:53:42.241 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-11-28T16:53:42.241 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-11-28T16:53:42.241 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-11-28T16:53:42.241 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-11-28T16:53:42.241 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-11-28T16:53:42.241 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-11-28T16:53:42.241 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-11-28T16:53:42.241 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-11-28T16:53:42.241 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_smem.ko 2023-11-28T16:53:42.241 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_therm.ko 2023-11-28T16:53:42.241 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/wire.ko 2023-11-28T16:53:42.241 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog 2023-11-28T16:53:42.242 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim1535_wdt.ko 2023-11-28T16:53:42.242 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim7101_wdt.ko 2023-11-28T16:53:42.242 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/f71808e_wdt.ko 2023-11-28T16:53:42.242 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/hpwdt.ko 2023-11-28T16:53:42.242 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/i6300esb.ko 2023-11-28T16:53:42.242 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-11-28T16:53:42.242 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_wdt.ko 2023-11-28T16:53:42.242 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ib700wdt.ko 2023-11-28T16:53:42.242 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ibmasr.ko 2023-11-28T16:53:42.242 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-11-28T16:53:42.242 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it8712f_wdt.ko 2023-11-28T16:53:42.242 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it87_wdt.ko 2023-11-28T16:53:42.243 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/machzwd.ko 2023-11-28T16:53:42.243 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/nv_tco.ko 2023-11-28T16:53:42.243 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_pci.ko 2023-11-28T16:53:42.243 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_usb.ko 2023-11-28T16:53:42.243 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-11-28T16:53:42.243 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sch311x_wdt.ko 2023-11-28T16:53:42.243 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/softdog.ko 2023-11-28T16:53:42.243 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sp5100_tco.ko 2023-11-28T16:53:42.243 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/via_wdt.ko 2023-11-28T16:53:42.243 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-11-28T16:53:42.243 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83877f_wdt.ko 2023-11-28T16:53:42.243 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83977f_wdt.ko 2023-11-28T16:53:42.244 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wdt_pci.ko 2023-11-28T16:53:42.244 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/xen_wdt.ko 2023-11-28T16:53:42.244 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen 2023-11-28T16:53:42.244 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-acpi-processor.ko 2023-11-28T16:53:42.244 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-evtchn.ko 2023-11-28T16:53:42.244 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntalloc.ko 2023-11-28T16:53:42.244 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntdev.ko 2023-11-28T16:53:42.244 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback 2023-11-28T16:53:42.244 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-11-28T16:53:42.244 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-privcmd.ko 2023-11-28T16:53:42.244 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs 2023-11-28T16:53:42.244 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs/xenfs.ko 2023-11-28T16:53:42.245 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs 2023-11-28T16:53:42.245 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p 2023-11-28T16:53:42.245 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p/9p.ko 2023-11-28T16:53:42.245 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs 2023-11-28T16:53:42.245 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs/affs.ko 2023-11-28T16:53:42.245 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs 2023-11-28T16:53:42.245 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs/befs.ko 2023-11-28T16:53:42.245 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/binfmt_misc.ko 2023-11-28T16:53:42.245 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs 2023-11-28T16:53:42.245 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs/btrfs.ko 2023-11-28T16:53:42.245 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles 2023-11-28T16:53:42.245 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles/cachefiles.ko 2023-11-28T16:53:42.245 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph 2023-11-28T16:53:42.246 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph/ceph.ko 2023-11-28T16:53:42.246 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda 2023-11-28T16:53:42.246 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda/coda.ko 2023-11-28T16:53:42.246 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs 2023-11-28T16:53:42.246 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs/cramfs.ko 2023-11-28T16:53:42.246 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm 2023-11-28T16:53:42.246 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm/dlm.ko 2023-11-28T16:53:42.246 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs 2023-11-28T16:53:42.246 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs/ecryptfs.ko 2023-11-28T16:53:42.246 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs 2023-11-28T16:53:42.246 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs/f2fs.ko 2023-11-28T16:53:42.246 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat 2023-11-28T16:53:42.246 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/fat.ko 2023-11-28T16:53:42.247 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/msdos.ko 2023-11-28T16:53:42.247 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/vfat.ko 2023-11-28T16:53:42.247 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache 2023-11-28T16:53:42.247 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache/fscache.ko 2023-11-28T16:53:42.247 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse 2023-11-28T16:53:42.247 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/cuse.ko 2023-11-28T16:53:42.247 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/fuse.ko 2023-11-28T16:53:42.247 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2 2023-11-28T16:53:42.247 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2/gfs2.ko 2023-11-28T16:53:42.247 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs 2023-11-28T16:53:42.247 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs/hfs.ko 2023-11-28T16:53:42.247 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus 2023-11-28T16:53:42.248 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus/hfsplus.ko 2023-11-28T16:53:42.248 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs 2023-11-28T16:53:42.248 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs/isofs.ko 2023-11-28T16:53:42.248 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs 2023-11-28T16:53:42.248 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs/jfs.ko 2023-11-28T16:53:42.248 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd 2023-11-28T16:53:42.248 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd/lockd.ko 2023-11-28T16:53:42.248 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix 2023-11-28T16:53:42.248 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix/minix.ko 2023-11-28T16:53:42.248 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs 2023-11-28T16:53:42.248 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs/netfs.ko 2023-11-28T16:53:42.248 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs 2023-11-28T16:53:42.248 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout 2023-11-28T16:53:42.249 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-11-28T16:53:42.249 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout 2023-11-28T16:53:42.249 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-11-28T16:53:42.249 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout 2023-11-28T16:53:42.249 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-11-28T16:53:42.249 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfs.ko 2023-11-28T16:53:42.249 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv3.ko 2023-11-28T16:53:42.249 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv4.ko 2023-11-28T16:53:42.249 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common 2023-11-28T16:53:42.249 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/grace.ko 2023-11-28T16:53:42.249 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/nfs_acl.ko 2023-11-28T16:53:42.249 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd 2023-11-28T16:53:42.249 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd/nfsd.ko 2023-11-28T16:53:42.249 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2 2023-11-28T16:53:42.250 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2/nilfs2.ko 2023-11-28T16:53:42.250 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls 2023-11-28T16:53:42.250 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-celtic.ko 2023-11-28T16:53:42.250 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-centeuro.ko 2023-11-28T16:53:42.250 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-croatian.ko 2023-11-28T16:53:42.250 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-cyrillic.ko 2023-11-28T16:53:42.250 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-gaelic.ko 2023-11-28T16:53:42.250 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-greek.ko 2023-11-28T16:53:42.250 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-iceland.ko 2023-11-28T16:53:42.250 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-inuit.ko 2023-11-28T16:53:42.250 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-roman.ko 2023-11-28T16:53:42.250 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-romanian.ko 2023-11-28T16:53:42.250 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-turkish.ko 2023-11-28T16:53:42.251 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1250.ko 2023-11-28T16:53:42.251 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1251.ko 2023-11-28T16:53:42.251 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1255.ko 2023-11-28T16:53:42.251 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp737.ko 2023-11-28T16:53:42.251 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp775.ko 2023-11-28T16:53:42.251 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp850.ko 2023-11-28T16:53:42.251 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp852.ko 2023-11-28T16:53:42.251 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp855.ko 2023-11-28T16:53:42.251 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp857.ko 2023-11-28T16:53:42.251 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp860.ko 2023-11-28T16:53:42.251 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp861.ko 2023-11-28T16:53:42.251 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp862.ko 2023-11-28T16:53:42.252 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp863.ko 2023-11-28T16:53:42.252 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp864.ko 2023-11-28T16:53:42.252 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp865.ko 2023-11-28T16:53:42.252 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp866.ko 2023-11-28T16:53:42.252 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp869.ko 2023-11-28T16:53:42.252 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp874.ko 2023-11-28T16:53:42.252 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp932.ko 2023-11-28T16:53:42.252 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp936.ko 2023-11-28T16:53:42.252 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp949.ko 2023-11-28T16:53:42.252 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp950.ko 2023-11-28T16:53:42.252 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_euc-jp.ko 2023-11-28T16:53:42.252 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-1.ko 2023-11-28T16:53:42.252 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-13.ko 2023-11-28T16:53:42.253 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-14.ko 2023-11-28T16:53:42.253 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-15.ko 2023-11-28T16:53:42.253 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-2.ko 2023-11-28T16:53:42.253 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-3.ko 2023-11-28T16:53:42.253 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-4.ko 2023-11-28T16:53:42.253 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-5.ko 2023-11-28T16:53:42.253 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-6.ko 2023-11-28T16:53:42.253 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-7.ko 2023-11-28T16:53:42.253 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-9.ko 2023-11-28T16:53:42.253 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-r.ko 2023-11-28T16:53:42.253 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-ru.ko 2023-11-28T16:53:42.253 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-u.ko 2023-11-28T16:53:42.254 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_ucs2_utils.ko 2023-11-28T16:53:42.254 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_utf8.ko 2023-11-28T16:53:42.254 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2 2023-11-28T16:53:42.254 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster 2023-11-28T16:53:42.254 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-11-28T16:53:42.254 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm 2023-11-28T16:53:42.254 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-11-28T16:53:42.254 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs 2023-11-28T16:53:42.254 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-11-28T16:53:42.254 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2.ko 2023-11-28T16:53:42.254 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-11-28T16:53:42.254 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-11-28T16:53:42.255 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-11-28T16:53:42.255 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs 2023-11-28T16:53:42.255 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs/overlay.ko 2023-11-28T16:53:42.255 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore 2023-11-28T16:53:42.255 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore/ramoops.ko 2023-11-28T16:53:42.255 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs 2023-11-28T16:53:42.255 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs/reiserfs.ko 2023-11-28T16:53:42.255 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs 2023-11-28T16:53:42.255 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs/romfs.ko 2023-11-28T16:53:42.255 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb 2023-11-28T16:53:42.255 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client 2023-11-28T16:53:42.255 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client/cifs.ko 2023-11-28T16:53:42.256 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common 2023-11-28T16:53:42.256 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_arc4.ko 2023-11-28T16:53:42.256 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_md4.ko 2023-11-28T16:53:42.256 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs 2023-11-28T16:53:42.256 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs/squashfs.ko 2023-11-28T16:53:42.256 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv 2023-11-28T16:53:42.256 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv/sysv.ko 2023-11-28T16:53:42.256 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf 2023-11-28T16:53:42.256 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf/udf.ko 2023-11-28T16:53:42.256 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs 2023-11-28T16:53:42.256 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs/ufs.ko 2023-11-28T16:53:42.256 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs 2023-11-28T16:53:42.256 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs/xfs.ko 2023-11-28T16:53:42.257 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel 2023-11-28T16:53:42.257 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu 2023-11-28T16:53:42.257 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu/rcutorture.ko 2023-11-28T16:53:42.257 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/torture.ko 2023-11-28T16:53:42.257 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace 2023-11-28T16:53:42.257 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-11-28T16:53:42.257 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib 2023-11-28T16:53:42.257 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/asn1_encoder.ko 2023-11-28T16:53:42.257 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc-itu-t.ko 2023-11-28T16:53:42.257 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc8.ko 2023-11-28T16:53:42.257 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto 2023-11-28T16:53:42.257 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libarc4.ko 2023-11-28T16:53:42.258 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libdes.ko 2023-11-28T16:53:42.258 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/libcrc32c.ko 2023-11-28T16:53:42.258 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lru_cache.ko 2023-11-28T16:53:42.258 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4 2023-11-28T16:53:42.258 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4_compress.ko 2023-11-28T16:53:42.258 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4hc_compress.ko 2023-11-28T16:53:42.258 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math 2023-11-28T16:53:42.258 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math/cordic.ko 2023-11-28T16:53:42.258 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6 2023-11-28T16:53:42.258 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6/raid6_pq.ko 2023-11-28T16:53:42.258 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon 2023-11-28T16:53:42.258 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon/reed_solomon.ko 2023-11-28T16:53:42.258 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_bm.ko 2023-11-28T16:53:42.259 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_fsm.ko 2023-11-28T16:53:42.259 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_kmp.ko 2023-11-28T16:53:42.259 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd 2023-11-28T16:53:42.259 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd/zstd_compress.ko 2023-11-28T16:53:42.259 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm 2023-11-28T16:53:42.259 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm/hwpoison-inject.ko 2023-11-28T16:53:42.259 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net 2023-11-28T16:53:42.259 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan 2023-11-28T16:53:42.259 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/6lowpan.ko 2023-11-28T16:53:42.259 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_dest.ko 2023-11-28T16:53:42.259 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_fragment.ko 2023-11-28T16:53:42.259 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_hop.ko 2023-11-28T16:53:42.260 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_ipv6.ko 2023-11-28T16:53:42.260 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_mobility.ko 2023-11-28T16:53:42.260 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_routing.ko 2023-11-28T16:53:42.260 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_udp.ko 2023-11-28T16:53:42.260 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802 2023-11-28T16:53:42.260 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/garp.ko 2023-11-28T16:53:42.260 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/mrp.ko 2023-11-28T16:53:42.260 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/p8022.ko 2023-11-28T16:53:42.260 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/psnap.ko 2023-11-28T16:53:42.260 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/stp.ko 2023-11-28T16:53:42.260 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q 2023-11-28T16:53:42.260 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q/8021q.ko 2023-11-28T16:53:42.260 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p 2023-11-28T16:53:42.261 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet.ko 2023-11-28T16:53:42.261 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_fd.ko 2023-11-28T16:53:42.261 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_rdma.ko 2023-11-28T16:53:42.261 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_virtio.ko 2023-11-28T16:53:42.261 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv 2023-11-28T16:53:42.261 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv/batman-adv.ko 2023-11-28T16:53:42.261 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge 2023-11-28T16:53:42.261 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/br_netfilter.ko 2023-11-28T16:53:42.261 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/bridge.ko 2023-11-28T16:53:42.261 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter 2023-11-28T16:53:42.261 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-11-28T16:53:42.261 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_among.ko 2023-11-28T16:53:42.261 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arp.ko 2023-11-28T16:53:42.261 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-11-28T16:53:42.262 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-11-28T16:53:42.262 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip.ko 2023-11-28T16:53:42.262 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-11-28T16:53:42.262 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_limit.ko 2023-11-28T16:53:42.262 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_log.ko 2023-11-28T16:53:42.262 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark.ko 2023-11-28T16:53:42.262 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-11-28T16:53:42.262 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-11-28T16:53:42.262 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-11-28T16:53:42.262 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-11-28T16:53:42.262 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_snat.ko 2023-11-28T16:53:42.262 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_stp.ko 2023-11-28T16:53:42.262 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-11-28T16:53:42.263 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-11-28T16:53:42.263 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-11-28T16:53:42.263 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-11-28T16:53:42.263 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtables.ko 2023-11-28T16:53:42.263 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-11-28T16:53:42.263 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-11-28T16:53:42.263 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-11-28T16:53:42.263 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph 2023-11-28T16:53:42.263 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph/libceph.ko 2023-11-28T16:53:42.263 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core 2023-11-28T16:53:42.263 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/failover.ko 2023-11-28T16:53:42.263 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/pktgen.ko 2023-11-28T16:53:42.263 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp 2023-11-28T16:53:42.264 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp.ko 2023-11-28T16:53:42.264 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_diag.ko 2023-11-28T16:53:42.264 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv4.ko 2023-11-28T16:53:42.264 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv6.ko 2023-11-28T16:53:42.264 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver 2023-11-28T16:53:42.264 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver/dns_resolver.ko 2023-11-28T16:53:42.264 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa 2023-11-28T16:53:42.264 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/dsa_core.ko 2023-11-28T16:53:42.264 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_brcm.ko 2023-11-28T16:53:42.264 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_dsa.ko 2023-11-28T16:53:42.264 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_none.ko 2023-11-28T16:53:42.264 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_trailer.ko 2023-11-28T16:53:42.264 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154 2023-11-28T16:53:42.265 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan 2023-11-28T16:53:42.265 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-11-28T16:53:42.265 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154.ko 2023-11-28T16:53:42.265 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154_socket.ko 2023-11-28T16:53:42.265 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4 2023-11-28T16:53:42.265 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ah4.ko 2023-11-28T16:53:42.265 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/esp4.ko 2023-11-28T16:53:42.265 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/fou.ko 2023-11-28T16:53:42.265 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/gre.ko 2023-11-28T16:53:42.265 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/inet_diag.ko 2023-11-28T16:53:42.265 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_gre.ko 2023-11-28T16:53:42.265 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_tunnel.ko 2023-11-28T16:53:42.266 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_vti.ko 2023-11-28T16:53:42.266 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipcomp.ko 2023-11-28T16:53:42.266 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipip.ko 2023-11-28T16:53:42.266 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter 2023-11-28T16:53:42.266 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arp_tables.ko 2023-11-28T16:53:42.266 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-11-28T16:53:42.266 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-11-28T16:53:42.266 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-11-28T16:53:42.266 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-11-28T16:53:42.266 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-11-28T16:53:42.266 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-11-28T16:53:42.266 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-11-28T16:53:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-11-28T16:53:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-11-28T16:53:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_security.ko 2023-11-28T16:53:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-11-28T16:53:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-11-28T16:53:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-11-28T16:53:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-11-28T16:53:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-11-28T16:53:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-11-28T16:53:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-11-28T16:53:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-11-28T16:53:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-11-28T16:53:42.267 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-11-28T16:53:42.268 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_bic.ko 2023-11-28T16:53:42.268 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_dctcp.ko 2023-11-28T16:53:42.268 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_diag.ko 2023-11-28T16:53:42.268 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_highspeed.ko 2023-11-28T16:53:42.268 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_htcp.ko 2023-11-28T16:53:42.268 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_hybla.ko 2023-11-28T16:53:42.268 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_illinois.ko 2023-11-28T16:53:42.268 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_lp.ko 2023-11-28T16:53:42.268 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_scalable.ko 2023-11-28T16:53:42.268 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_vegas.ko 2023-11-28T16:53:42.268 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_veno.ko 2023-11-28T16:53:42.268 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_westwood.ko 2023-11-28T16:53:42.269 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_yeah.ko 2023-11-28T16:53:42.269 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tunnel4.ko 2023-11-28T16:53:42.269 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_diag.ko 2023-11-28T16:53:42.269 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_tunnel.ko 2023-11-28T16:53:42.269 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/xfrm4_tunnel.ko 2023-11-28T16:53:42.269 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6 2023-11-28T16:53:42.269 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ah6.ko 2023-11-28T16:53:42.269 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/esp6.ko 2023-11-28T16:53:42.269 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/fou6.ko 2023-11-28T16:53:42.269 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_tunnel.ko 2023-11-28T16:53:42.269 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-11-28T16:53:42.269 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_vti.ko 2023-11-28T16:53:42.269 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ipcomp6.ko 2023-11-28T16:53:42.270 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter 2023-11-28T16:53:42.270 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-11-28T16:53:42.270 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-11-28T16:53:42.270 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-11-28T16:53:42.270 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-11-28T16:53:42.270 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-11-28T16:53:42.270 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-11-28T16:53:42.270 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-11-28T16:53:42.270 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-11-28T16:53:42.270 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-11-28T16:53:42.270 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-11-28T16:53:42.270 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-11-28T16:53:42.271 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-11-28T16:53:42.271 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-11-28T16:53:42.271 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-11-28T16:53:42.271 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-11-28T16:53:42.271 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-11-28T16:53:42.271 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-11-28T16:53:42.271 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-11-28T16:53:42.271 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-11-28T16:53:42.271 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-11-28T16:53:42.271 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-11-28T16:53:42.271 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-11-28T16:53:42.271 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-11-28T16:53:42.272 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-11-28T16:53:42.272 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/sit.ko 2023-11-28T16:53:42.272 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/tunnel6.ko 2023-11-28T16:53:42.272 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/xfrm6_tunnel.ko 2023-11-28T16:53:42.272 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key 2023-11-28T16:53:42.272 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key/af_key.ko 2023-11-28T16:53:42.272 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp 2023-11-28T16:53:42.272 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_core.ko 2023-11-28T16:53:42.272 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_debugfs.ko 2023-11-28T16:53:42.272 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_eth.ko 2023-11-28T16:53:42.272 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip.ko 2023-11-28T16:53:42.272 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip6.ko 2023-11-28T16:53:42.273 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_netlink.ko 2023-11-28T16:53:42.273 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ppp.ko 2023-11-28T16:53:42.273 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc 2023-11-28T16:53:42.273 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc/llc.ko 2023-11-28T16:53:42.273 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154 2023-11-28T16:53:42.273 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154/mac802154.ko 2023-11-28T16:53:42.273 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls 2023-11-28T16:53:42.273 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls/mpls_gso.ko 2023-11-28T16:53:42.273 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter 2023-11-28T16:53:42.273 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset 2023-11-28T16:53:42.273 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set.ko 2023-11-28T16:53:42.273 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-11-28T16:53:42.273 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-11-28T16:53:42.274 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-11-28T16:53:42.274 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-11-28T16:53:42.274 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-11-28T16:53:42.274 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-11-28T16:53:42.274 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-11-28T16:53:42.274 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-11-28T16:53:42.274 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-11-28T16:53:42.274 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-11-28T16:53:42.274 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-11-28T16:53:42.274 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-11-28T16:53:42.274 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-11-28T16:53:42.274 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-11-28T16:53:42.275 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-11-28T16:53:42.275 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs 2023-11-28T16:53:42.275 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs.ko 2023-11-28T16:53:42.275 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-11-28T16:53:42.275 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-11-28T16:53:42.275 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-11-28T16:53:42.275 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-11-28T16:53:42.275 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-11-28T16:53:42.275 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-11-28T16:53:42.275 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-11-28T16:53:42.275 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-11-28T16:53:42.275 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-11-28T16:53:42.275 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-11-28T16:53:42.276 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-11-28T16:53:42.276 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-11-28T16:53:42.276 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-11-28T16:53:42.276 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conncount.ko 2023-11-28T16:53:42.276 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack.ko 2023-11-28T16:53:42.276 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-11-28T16:53:42.276 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-11-28T16:53:42.276 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-11-28T16:53:42.276 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_h323.ko 2023-11-28T16:53:42.276 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_irc.ko 2023-11-28T16:53:42.276 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-11-28T16:53:42.276 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-11-28T16:53:42.276 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-11-28T16:53:42.277 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sane.ko 2023-11-28T16:53:42.277 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sip.ko 2023-11-28T16:53:42.277 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-11-28T16:53:42.277 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-11-28T16:53:42.277 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_dup_netdev.ko 2023-11-28T16:53:42.277 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table.ko 2023-11-28T16:53:42.277 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table_inet.ko 2023-11-28T16:53:42.277 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_log_syslog.ko 2023-11-28T16:53:42.277 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat.ko 2023-11-28T16:53:42.277 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_amanda.ko 2023-11-28T16:53:42.277 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_ftp.ko 2023-11-28T16:53:42.277 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_irc.ko 2023-11-28T16:53:42.277 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_sip.ko 2023-11-28T16:53:42.277 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_tftp.ko 2023-11-28T16:53:42.278 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_synproxy_core.ko 2023-11-28T16:53:42.278 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_tables.ko 2023-11-28T16:53:42.278 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink.ko 2023-11-28T16:53:42.278 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_acct.ko 2023-11-28T16:53:42.278 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_log.ko 2023-11-28T16:53:42.278 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_osf.ko 2023-11-28T16:53:42.278 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_queue.ko 2023-11-28T16:53:42.278 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_chain_nat.ko 2023-11-28T16:53:42.278 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_compat.ko 2023-11-28T16:53:42.278 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_ct.ko 2023-11-28T16:53:42.278 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_dup_netdev.ko 2023-11-28T16:53:42.278 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fib.ko 2023-11-28T16:53:42.278 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fwd_netdev.ko 2023-11-28T16:53:42.279 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_hash.ko 2023-11-28T16:53:42.279 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_limit.ko 2023-11-28T16:53:42.279 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_log.ko 2023-11-28T16:53:42.279 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_masq.ko 2023-11-28T16:53:42.279 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_nat.ko 2023-11-28T16:53:42.279 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_numgen.ko 2023-11-28T16:53:42.279 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_queue.ko 2023-11-28T16:53:42.279 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_quota.ko 2023-11-28T16:53:42.279 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_redir.ko 2023-11-28T16:53:42.279 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject.ko 2023-11-28T16:53:42.279 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject_inet.ko 2023-11-28T16:53:42.279 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_socket.ko 2023-11-28T16:53:42.279 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_tunnel.ko 2023-11-28T16:53:42.280 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_AUDIT.ko 2023-11-28T16:53:42.280 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CHECKSUM.ko 2023-11-28T16:53:42.280 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CLASSIFY.ko 2023-11-28T16:53:42.280 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-11-28T16:53:42.280 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CT.ko 2023-11-28T16:53:42.280 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_DSCP.ko 2023-11-28T16:53:42.280 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HL.ko 2023-11-28T16:53:42.280 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HMARK.ko 2023-11-28T16:53:42.280 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_IDLETIMER.ko 2023-11-28T16:53:42.280 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LED.ko 2023-11-28T16:53:42.280 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LOG.ko 2023-11-28T16:53:42.280 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_MASQUERADE.ko 2023-11-28T16:53:42.280 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NETMAP.ko 2023-11-28T16:53:42.280 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFLOG.ko 2023-11-28T16:53:42.281 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFQUEUE.ko 2023-11-28T16:53:42.281 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_RATEEST.ko 2023-11-28T16:53:42.281 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_REDIRECT.ko 2023-11-28T16:53:42.281 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_SECMARK.ko 2023-11-28T16:53:42.281 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPMSS.ko 2023-11-28T16:53:42.281 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-11-28T16:53:42.281 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TEE.ko 2023-11-28T16:53:42.281 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TPROXY.ko 2023-11-28T16:53:42.281 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TRACE.ko 2023-11-28T16:53:42.281 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_addrtype.ko 2023-11-28T16:53:42.281 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_bpf.ko 2023-11-28T16:53:42.281 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cgroup.ko 2023-11-28T16:53:42.282 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cluster.ko 2023-11-28T16:53:42.282 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_comment.ko 2023-11-28T16:53:42.282 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connbytes.ko 2023-11-28T16:53:42.282 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlabel.ko 2023-11-28T16:53:42.282 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlimit.ko 2023-11-28T16:53:42.282 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connmark.ko 2023-11-28T16:53:42.282 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_conntrack.ko 2023-11-28T16:53:42.282 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cpu.ko 2023-11-28T16:53:42.282 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dccp.ko 2023-11-28T16:53:42.282 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_devgroup.ko 2023-11-28T16:53:42.282 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dscp.ko 2023-11-28T16:53:42.282 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ecn.ko 2023-11-28T16:53:42.283 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_esp.ko 2023-11-28T16:53:42.283 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hashlimit.ko 2023-11-28T16:53:42.283 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_helper.ko 2023-11-28T16:53:42.283 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hl.ko 2023-11-28T16:53:42.283 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipcomp.ko 2023-11-28T16:53:42.283 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_iprange.ko 2023-11-28T16:53:42.283 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipvs.ko 2023-11-28T16:53:42.283 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_l2tp.ko 2023-11-28T16:53:42.283 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_length.ko 2023-11-28T16:53:42.283 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_limit.ko 2023-11-28T16:53:42.283 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mac.ko 2023-11-28T16:53:42.283 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mark.ko 2023-11-28T16:53:42.283 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_multiport.ko 2023-11-28T16:53:42.284 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nat.ko 2023-11-28T16:53:42.284 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nfacct.ko 2023-11-28T16:53:42.284 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_osf.ko 2023-11-28T16:53:42.284 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_owner.ko 2023-11-28T16:53:42.284 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_physdev.ko 2023-11-28T16:53:42.284 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_pkttype.ko 2023-11-28T16:53:42.284 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_policy.ko 2023-11-28T16:53:42.284 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_quota.ko 2023-11-28T16:53:42.284 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_rateest.ko 2023-11-28T16:53:42.284 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_realm.ko 2023-11-28T16:53:42.284 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_recent.ko 2023-11-28T16:53:42.284 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_sctp.ko 2023-11-28T16:53:42.285 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_set.ko 2023-11-28T16:53:42.285 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_socket.ko 2023-11-28T16:53:42.285 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_state.ko 2023-11-28T16:53:42.285 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_statistic.ko 2023-11-28T16:53:42.285 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_string.ko 2023-11-28T16:53:42.285 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_tcpmss.ko 2023-11-28T16:53:42.285 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_time.ko 2023-11-28T16:53:42.285 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_u32.ko 2023-11-28T16:53:42.285 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink 2023-11-28T16:53:42.285 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink/netlink_diag.ko 2023-11-28T16:53:42.285 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh 2023-11-28T16:53:42.285 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh/nsh.ko 2023-11-28T16:53:42.285 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch 2023-11-28T16:53:42.286 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/openvswitch.ko 2023-11-28T16:53:42.286 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-geneve.ko 2023-11-28T16:53:42.286 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-gre.ko 2023-11-28T16:53:42.286 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-vxlan.ko 2023-11-28T16:53:42.286 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet 2023-11-28T16:53:42.286 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet/af_packet_diag.ko 2023-11-28T16:53:42.286 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds 2023-11-28T16:53:42.286 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds.ko 2023-11-28T16:53:42.286 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_rdma.ko 2023-11-28T16:53:42.286 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_tcp.ko 2023-11-28T16:53:42.286 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched 2023-11-28T16:53:42.286 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_csum.ko 2023-11-28T16:53:42.287 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_gact.ko 2023-11-28T16:53:42.287 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_ipt.ko 2023-11-28T16:53:42.287 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_mirred.ko 2023-11-28T16:53:42.287 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_nat.ko 2023-11-28T16:53:42.287 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_pedit.ko 2023-11-28T16:53:42.287 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_police.ko 2023-11-28T16:53:42.287 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_simple.ko 2023-11-28T16:53:42.287 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_skbedit.ko 2023-11-28T16:53:42.287 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_vlan.ko 2023-11-28T16:53:42.287 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_basic.ko 2023-11-28T16:53:42.287 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_bpf.ko 2023-11-28T16:53:42.287 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_flow.ko 2023-11-28T16:53:42.287 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_fw.ko 2023-11-28T16:53:42.288 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_route.ko 2023-11-28T16:53:42.288 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_u32.ko 2023-11-28T16:53:42.288 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_cmp.ko 2023-11-28T16:53:42.288 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_ipset.ko 2023-11-28T16:53:42.288 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_meta.ko 2023-11-28T16:53:42.288 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_nbyte.ko 2023-11-28T16:53:42.288 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_text.ko 2023-11-28T16:53:42.288 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_u32.ko 2023-11-28T16:53:42.288 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_choke.ko 2023-11-28T16:53:42.288 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_codel.ko 2023-11-28T16:53:42.288 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_drr.ko 2023-11-28T16:53:42.288 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_fq.ko 2023-11-28T16:53:42.289 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_gred.ko 2023-11-28T16:53:42.289 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hfsc.ko 2023-11-28T16:53:42.289 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hhf.ko 2023-11-28T16:53:42.289 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_htb.ko 2023-11-28T16:53:42.289 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_ingress.ko 2023-11-28T16:53:42.289 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio.ko 2023-11-28T16:53:42.289 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio_lib.ko 2023-11-28T16:53:42.289 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_multiq.ko 2023-11-28T16:53:42.289 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_netem.ko 2023-11-28T16:53:42.289 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_pie.ko 2023-11-28T16:53:42.289 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_plug.ko 2023-11-28T16:53:42.289 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_prio.ko 2023-11-28T16:53:42.289 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_qfq.ko 2023-11-28T16:53:42.290 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_red.ko 2023-11-28T16:53:42.290 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfb.ko 2023-11-28T16:53:42.290 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfq.ko 2023-11-28T16:53:42.290 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_tbf.ko 2023-11-28T16:53:42.290 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_teql.ko 2023-11-28T16:53:42.290 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp 2023-11-28T16:53:42.290 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp.ko 2023-11-28T16:53:42.290 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp_diag.ko 2023-11-28T16:53:42.290 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc 2023-11-28T16:53:42.290 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss 2023-11-28T16:53:42.290 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-11-28T16:53:42.290 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-11-28T16:53:42.291 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/sunrpc.ko 2023-11-28T16:53:42.291 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma 2023-11-28T16:53:42.291 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-11-28T16:53:42.291 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc 2023-11-28T16:53:42.292 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/diag.ko 2023-11-28T16:53:42.292 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/tipc.ko 2023-11-28T16:53:42.292 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix 2023-11-28T16:53:42.292 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix/unix_diag.ko 2023-11-28T16:53:42.292 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock 2023-11-28T16:53:42.292 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-11-28T16:53:42.292 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-11-28T16:53:42.292 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock.ko 2023-11-28T16:53:42.292 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_diag.ko 2023-11-28T16:53:42.292 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_loopback.ko 2023-11-28T16:53:42.292 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm 2023-11-28T16:53:42.292 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/xfrm_ipcomp.ko 2023-11-28T16:53:42.293 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security 2023-11-28T16:53:42.293 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys 2023-11-28T16:53:42.293 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys 2023-11-28T16:53:42.293 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-11-28T16:53:42.293 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys 2023-11-28T16:53:42.293 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys/trusted.ko 2023-11-28T16:53:42.293 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt 2023-11-28T16:53:42.293 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib 2023-11-28T16:53:42.293 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib/irqbypass.ko 2023-11-28T16:53:42.293 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias 2023-11-28T16:53:42.293 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias.bin 2023-11-28T16:53:42.293 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin 2023-11-28T16:53:42.293 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.alias.bin 2023-11-28T16:53:42.294 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.bin 2023-11-28T16:53:42.294 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.modinfo 2023-11-28T16:53:42.294 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep 2023-11-28T16:53:42.294 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep.bin 2023-11-28T16:53:42.294 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.devname 2023-11-28T16:53:42.294 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.order 2023-11-28T16:53:42.294 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.softdep 2023-11-28T16:53:42.294 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols 2023-11-28T16:53:42.294 INFO:teuthology.orchestra.run.smithi156.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols.bin 2023-11-28T16:53:42.295 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-g8247bbb5b470 2023-11-28T16:53:42.295 DEBUG:teuthology.orchestra.run.smithi156:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-11-28T16:53:42.318 INFO:teuthology.orchestra.run.smithi156.stdout:postinstall scriptlet (using /bin/sh): 2023-11-28T16:53:42.318 INFO:teuthology.orchestra.run.smithi156.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-g8247bbb5b470 -a -r /boot/System.map-6.6.0-g8247bbb5b470 ]; then 2023-11-28T16:53:42.318 INFO:teuthology.orchestra.run.smithi156.stdout:cp /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm 2023-11-28T16:53:42.318 INFO:teuthology.orchestra.run.smithi156.stdout:cp /boot/System.map-6.6.0-g8247bbb5b470 /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T16:53:42.318 INFO:teuthology.orchestra.run.smithi156.stdout:rm -f /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/System.map-6.6.0-g8247bbb5b470 2023-11-28T16:53:42.318 INFO:teuthology.orchestra.run.smithi156.stdout:/sbin/installkernel 6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T16:53:42.318 INFO:teuthology.orchestra.run.smithi156.stdout:rm -f /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T16:53:42.318 INFO:teuthology.orchestra.run.smithi156.stdout:fi 2023-11-28T16:53:42.318 INFO:teuthology.orchestra.run.smithi156.stdout:preuninstall scriptlet (using /bin/sh): 2023-11-28T16:53:42.319 INFO:teuthology.orchestra.run.smithi156.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-11-28T16:53:42.319 INFO:teuthology.orchestra.run.smithi156.stdout:new-kernel-pkg --remove 6.6.0-g8247bbb5b470 --rminitrd --initrdfile=/boot/initramfs-6.6.0-g8247bbb5b470.img 2023-11-28T16:53:42.319 INFO:teuthology.orchestra.run.smithi156.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-11-28T16:53:42.319 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-install remove 6.6.0-g8247bbb5b470 2023-11-28T16:53:42.319 INFO:teuthology.orchestra.run.smithi156.stdout:fi 2023-11-28T16:53:42.319 INFO:teuthology.orchestra.run.smithi156.stdout:postuninstall scriptlet (using /bin/sh): 2023-11-28T16:53:42.319 INFO:teuthology.orchestra.run.smithi156.stdout:if [ -x /sbin/update-bootloader ]; then 2023-11-28T16:53:42.319 INFO:teuthology.orchestra.run.smithi156.stdout:/sbin/update-bootloader --remove 6.6.0-g8247bbb5b470 2023-11-28T16:53:42.319 INFO:teuthology.orchestra.run.smithi156.stdout:fi 2023-11-28T16:53:42.320 DEBUG:teuthology.orchestra.run.smithi156:> sudo rpm -qi grub2-tools 2023-11-28T16:53:42.404 INFO:teuthology.orchestra.run.smithi156.stdout:Name : grub2-tools 2023-11-28T16:53:42.404 INFO:teuthology.orchestra.run.smithi156.stdout:Epoch : 1 2023-11-28T16:53:42.404 INFO:teuthology.orchestra.run.smithi156.stdout:Version : 2.06 2023-11-28T16:53:42.404 INFO:teuthology.orchestra.run.smithi156.stdout:Release : 68.el9 2023-11-28T16:53:42.404 INFO:teuthology.orchestra.run.smithi156.stdout:Architecture: x86_64 2023-11-28T16:53:42.404 INFO:teuthology.orchestra.run.smithi156.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-28T16:53:42.405 INFO:teuthology.orchestra.run.smithi156.stdout:Group : Unspecified 2023-11-28T16:53:42.405 INFO:teuthology.orchestra.run.smithi156.stdout:Size : 8264254 2023-11-28T16:53:42.405 INFO:teuthology.orchestra.run.smithi156.stdout:License : GPLv3+ 2023-11-28T16:53:42.405 INFO:teuthology.orchestra.run.smithi156.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-28T16:53:42.405 INFO:teuthology.orchestra.run.smithi156.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-28T16:53:42.405 INFO:teuthology.orchestra.run.smithi156.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-28T16:53:42.405 INFO:teuthology.orchestra.run.smithi156.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-28T16:53:42.405 INFO:teuthology.orchestra.run.smithi156.stdout:Packager : builder@centos.org 2023-11-28T16:53:42.405 INFO:teuthology.orchestra.run.smithi156.stdout:Vendor : CentOS 2023-11-28T16:53:42.405 INFO:teuthology.orchestra.run.smithi156.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-28T16:53:42.405 INFO:teuthology.orchestra.run.smithi156.stdout:Summary : Support tools for GRUB. 2023-11-28T16:53:42.405 INFO:teuthology.orchestra.run.smithi156.stdout:Description : 2023-11-28T16:53:42.406 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T16:53:42.406 INFO:teuthology.orchestra.run.smithi156.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-28T16:53:42.406 INFO:teuthology.orchestra.run.smithi156.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-28T16:53:42.406 INFO:teuthology.orchestra.run.smithi156.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-28T16:53:42.406 INFO:teuthology.orchestra.run.smithi156.stdout:hardware devices. 2023-11-28T16:53:42.406 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T16:53:42.406 INFO:teuthology.orchestra.run.smithi156.stdout:This subpackage provides tools for support of all platforms. 2023-11-28T16:53:42.407 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-28T16:53:42.407 INFO:teuthology.task.kernel:Updating grub on smithi156 to boot 6.6.0-g8247bbb5b470 2023-11-28T16:53:42.407 DEBUG:teuthology.orchestra.run.smithi156:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-28T16:53:43.059 INFO:teuthology.orchestra.run.smithi156.stderr:Generating grub configuration file ... 2023-11-28T16:53:44.464 INFO:teuthology.orchestra.run.smithi156.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-28T16:53:44.484 INFO:teuthology.orchestra.run.smithi156.stderr:done 2023-11-28T16:53:44.487 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2023-11-28T16:53:44.502 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.EF0zyR0AOM 2023-11-28T16:53:44.502 DEBUG:teuthology.orchestra.run.smithi156:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EF0zyR0AOM 2023-11-28T16:53:44.575 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.EF0zyR0AOM 2023-11-28T16:53:44.715 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.EF0zyR0AOM is 6KB 2023-11-28T16:53:44.766 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.EF0zyR0AOM 2023-11-28T16:53:44.780 DEBUG:teuthology.orchestra.run.smithi156:> sudo /bin/ls /boot/loader/entries || true 2023-11-28T16:53:44.855 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-28T16:53:44.855 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-28T16:53:44.855 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-28T16:53:44.855 INFO:teuthology.orchestra.run.smithi156.stdout:15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470.conf 2023-11-28T16:53:44.856 DEBUG:teuthology.orchestra.run.smithi156:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470 2023-11-28T16:53:45.038 DEBUG:teuthology.orchestra.run.smithi156:> sudo shutdown -r now 2023-11-28T16:53:51.170 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-11-28T16:53:51.171 DEBUG:teuthology.orchestra.run.smithi116:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-11-28T16:53:51.244 INFO:teuthology.orchestra.run.smithi116.stdout:/boot/System.map-6.6.0-g8247bbb5b470 2023-11-28T16:53:51.245 INFO:teuthology.orchestra.run.smithi116.stdout:/boot/config-6.6.0-g8247bbb5b470 2023-11-28T16:53:51.245 INFO:teuthology.orchestra.run.smithi116.stdout:/boot/vmlinuz-6.6.0-g8247bbb5b470 2023-11-28T16:53:51.245 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470 2023-11-28T16:53:51.245 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel 2023-11-28T16:53:51.245 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch 2023-11-28T16:53:51.245 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86 2023-11-28T16:53:51.246 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto 2023-11-28T16:53:51.246 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-11-28T16:53:51.246 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-11-28T16:53:51.246 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-11-28T16:53:51.246 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-11-28T16:53:51.246 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-11-28T16:53:51.246 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-11-28T16:53:51.246 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-11-28T16:53:51.246 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crc32c-intel.ko 2023-11-28T16:53:51.247 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-11-28T16:53:51.247 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-11-28T16:53:51.247 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-11-28T16:53:51.247 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-11-28T16:53:51.247 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-avx2.ko 2023-11-28T16:53:51.247 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-11-28T16:53:51.247 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-11-28T16:53:51.247 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-11-28T16:53:51.247 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-11-28T16:53:51.247 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-11-28T16:53:51.247 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-11-28T16:53:51.248 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-11-28T16:53:51.248 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel 2023-11-28T16:53:51.248 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu 2023-11-28T16:53:51.248 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce 2023-11-28T16:53:51.248 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-11-28T16:53:51.248 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm 2023-11-28T16:53:51.248 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-amd.ko 2023-11-28T16:53:51.248 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm-intel.ko 2023-11-28T16:53:51.248 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/arch/x86/kvm/kvm.ko 2023-11-28T16:53:51.248 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto 2023-11-28T16:53:51.248 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ansi_cprng.ko 2023-11-28T16:53:51.249 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/anubis.ko 2023-11-28T16:53:51.249 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/arc4.ko 2023-11-28T16:53:51.249 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx 2023-11-28T16:53:51.249 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_memcpy.ko 2023-11-28T16:53:51.249 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_pq.ko 2023-11-28T16:53:51.249 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_raid6_recov.ko 2023-11-28T16:53:51.249 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_tx.ko 2023-11-28T16:53:51.249 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/async_xor.ko 2023-11-28T16:53:51.249 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/async_tx/raid6test.ko 2023-11-28T16:53:51.249 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authenc.ko 2023-11-28T16:53:51.249 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/authencesn.ko 2023-11-28T16:53:51.250 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blake2b_generic.ko 2023-11-28T16:53:51.250 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_common.ko 2023-11-28T16:53:51.250 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/blowfish_generic.ko 2023-11-28T16:53:51.251 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/camellia_generic.ko 2023-11-28T16:53:51.251 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast5_generic.ko 2023-11-28T16:53:51.251 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast6_generic.ko 2023-11-28T16:53:51.251 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cast_common.ko 2023-11-28T16:53:51.251 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/ccm.ko 2023-11-28T16:53:51.251 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cmac.ko 2023-11-28T16:53:51.251 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crc32_generic.ko 2023-11-28T16:53:51.251 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_engine.ko 2023-11-28T16:53:51.251 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/crypto_user.ko 2023-11-28T16:53:51.251 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/cts.ko 2023-11-28T16:53:51.251 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/des_generic.ko 2023-11-28T16:53:51.252 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/dh_generic.ko 2023-11-28T16:53:51.252 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/echainiv.ko 2023-11-28T16:53:51.252 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/essiv.ko 2023-11-28T16:53:51.252 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/fcrypt.ko 2023-11-28T16:53:51.252 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/khazad.ko 2023-11-28T16:53:51.252 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4.ko 2023-11-28T16:53:51.252 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/lz4hc.ko 2023-11-28T16:53:51.252 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/md4.ko 2023-11-28T16:53:51.252 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/michael_mic.ko 2023-11-28T16:53:51.252 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcbc.ko 2023-11-28T16:53:51.252 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/pcrypt.ko 2023-11-28T16:53:51.252 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/rmd160.ko 2023-11-28T16:53:51.253 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/seed.ko 2023-11-28T16:53:51.253 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/serpent_generic.ko 2023-11-28T16:53:51.253 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tcrypt.ko 2023-11-28T16:53:51.253 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/tea.ko 2023-11-28T16:53:51.253 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_common.ko 2023-11-28T16:53:51.253 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/twofish_generic.ko 2023-11-28T16:53:51.253 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/vmac.ko 2023-11-28T16:53:51.253 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/wp512.ko 2023-11-28T16:53:51.253 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xcbc.ko 2023-11-28T16:53:51.253 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xor.ko 2023-11-28T16:53:51.253 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/crypto/xxhash_generic.ko 2023-11-28T16:53:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers 2023-11-28T16:53:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi 2023-11-28T16:53:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_ipmi.ko 2023-11-28T16:53:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/acpi_pad.ko 2023-11-28T16:53:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/custom_method.ko 2023-11-28T16:53:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/ec_sys.ko 2023-11-28T16:53:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/platform_profile.ko 2023-11-28T16:53:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbs.ko 2023-11-28T16:53:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/sbshc.ko 2023-11-28T16:53:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/acpi/video.ko 2023-11-28T16:53:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata 2023-11-28T16:53:51.254 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/acard-ahci.ko 2023-11-28T16:53:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ahci_platform.ko 2023-11-28T16:53:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/ata_generic.ko 2023-11-28T16:53:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/libahci_platform.ko 2023-11-28T16:53:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_acpi.ko 2023-11-28T16:53:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ali.ko 2023-11-28T16:53:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_amd.ko 2023-11-28T16:53:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_artop.ko 2023-11-28T16:53:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atiixp.ko 2023-11-28T16:53:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_atp867x.ko 2023-11-28T16:53:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd640.ko 2023-11-28T16:53:51.255 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cmd64x.ko 2023-11-28T16:53:51.256 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_cypress.ko 2023-11-28T16:53:51.256 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_efar.ko 2023-11-28T16:53:51.256 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt366.ko 2023-11-28T16:53:51.256 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt37x.ko 2023-11-28T16:53:51.256 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x2n.ko 2023-11-28T16:53:51.256 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_hpt3x3.ko 2023-11-28T16:53:51.256 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it8213.ko 2023-11-28T16:53:51.256 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_it821x.ko 2023-11-28T16:53:51.256 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_jmicron.ko 2023-11-28T16:53:51.256 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_marvell.ko 2023-11-28T16:53:51.256 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_mpiix.ko 2023-11-28T16:53:51.257 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_netcell.ko 2023-11-28T16:53:51.257 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ninja32.ko 2023-11-28T16:53:51.257 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87410.ko 2023-11-28T16:53:51.257 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_ns87415.ko 2023-11-28T16:53:51.257 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_oldpiix.ko 2023-11-28T16:53:51.257 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_opti.ko 2023-11-28T16:53:51.257 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_optidma.ko 2023-11-28T16:53:51.257 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pcmcia.ko 2023-11-28T16:53:51.257 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc2027x.ko 2023-11-28T16:53:51.257 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-11-28T16:53:51.257 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_piccolo.ko 2023-11-28T16:53:51.258 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_rdc.ko 2023-11-28T16:53:51.258 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sch.ko 2023-11-28T16:53:51.258 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_serverworks.ko 2023-11-28T16:53:51.258 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sil680.ko 2023-11-28T16:53:51.258 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sis.ko 2023-11-28T16:53:51.258 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_sl82c105.ko 2023-11-28T16:53:51.258 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_triflex.ko 2023-11-28T16:53:51.258 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pata_via.ko 2023-11-28T16:53:51.258 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/pdc_adma.ko 2023-11-28T16:53:51.258 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_inic162x.ko 2023-11-28T16:53:51.258 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_mv.ko 2023-11-28T16:53:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_nv.ko 2023-11-28T16:53:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_promise.ko 2023-11-28T16:53:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_qstor.ko 2023-11-28T16:53:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil.ko 2023-11-28T16:53:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sil24.ko 2023-11-28T16:53:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sis.ko 2023-11-28T16:53:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_svw.ko 2023-11-28T16:53:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_sx4.ko 2023-11-28T16:53:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_uli.ko 2023-11-28T16:53:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_via.ko 2023-11-28T16:53:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ata/sata_vsc.ko 2023-11-28T16:53:51.259 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay 2023-11-28T16:53:51.260 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/auxdisplay/ks0108.ko 2023-11-28T16:53:51.260 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base 2023-11-28T16:53:51.260 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap 2023-11-28T16:53:51.260 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-i2c.ko 2023-11-28T16:53:51.260 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/base/regmap/regmap-mmio.ko 2023-11-28T16:53:51.260 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma 2023-11-28T16:53:51.260 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/bcma/bcma.ko 2023-11-28T16:53:51.260 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block 2023-11-28T16:53:51.260 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe 2023-11-28T16:53:51.260 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/aoe/aoe.ko 2023-11-28T16:53:51.260 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/brd.ko 2023-11-28T16:53:51.261 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd 2023-11-28T16:53:51.261 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/drbd/drbd.ko 2023-11-28T16:53:51.261 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/floppy.ko 2023-11-28T16:53:51.261 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/loop.ko 2023-11-28T16:53:51.261 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx 2023-11-28T16:53:51.261 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-11-28T16:53:51.261 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/nbd.ko 2023-11-28T16:53:51.261 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk 2023-11-28T16:53:51.261 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/null_blk/null_blk.ko 2023-11-28T16:53:51.261 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/pktcdvd.ko 2023-11-28T16:53:51.261 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/rbd.ko 2023-11-28T16:53:51.262 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/virtio_blk.ko 2023-11-28T16:53:51.262 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback 2023-11-28T16:53:51.262 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-11-28T16:53:51.262 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/xen-blkfront.ko 2023-11-28T16:53:51.262 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram 2023-11-28T16:53:51.262 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/block/zram/zram.ko 2023-11-28T16:53:51.262 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char 2023-11-28T16:53:51.262 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hangcheck-timer.ko 2023-11-28T16:53:51.262 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random 2023-11-28T16:53:51.262 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/amd-rng.ko 2023-11-28T16:53:51.262 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/intel-rng.ko 2023-11-28T16:53:51.262 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-11-28T16:53:51.263 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/via-rng.ko 2023-11-28T16:53:51.263 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/hw_random/virtio-rng.ko 2023-11-28T16:53:51.263 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi 2023-11-28T16:53:51.263 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-11-28T16:53:51.263 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-11-28T16:53:51.263 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-11-28T16:53:51.263 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_si.ko 2023-11-28T16:53:51.263 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-11-28T16:53:51.263 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-11-28T16:53:51.263 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/lp.ko 2023-11-28T16:53:51.264 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave 2023-11-28T16:53:51.264 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/mwave/mwave.ko 2023-11-28T16:53:51.264 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/ppdev.ko 2023-11-28T16:53:51.264 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tlclk.ko 2023-11-28T16:53:51.264 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm 2023-11-28T16:53:51.264 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm.ko 2023-11-28T16:53:51.264 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_atmel.ko 2023-11-28T16:53:51.264 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_infineon.ko 2023-11-28T16:53:51.264 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_nsc.ko 2023-11-28T16:53:51.264 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis.ko 2023-11-28T16:53:51.264 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-11-28T16:53:51.264 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/uv_mmtimer.ko 2023-11-28T16:53:51.265 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/char/virtio_console.ko 2023-11-28T16:53:51.265 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq 2023-11-28T16:53:51.265 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-11-28T16:53:51.265 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-11-28T16:53:51.265 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/p4-clockmod.ko 2023-11-28T16:53:51.265 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-11-28T16:53:51.265 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/powernow-k8.ko 2023-11-28T16:53:51.265 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/cpufreq/speedstep-lib.ko 2023-11-28T16:53:51.265 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto 2023-11-28T16:53:51.265 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp 2023-11-28T16:53:51.265 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-11-28T16:53:51.266 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/ccp/ccp.ko 2023-11-28T16:53:51.266 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel 2023-11-28T16:53:51.266 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat 2023-11-28T16:53:51.266 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common 2023-11-28T16:53:51.266 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-11-28T16:53:51.266 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-11-28T16:53:51.266 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-11-28T16:53:51.266 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-aes.ko 2023-11-28T16:53:51.266 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/padlock-sha.ko 2023-11-28T16:53:51.266 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio 2023-11-28T16:53:51.266 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-11-28T16:53:51.266 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca 2023-11-28T16:53:51.267 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dca/dca.ko 2023-11-28T16:53:51.267 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq 2023-11-28T16:53:51.267 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-11-28T16:53:51.267 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma 2023-11-28T16:53:51.267 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw 2023-11-28T16:53:51.267 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/dw/dw_dmac.ko 2023-11-28T16:53:51.267 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat 2023-11-28T16:53:51.267 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/dma/ioat/ioatdma.ko 2023-11-28T16:53:51.267 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac 2023-11-28T16:53:51.267 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/amd64_edac.ko 2023-11-28T16:53:51.267 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/e752x_edac.ko 2023-11-28T16:53:51.268 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/edac_mce_amd.ko 2023-11-28T16:53:51.268 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3000_edac.ko 2023-11-28T16:53:51.268 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i3200_edac.ko 2023-11-28T16:53:51.268 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5100_edac.ko 2023-11-28T16:53:51.268 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i5400_edac.ko 2023-11-28T16:53:51.268 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7300_edac.ko 2023-11-28T16:53:51.268 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i7core_edac.ko 2023-11-28T16:53:51.268 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/i82975x_edac.ko 2023-11-28T16:53:51.268 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/ie31200_edac.ko 2023-11-28T16:53:51.268 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/sb_edac.ko 2023-11-28T16:53:51.268 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/edac/x38_edac.ko 2023-11-28T16:53:51.269 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware 2023-11-28T16:53:51.269 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/edd.ko 2023-11-28T16:53:51.269 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/firmware/iscsi_ibft.ko 2023-11-28T16:53:51.269 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio 2023-11-28T16:53:51.269 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-ich.ko 2023-11-28T16:53:51.269 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/gpio/gpio-viperboard.ko 2023-11-28T16:53:51.269 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid 2023-11-28T16:53:51.269 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-appleir.ko 2023-11-28T16:53:51.269 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-aureal.ko 2023-11-28T16:53:51.269 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-axff.ko 2023-11-28T16:53:51.269 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-dr.ko 2023-11-28T16:53:51.269 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elecom.ko 2023-11-28T16:53:51.270 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-elo.ko 2023-11-28T16:53:51.270 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-emsff.ko 2023-11-28T16:53:51.270 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gaff.ko 2023-11-28T16:53:51.270 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gt683r.ko 2023-11-28T16:53:51.270 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-gyration.ko 2023-11-28T16:53:51.270 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-kbd.ko 2023-11-28T16:53:51.270 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtek-mouse.ko 2023-11-28T16:53:51.270 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-holtekff.ko 2023-11-28T16:53:51.270 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-hyperv.ko 2023-11-28T16:53:51.270 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-icade.ko 2023-11-28T16:53:51.270 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-keytouch.ko 2023-11-28T16:53:51.271 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-kye.ko 2023-11-28T16:53:51.271 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lcpower.ko 2023-11-28T16:53:51.271 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-led.ko 2023-11-28T16:53:51.271 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-lenovo.ko 2023-11-28T16:53:51.271 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-dj.ko 2023-11-28T16:53:51.271 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-11-28T16:53:51.271 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-multitouch.ko 2023-11-28T16:53:51.271 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-ortek.ko 2023-11-28T16:53:51.271 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-penmount.ko 2023-11-28T16:53:51.271 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-petalynx.ko 2023-11-28T16:53:51.271 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-picolcd.ko 2023-11-28T16:53:51.271 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-pl.ko 2023-11-28T16:53:51.272 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-plantronics.ko 2023-11-28T16:53:51.272 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-primax.ko 2023-11-28T16:53:51.272 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-rmi.ko 2023-11-28T16:53:51.272 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-arvo.ko 2023-11-28T16:53:51.272 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-common.ko 2023-11-28T16:53:51.272 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-isku.ko 2023-11-28T16:53:51.272 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kone.ko 2023-11-28T16:53:51.272 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-11-28T16:53:51.272 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-konepure.ko 2023-11-28T16:53:51.272 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-11-28T16:53:51.272 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-lua.ko 2023-11-28T16:53:51.273 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-pyra.ko 2023-11-28T16:53:51.273 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-ryos.ko 2023-11-28T16:53:51.273 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat-savu.ko 2023-11-28T16:53:51.273 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-roccat.ko 2023-11-28T16:53:51.273 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-saitek.ko 2023-11-28T16:53:51.273 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-samsung.ko 2023-11-28T16:53:51.273 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sensor-hub.ko 2023-11-28T16:53:51.273 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sjoy.ko 2023-11-28T16:53:51.273 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sony.ko 2023-11-28T16:53:51.273 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-speedlink.ko 2023-11-28T16:53:51.273 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-steelseries.ko 2023-11-28T16:53:51.274 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-sunplus.ko 2023-11-28T16:53:51.274 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-thrustmaster.ko 2023-11-28T16:53:51.274 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tivo.ko 2023-11-28T16:53:51.274 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-tmff.ko 2023-11-28T16:53:51.274 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-topseed.ko 2023-11-28T16:53:51.274 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-twinhan.ko 2023-11-28T16:53:51.274 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-uclogic.ko 2023-11-28T16:53:51.274 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-waltop.ko 2023-11-28T16:53:51.274 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-wiimote.ko 2023-11-28T16:53:51.274 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-xinmo.ko 2023-11-28T16:53:51.274 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zpff.ko 2023-11-28T16:53:51.275 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/hid-zydacron.ko 2023-11-28T16:53:51.275 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/uhid.ko 2023-11-28T16:53:51.275 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hid/wacom.ko 2023-11-28T16:53:51.275 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv 2023-11-28T16:53:51.275 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_balloon.ko 2023-11-28T16:53:51.275 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_utils.ko 2023-11-28T16:53:51.275 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hv/hv_vmbus.ko 2023-11-28T16:53:51.275 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon 2023-11-28T16:53:51.275 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru.ko 2023-11-28T16:53:51.275 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/abituguru3.ko 2023-11-28T16:53:51.275 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/acpi_power_meter.ko 2023-11-28T16:53:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7414.ko 2023-11-28T16:53:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ad7418.ko 2023-11-28T16:53:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adc128d818.ko 2023-11-28T16:53:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1025.ko 2023-11-28T16:53:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1026.ko 2023-11-28T16:53:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1029.ko 2023-11-28T16:53:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm1031.ko 2023-11-28T16:53:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adm9240.ko 2023-11-28T16:53:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ads7828.ko 2023-11-28T16:53:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7410.ko 2023-11-28T16:53:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7411.ko 2023-11-28T16:53:51.276 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7462.ko 2023-11-28T16:53:51.277 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7470.ko 2023-11-28T16:53:51.277 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7475.ko 2023-11-28T16:53:51.277 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/adt7x10.ko 2023-11-28T16:53:51.277 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/amc6821.ko 2023-11-28T16:53:51.277 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/applesmc.ko 2023-11-28T16:53:51.277 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asb100.ko 2023-11-28T16:53:51.277 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asc7621.ko 2023-11-28T16:53:51.277 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/asus_atk0110.ko 2023-11-28T16:53:51.277 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/atxp1.ko 2023-11-28T16:53:51.277 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/coretemp.ko 2023-11-28T16:53:51.277 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-11-28T16:53:51.277 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/dme1737.ko 2023-11-28T16:53:51.278 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds1621.ko 2023-11-28T16:53:51.278 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ds620.ko 2023-11-28T16:53:51.278 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc1403.ko 2023-11-28T16:53:51.278 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/emc6w201.ko 2023-11-28T16:53:51.278 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71805f.ko 2023-11-28T16:53:51.278 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f71882fg.ko 2023-11-28T16:53:51.278 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/f75375s.ko 2023-11-28T16:53:51.278 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fam15h_power.ko 2023-11-28T16:53:51.278 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/fschmd.ko 2023-11-28T16:53:51.278 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g760a.ko 2023-11-28T16:53:51.278 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/g762.ko 2023-11-28T16:53:51.279 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl518sm.ko 2023-11-28T16:53:51.279 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/gl520sm.ko 2023-11-28T16:53:51.279 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/hwmon-vid.ko 2023-11-28T16:53:51.279 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5500_temp.ko 2023-11-28T16:53:51.279 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/i5k_amb.ko 2023-11-28T16:53:51.279 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmaem.ko 2023-11-28T16:53:51.279 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ibmpex.ko 2023-11-28T16:53:51.279 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina209.ko 2023-11-28T16:53:51.279 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ina2xx.ko 2023-11-28T16:53:51.279 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/it87.ko 2023-11-28T16:53:51.279 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k10temp.ko 2023-11-28T16:53:51.280 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/k8temp.ko 2023-11-28T16:53:51.280 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lineage-pem.ko 2023-11-28T16:53:51.280 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm63.ko 2023-11-28T16:53:51.280 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm73.ko 2023-11-28T16:53:51.280 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm75.ko 2023-11-28T16:53:51.280 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm77.ko 2023-11-28T16:53:51.280 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm78.ko 2023-11-28T16:53:51.280 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm80.ko 2023-11-28T16:53:51.280 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm83.ko 2023-11-28T16:53:51.280 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm85.ko 2023-11-28T16:53:51.280 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm87.ko 2023-11-28T16:53:51.281 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm90.ko 2023-11-28T16:53:51.281 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm92.ko 2023-11-28T16:53:51.281 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm93.ko 2023-11-28T16:53:51.281 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95234.ko 2023-11-28T16:53:51.281 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95241.ko 2023-11-28T16:53:51.281 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/lm95245.ko 2023-11-28T16:53:51.281 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc2945.ko 2023-11-28T16:53:51.281 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4151.ko 2023-11-28T16:53:51.281 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4215.ko 2023-11-28T16:53:51.281 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4222.ko 2023-11-28T16:53:51.281 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4245.ko 2023-11-28T16:53:51.281 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4260.ko 2023-11-28T16:53:51.282 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/ltc4261.ko 2023-11-28T16:53:51.282 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max16065.ko 2023-11-28T16:53:51.282 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1619.ko 2023-11-28T16:53:51.282 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max1668.ko 2023-11-28T16:53:51.282 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max197.ko 2023-11-28T16:53:51.282 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6639.ko 2023-11-28T16:53:51.282 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6650.ko 2023-11-28T16:53:51.282 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/max6697.ko 2023-11-28T16:53:51.282 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/mcp3021.ko 2023-11-28T16:53:51.282 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6683.ko 2023-11-28T16:53:51.282 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775-core.ko 2023-11-28T16:53:51.283 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct6775.ko 2023-11-28T16:53:51.283 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/nct7802.ko 2023-11-28T16:53:51.283 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87360.ko 2023-11-28T16:53:51.283 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pc87427.ko 2023-11-28T16:53:51.283 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pcf8591.ko 2023-11-28T16:53:51.283 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus 2023-11-28T16:53:51.283 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-11-28T16:53:51.283 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-11-28T16:53:51.283 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-11-28T16:53:51.283 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max16064.ko 2023-11-28T16:53:51.283 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max34440.ko 2023-11-28T16:53:51.283 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/max8688.ko 2023-11-28T16:53:51.284 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-11-28T16:53:51.284 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-11-28T16:53:51.284 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-11-28T16:53:51.284 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-11-28T16:53:51.284 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-11-28T16:53:51.284 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-11-28T16:53:51.284 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/powr1220.ko 2023-11-28T16:53:51.284 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5627.ko 2023-11-28T16:53:51.284 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch5636.ko 2023-11-28T16:53:51.284 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sch56xx-common.ko 2023-11-28T16:53:51.284 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht15.ko 2023-11-28T16:53:51.285 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sht21.ko 2023-11-28T16:53:51.285 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/shtc1.ko 2023-11-28T16:53:51.285 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/sis5595.ko 2023-11-28T16:53:51.285 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47b397.ko 2023-11-28T16:53:51.285 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m1.ko 2023-11-28T16:53:51.285 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/smsc47m192.ko 2023-11-28T16:53:51.285 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/thmc50.ko 2023-11-28T16:53:51.285 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp102.ko 2023-11-28T16:53:51.285 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp103.ko 2023-11-28T16:53:51.285 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp401.ko 2023-11-28T16:53:51.286 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/tmp421.ko 2023-11-28T16:53:51.286 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via-cputemp.ko 2023-11-28T16:53:51.286 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/via686a.ko 2023-11-28T16:53:51.286 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt1211.ko 2023-11-28T16:53:51.286 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/vt8231.ko 2023-11-28T16:53:51.286 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627ehf.ko 2023-11-28T16:53:51.286 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83627hf.ko 2023-11-28T16:53:51.286 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83781d.ko 2023-11-28T16:53:51.286 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83791d.ko 2023-11-28T16:53:51.286 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83792d.ko 2023-11-28T16:53:51.286 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83793.ko 2023-11-28T16:53:51.287 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83795.ko 2023-11-28T16:53:51.287 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l785ts.ko 2023-11-28T16:53:51.287 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/hwmon/w83l786ng.ko 2023-11-28T16:53:51.287 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c 2023-11-28T16:53:51.287 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos 2023-11-28T16:53:51.287 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-11-28T16:53:51.287 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-11-28T16:53:51.287 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses 2023-11-28T16:53:51.287 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-11-28T16:53:51.287 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-11-28T16:53:51.287 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-11-28T16:53:51.287 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-11-28T16:53:51.288 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-11-28T16:53:51.288 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-11-28T16:53:51.288 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-11-28T16:53:51.288 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-11-28T16:53:51.288 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-i801.ko 2023-11-28T16:53:51.288 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-isch.ko 2023-11-28T16:53:51.288 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-11-28T16:53:51.288 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-11-28T16:53:51.288 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-11-28T16:53:51.288 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-parport.ko 2023-11-28T16:53:51.288 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-11-28T16:53:51.289 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-11-28T16:53:51.289 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-11-28T16:53:51.289 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-11-28T16:53:51.289 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-11-28T16:53:51.289 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-11-28T16:53:51.289 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-via.ko 2023-11-28T16:53:51.289 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-11-28T16:53:51.289 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-11-28T16:53:51.289 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-dev.ko 2023-11-28T16:53:51.289 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-mux.ko 2023-11-28T16:53:51.289 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-11-28T16:53:51.289 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-smbus.ko 2023-11-28T16:53:51.290 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/i2c/i2c-stub.ko 2023-11-28T16:53:51.290 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband 2023-11-28T16:53:51.290 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core 2023-11-28T16:53:51.290 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_cm.ko 2023-11-28T16:53:51.290 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_core.ko 2023-11-28T16:53:51.290 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_umad.ko 2023-11-28T16:53:51.290 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-11-28T16:53:51.290 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/iw_cm.ko 2023-11-28T16:53:51.290 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_cm.ko 2023-11-28T16:53:51.290 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-11-28T16:53:51.291 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw 2023-11-28T16:53:51.291 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4 2023-11-28T16:53:51.291 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-11-28T16:53:51.291 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4 2023-11-28T16:53:51.291 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-11-28T16:53:51.291 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5 2023-11-28T16:53:51.291 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-11-28T16:53:51.291 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca 2023-11-28T16:53:51.291 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-11-28T16:53:51.291 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp 2023-11-28T16:53:51.291 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib 2023-11-28T16:53:51.291 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-11-28T16:53:51.292 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser 2023-11-28T16:53:51.292 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-11-28T16:53:51.292 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert 2023-11-28T16:53:51.292 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-11-28T16:53:51.292 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp 2023-11-28T16:53:51.292 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-11-28T16:53:51.292 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt 2023-11-28T16:53:51.292 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-11-28T16:53:51.292 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input 2023-11-28T16:53:51.292 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport 2023-11-28T16:53:51.292 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-11-28T16:53:51.293 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/fm801-gp.ko 2023-11-28T16:53:51.293 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/gameport/gameport.ko 2023-11-28T16:53:51.293 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/joydev.ko 2023-11-28T16:53:51.293 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard 2023-11-28T16:53:51.293 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/keyboard/gpio_keys.ko 2023-11-28T16:53:51.293 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse 2023-11-28T16:53:51.293 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/appletouch.ko 2023-11-28T16:53:51.293 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/bcm5974.ko 2023-11-28T16:53:51.293 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/cyapatp.ko 2023-11-28T16:53:51.293 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/elan_i2c.ko 2023-11-28T16:53:51.293 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/sermouse.ko 2023-11-28T16:53:51.294 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-11-28T16:53:51.294 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/synaptics_usb.ko 2023-11-28T16:53:51.294 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/mouse/vsxxxaa.ko 2023-11-28T16:53:51.294 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4 2023-11-28T16:53:51.294 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/rmi4/rmi_core.ko 2023-11-28T16:53:51.294 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio 2023-11-28T16:53:51.294 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/altera_ps2.ko 2023-11-28T16:53:51.294 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/arc_ps2.ko 2023-11-28T16:53:51.294 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-11-28T16:53:51.294 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/serio/serio_raw.ko 2023-11-28T16:53:51.294 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/input/sparse-keymap.ko 2023-11-28T16:53:51.295 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu 2023-11-28T16:53:51.295 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd 2023-11-28T16:53:51.295 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/iommu/amd/iommu_v2.ko 2023-11-28T16:53:51.295 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds 2023-11-28T16:53:51.295 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-blinkm.ko 2023-11-28T16:53:51.295 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lm3530.ko 2023-11-28T16:53:51.295 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-lp3944.ko 2023-11-28T16:53:51.295 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/leds-ss4200.ko 2023-11-28T16:53:51.295 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger 2023-11-28T16:53:51.295 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-11-28T16:53:51.295 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-11-28T16:53:51.295 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-11-28T16:53:51.296 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-11-28T16:53:51.296 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-11-28T16:53:51.296 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-11-28T16:53:51.296 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-11-28T16:53:51.296 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-11-28T16:53:51.296 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md 2023-11-28T16:53:51.296 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache 2023-11-28T16:53:51.296 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/bcache/bcache.ko 2023-11-28T16:53:51.296 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-bio-prison.ko 2023-11-28T16:53:51.296 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache-smq.ko 2023-11-28T16:53:51.296 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-cache.ko 2023-11-28T16:53:51.297 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-crypt.ko 2023-11-28T16:53:51.297 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-delay.ko 2023-11-28T16:53:51.297 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-flakey.ko 2023-11-28T16:53:51.297 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-log-userspace.ko 2023-11-28T16:53:51.297 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-multipath.ko 2023-11-28T16:53:51.297 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-queue-length.ko 2023-11-28T16:53:51.297 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-raid.ko 2023-11-28T16:53:51.297 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-round-robin.ko 2023-11-28T16:53:51.297 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-service-time.ko 2023-11-28T16:53:51.297 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-switch.ko 2023-11-28T16:53:51.297 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-thin-pool.ko 2023-11-28T16:53:51.297 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/dm-verity.ko 2023-11-28T16:53:51.298 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/faulty.ko 2023-11-28T16:53:51.298 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/linear.ko 2023-11-28T16:53:51.298 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/multipath.ko 2023-11-28T16:53:51.298 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data 2023-11-28T16:53:51.298 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-11-28T16:53:51.298 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid0.ko 2023-11-28T16:53:51.298 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid1.ko 2023-11-28T16:53:51.298 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid10.ko 2023-11-28T16:53:51.298 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/md/raid456.ko 2023-11-28T16:53:51.298 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message 2023-11-28T16:53:51.299 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion 2023-11-28T16:53:51.299 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptbase.ko 2023-11-28T16:53:51.299 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptctl.ko 2023-11-28T16:53:51.299 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptfc.ko 2023-11-28T16:53:51.299 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptlan.ko 2023-11-28T16:53:51.299 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptsas.ko 2023-11-28T16:53:51.299 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptscsih.ko 2023-11-28T16:53:51.299 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/message/fusion/mptspi.ko 2023-11-28T16:53:51.299 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd 2023-11-28T16:53:51.300 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_ich.ko 2023-11-28T16:53:51.300 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/lpc_sch.ko 2023-11-28T16:53:51.300 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/mfd-core.ko 2023-11-28T16:53:51.300 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/sm501.ko 2023-11-28T16:53:51.300 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/viperboard.ko 2023-11-28T16:53:51.300 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/vx855.ko 2023-11-28T16:53:51.300 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/mfd/wl1273-core.ko 2023-11-28T16:53:51.300 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc 2023-11-28T16:53:51.300 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl 2023-11-28T16:53:51.300 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-11-28T16:53:51.300 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds9802als.ko 2023-11-28T16:53:51.301 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/apds990x.ko 2023-11-28T16:53:51.301 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/bh1770glc.ko 2023-11-28T16:53:51.301 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710 2023-11-28T16:53:51.301 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/cb710/cb710.ko 2023-11-28T16:53:51.301 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom 2023-11-28T16:53:51.301 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/at24.ko 2023-11-28T16:53:51.301 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom.ko 2023-11-28T16:53:51.301 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-11-28T16:53:51.301 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/eeprom/max6875.ko 2023-11-28T16:53:51.301 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/enclosure.ko 2023-11-28T16:53:51.301 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/hpilo.ko 2023-11-28T16:53:51.302 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29003.ko 2023-11-28T16:53:51.302 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/isl29020.ko 2023-11-28T16:53:51.302 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d 2023-11-28T16:53:51.302 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-11-28T16:53:51.302 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-11-28T16:53:51.302 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei 2023-11-28T16:53:51.302 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-me.ko 2023-11-28T16:53:51.302 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei-txe.ko 2023-11-28T16:53:51.302 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/mei/mei.ko 2023-11-28T16:53:51.302 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru 2023-11-28T16:53:51.302 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-gru/gru.ko 2023-11-28T16:53:51.303 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp 2023-11-28T16:53:51.303 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xp.ko 2023-11-28T16:53:51.303 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpc.ko 2023-11-28T16:53:51.303 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-11-28T16:53:51.303 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_7xx1.ko 2023-11-28T16:53:51.303 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tifm_core.ko 2023-11-28T16:53:51.303 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/tsl2550.ko 2023-11-28T16:53:51.303 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_balloon.ko 2023-11-28T16:53:51.303 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci 2023-11-28T16:53:51.303 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-11-28T16:53:51.303 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net 2023-11-28T16:53:51.303 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding 2023-11-28T16:53:51.304 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/bonding/bonding.ko 2023-11-28T16:53:51.304 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa 2023-11-28T16:53:51.304 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53 2023-11-28T16:53:51.304 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/b53/b53_common.ko 2023-11-28T16:53:51.304 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/bcm-sf2.ko 2023-11-28T16:53:51.304 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6060.ko 2023-11-28T16:53:51.304 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx 2023-11-28T16:53:51.304 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-11-28T16:53:51.304 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/dummy.ko 2023-11-28T16:53:51.304 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/eql.ko 2023-11-28T16:53:51.304 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet 2023-11-28T16:53:51.305 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com 2023-11-28T16:53:51.305 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-11-28T16:53:51.305 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-11-28T16:53:51.305 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-11-28T16:53:51.305 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-11-28T16:53:51.305 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390 2023-11-28T16:53:51.305 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/8390.ko 2023-11-28T16:53:51.305 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-11-28T16:53:51.305 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-11-28T16:53:51.305 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-11-28T16:53:51.305 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec 2023-11-28T16:53:51.306 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-11-28T16:53:51.306 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere 2023-11-28T16:53:51.306 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/agere/et131x.ko 2023-11-28T16:53:51.306 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon 2023-11-28T16:53:51.306 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-11-28T16:53:51.306 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera 2023-11-28T16:53:51.306 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-11-28T16:53:51.306 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd 2023-11-28T16:53:51.306 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-11-28T16:53:51.306 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-11-28T16:53:51.306 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-11-28T16:53:51.306 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros 2023-11-28T16:53:51.307 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx 2023-11-28T16:53:51.307 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-11-28T16:53:51.307 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c 2023-11-28T16:53:51.307 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-11-28T16:53:51.307 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e 2023-11-28T16:53:51.307 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-11-28T16:53:51.307 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx 2023-11-28T16:53:51.307 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-11-28T16:53:51.307 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-11-28T16:53:51.307 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom 2023-11-28T16:53:51.307 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-11-28T16:53:51.308 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-11-28T16:53:51.308 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-11-28T16:53:51.308 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-11-28T16:53:51.308 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-11-28T16:53:51.308 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet 2023-11-28T16:53:51.308 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-11-28T16:53:51.308 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-11-28T16:53:51.308 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade 2023-11-28T16:53:51.308 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna 2023-11-28T16:53:51.308 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-11-28T16:53:51.308 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio 2023-11-28T16:53:51.308 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb 2023-11-28T16:53:51.309 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-11-28T16:53:51.309 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-11-28T16:53:51.309 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-11-28T16:53:51.309 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-11-28T16:53:51.309 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-11-28T16:53:51.309 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-11-28T16:53:51.309 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-11-28T16:53:51.309 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-11-28T16:53:51.309 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-11-28T16:53:51.309 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco 2023-11-28T16:53:51.309 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic 2023-11-28T16:53:51.310 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-11-28T16:53:51.310 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec 2023-11-28T16:53:51.310 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip 2023-11-28T16:53:51.310 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-11-28T16:53:51.310 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-11-28T16:53:51.310 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-11-28T16:53:51.310 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-11-28T16:53:51.310 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-11-28T16:53:51.310 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-11-28T16:53:51.310 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink 2023-11-28T16:53:51.311 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-11-28T16:53:51.311 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-11-28T16:53:51.311 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/dnet.ko 2023-11-28T16:53:51.311 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex 2023-11-28T16:53:51.311 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet 2023-11-28T16:53:51.311 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-11-28T16:53:51.311 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ethoc.ko 2023-11-28T16:53:51.311 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/fealnx.ko 2023-11-28T16:53:51.311 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel 2023-11-28T16:53:51.311 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e100.ko 2023-11-28T16:53:51.311 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000 2023-11-28T16:53:51.312 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-11-28T16:53:51.312 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e 2023-11-28T16:53:51.312 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-11-28T16:53:51.312 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k 2023-11-28T16:53:51.312 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-11-28T16:53:51.312 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e 2023-11-28T16:53:51.312 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-11-28T16:53:51.312 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf 2023-11-28T16:53:51.312 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-11-28T16:53:51.312 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb 2023-11-28T16:53:51.312 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-11-28T16:53:51.313 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf 2023-11-28T16:53:51.313 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-11-28T16:53:51.313 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe 2023-11-28T16:53:51.313 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-11-28T16:53:51.313 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf 2023-11-28T16:53:51.313 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-11-28T16:53:51.313 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/jme.ko 2023-11-28T16:53:51.313 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell 2023-11-28T16:53:51.313 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-11-28T16:53:51.313 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/skge.ko 2023-11-28T16:53:51.313 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-11-28T16:53:51.314 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox 2023-11-28T16:53:51.314 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4 2023-11-28T16:53:51.314 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-11-28T16:53:51.314 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-11-28T16:53:51.314 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5 2023-11-28T16:53:51.314 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-11-28T16:53:51.314 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-11-28T16:53:51.314 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel 2023-11-28T16:53:51.314 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-11-28T16:53:51.314 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom 2023-11-28T16:53:51.314 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge 2023-11-28T16:53:51.315 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-11-28T16:53:51.315 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi 2023-11-28T16:53:51.315 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-11-28T16:53:51.315 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-11-28T16:53:51.315 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion 2023-11-28T16:53:51.315 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-11-28T16:53:51.315 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia 2023-11-28T16:53:51.315 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-11-28T16:53:51.315 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines 2023-11-28T16:53:51.315 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-11-28T16:53:51.315 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-11-28T16:53:51.315 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic 2023-11-28T16:53:51.317 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen 2023-11-28T16:53:51.317 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-11-28T16:53:51.317 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-11-28T16:53:51.317 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-11-28T16:53:51.317 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-11-28T16:53:51.317 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc 2023-11-28T16:53:51.317 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-11-28T16:53:51.318 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek 2023-11-28T16:53:51.318 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-11-28T16:53:51.318 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-11-28T16:53:51.318 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/atp.ko 2023-11-28T16:53:51.318 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-11-28T16:53:51.318 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker 2023-11-28T16:53:51.318 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-11-28T16:53:51.318 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc 2023-11-28T16:53:51.318 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-11-28T16:53:51.318 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan 2023-11-28T16:53:51.318 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-11-28T16:53:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis 2023-11-28T16:53:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis190.ko 2023-11-28T16:53:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sis/sis900.ko 2023-11-28T16:53:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc 2023-11-28T16:53:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-11-28T16:53:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-11-28T16:53:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-11-28T16:53:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-11-28T16:53:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro 2023-11-28T16:53:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac 2023-11-28T16:53:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-11-28T16:53:51.319 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-11-28T16:53:51.320 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun 2023-11-28T16:53:51.320 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/cassini.ko 2023-11-28T16:53:51.320 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/niu.ko 2023-11-28T16:53:51.320 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sungem.ko 2023-11-28T16:53:51.320 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-11-28T16:53:51.320 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti 2023-11-28T16:53:51.320 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-11-28T16:53:51.320 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti 2023-11-28T16:53:51.320 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/ti/tlan.ko 2023-11-28T16:53:51.320 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via 2023-11-28T16:53:51.320 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-11-28T16:53:51.321 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-11-28T16:53:51.321 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet 2023-11-28T16:53:51.321 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-11-28T16:53:51.321 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-11-28T16:53:51.321 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom 2023-11-28T16:53:51.321 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-11-28T16:53:51.321 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/geneve.ko 2023-11-28T16:53:51.321 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv 2023-11-28T16:53:51.322 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-11-28T16:53:51.322 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154 2023-11-28T16:53:51.322 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ieee802154/fakelb.ko 2023-11-28T16:53:51.322 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ifb.ko 2023-11-28T16:53:51.322 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan 2023-11-28T16:53:51.322 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ipvlan/ipvlan.ko 2023-11-28T16:53:51.322 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvlan.ko 2023-11-28T16:53:51.322 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/macvtap.ko 2023-11-28T16:53:51.323 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio 2023-11-28T16:53:51.323 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio.ko 2023-11-28T16:53:51.323 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-11-28T16:53:51.323 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-11-28T16:53:51.323 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mdio/mdio-regmap.ko 2023-11-28T16:53:51.323 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/mii.ko 2023-11-28T16:53:51.323 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/net_failover.ko 2023-11-28T16:53:51.323 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/netconsole.ko 2023-11-28T16:53:51.323 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/nlmon.ko 2023-11-28T16:53:51.323 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs 2023-11-28T16:53:51.323 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs-lynx.ko 2023-11-28T16:53:51.324 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-11-28T16:53:51.324 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy 2023-11-28T16:53:51.324 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/amd.ko 2023-11-28T16:53:51.324 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-11-28T16:53:51.324 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-11-28T16:53:51.324 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm7xxx.ko 2023-11-28T16:53:51.324 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/bcm87xx.ko 2023-11-28T16:53:51.324 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/broadcom.ko 2023-11-28T16:53:51.324 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/cicada.ko 2023-11-28T16:53:51.324 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/davicom.ko 2023-11-28T16:53:51.324 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/dp83640.ko 2023-11-28T16:53:51.325 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/et1011c.ko 2023-11-28T16:53:51.325 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/icplus.ko 2023-11-28T16:53:51.325 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/lxt.ko 2023-11-28T16:53:51.325 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/marvell.ko 2023-11-28T16:53:51.325 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/micrel.ko 2023-11-28T16:53:51.325 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/national.ko 2023-11-28T16:53:51.325 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/phylink.ko 2023-11-28T16:53:51.325 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/qsemi.ko 2023-11-28T16:53:51.325 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/realtek.ko 2023-11-28T16:53:51.325 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/smsc.ko 2023-11-28T16:53:51.325 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/ste10Xp.ko 2023-11-28T16:53:51.325 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/phy/vitesse.ko 2023-11-28T16:53:51.326 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp 2023-11-28T16:53:51.326 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/bsd_comp.ko 2023-11-28T16:53:51.326 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_async.ko 2023-11-28T16:53:51.326 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_deflate.ko 2023-11-28T16:53:51.326 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_generic.ko 2023-11-28T16:53:51.326 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_mppe.ko 2023-11-28T16:53:51.326 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/ppp_synctty.ko 2023-11-28T16:53:51.326 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppoe.ko 2023-11-28T16:53:51.326 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pppox.ko 2023-11-28T16:53:51.326 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/ppp/pptp.ko 2023-11-28T16:53:51.326 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip 2023-11-28T16:53:51.327 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slhc.ko 2023-11-28T16:53:51.327 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/slip/slip.ko 2023-11-28T16:53:51.327 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/sungem_phy.ko 2023-11-28T16:53:51.327 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tap.ko 2023-11-28T16:53:51.327 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team 2023-11-28T16:53:51.327 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team.ko 2023-11-28T16:53:51.327 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_activebackup.ko 2023-11-28T16:53:51.327 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_broadcast.ko 2023-11-28T16:53:51.327 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-11-28T16:53:51.327 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_random.ko 2023-11-28T16:53:51.327 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-11-28T16:53:51.328 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/tun.ko 2023-11-28T16:53:51.328 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/veth.ko 2023-11-28T16:53:51.328 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/virtio_net.ko 2023-11-28T16:53:51.328 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3 2023-11-28T16:53:51.328 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-11-28T16:53:51.328 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan 2023-11-28T16:53:51.328 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/vxlan/vxlan.ko 2023-11-28T16:53:51.328 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback 2023-11-28T16:53:51.328 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netback/xen-netback.ko 2023-11-28T16:53:51.328 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/net/xen-netfront.ko 2023-11-28T16:53:51.328 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb 2023-11-28T16:53:51.328 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ntb/ntb.ko 2023-11-28T16:53:51.329 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme 2023-11-28T16:53:51.329 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host 2023-11-28T16:53:51.329 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme-core.ko 2023-11-28T16:53:51.329 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/nvme/host/nvme.ko 2023-11-28T16:53:51.329 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport 2023-11-28T16:53:51.329 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport.ko 2023-11-28T16:53:51.329 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_cs.ko 2023-11-28T16:53:51.329 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_pc.ko 2023-11-28T16:53:51.329 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/parport/parport_serial.ko 2023-11-28T16:53:51.329 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci 2023-11-28T16:53:51.329 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug 2023-11-28T16:53:51.329 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-11-28T16:53:51.330 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie 2023-11-28T16:53:51.330 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/pcie/aer_inject.ko 2023-11-28T16:53:51.330 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pci/xen-pcifront.ko 2023-11-28T16:53:51.330 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia 2023-11-28T16:53:51.330 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/i82092.ko 2023-11-28T16:53:51.330 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/pd6729.ko 2023-11-28T16:53:51.330 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pcmcia/yenta_socket.ko 2023-11-28T16:53:51.330 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl 2023-11-28T16:53:51.330 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel 2023-11-28T16:53:51.330 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-11-28T16:53:51.330 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform 2023-11-28T16:53:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86 2023-11-28T16:53:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acer-wmi.ko 2023-11-28T16:53:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/acerhdf.ko 2023-11-28T16:53:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/apple-gmux.ko 2023-11-28T16:53:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-laptop.ko 2023-11-28T16:53:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-11-28T16:53:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/asus-wmi.ko 2023-11-28T16:53:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/classmate-laptop.ko 2023-11-28T16:53:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-11-28T16:53:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-11-28T16:53:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-11-28T16:53:51.331 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-11-28T16:53:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/hdaps.ko 2023-11-28T16:53:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel 2023-11-28T16:53:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-11-28T16:53:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/intel_ips.ko 2023-11-28T16:53:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/msi-wmi.ko 2023-11-28T16:53:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/mxm-wmi.ko 2023-11-28T16:53:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-11-28T16:53:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-laptop.ko 2023-11-28T16:53:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/samsung-q10.ko 2023-11-28T16:53:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/topstar-laptop.ko 2023-11-28T16:53:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-11-28T16:53:51.332 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/toshiba_haps.ko 2023-11-28T16:53:51.333 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi-bmof.ko 2023-11-28T16:53:51.333 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/platform/x86/wmi.ko 2023-11-28T16:53:51.333 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap 2023-11-28T16:53:51.333 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_common.ko 2023-11-28T16:53:51.333 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/powercap/intel_rapl_msr.ko 2023-11-28T16:53:51.333 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps 2023-11-28T16:53:51.333 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients 2023-11-28T16:53:51.333 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-gpio.ko 2023-11-28T16:53:51.333 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps-ldisc.ko 2023-11-28T16:53:51.333 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/clients/pps_parport.ko 2023-11-28T16:53:51.333 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/pps/pps_core.ko 2023-11-28T16:53:51.334 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp 2023-11-28T16:53:51.334 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp.ko 2023-11-28T16:53:51.334 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ptp/ptp_kvm.ko 2023-11-28T16:53:51.334 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc 2023-11-28T16:53:51.334 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-bq32k.ko 2023-11-28T16:53:51.334 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1286.ko 2023-11-28T16:53:51.334 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1307.ko 2023-11-28T16:53:51.334 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1374.ko 2023-11-28T16:53:51.334 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1511.ko 2023-11-28T16:53:51.334 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1553.ko 2023-11-28T16:53:51.334 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1672.ko 2023-11-28T16:53:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds1742.ko 2023-11-28T16:53:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds2404.ko 2023-11-28T16:53:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-ds3232.ko 2023-11-28T16:53:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-em3027.ko 2023-11-28T16:53:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-fm3130.ko 2023-11-28T16:53:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl12022.ko 2023-11-28T16:53:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-isl1208.ko 2023-11-28T16:53:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m41t80.ko 2023-11-28T16:53:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t35.ko 2023-11-28T16:53:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-m48t59.ko 2023-11-28T16:53:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-max6900.ko 2023-11-28T16:53:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-msm6242.ko 2023-11-28T16:53:51.335 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf2127.ko 2023-11-28T16:53:51.336 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf85063.ko 2023-11-28T16:53:51.336 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8523.ko 2023-11-28T16:53:51.336 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8563.ko 2023-11-28T16:53:51.336 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-pcf8583.ko 2023-11-28T16:53:51.336 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rp5c01.ko 2023-11-28T16:53:51.336 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rs5c372.ko 2023-11-28T16:53:51.336 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-11-28T16:53:51.336 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8025.ko 2023-11-28T16:53:51.336 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-rx8581.ko 2023-11-28T16:53:51.336 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-11-28T16:53:51.336 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/rtc/rtc-x1205.ko 2023-11-28T16:53:51.336 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi 2023-11-28T16:53:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-9xxx.ko 2023-11-28T16:53:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-sas.ko 2023-11-28T16:53:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/3w-xxxx.ko 2023-11-28T16:53:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/BusLogic.ko 2023-11-28T16:53:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/a100u2w.ko 2023-11-28T16:53:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid 2023-11-28T16:53:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aacraid/aacraid.ko 2023-11-28T16:53:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/advansys.ko 2023-11-28T16:53:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx 2023-11-28T16:53:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-11-28T16:53:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-11-28T16:53:51.337 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/am53c974.ko 2023-11-28T16:53:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr 2023-11-28T16:53:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-11-28T16:53:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/atp870u.ko 2023-11-28T16:53:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi 2023-11-28T16:53:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-11-28T16:53:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa 2023-11-28T16:53:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bfa/bfa.ko 2023-11-28T16:53:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc 2023-11-28T16:53:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-11-28T16:53:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i 2023-11-28T16:53:51.338 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-11-28T16:53:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ch.ko 2023-11-28T16:53:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor 2023-11-28T16:53:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/csiostor/csiostor.ko 2023-11-28T16:53:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi 2023-11-28T16:53:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i 2023-11-28T16:53:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-11-28T16:53:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i 2023-11-28T16:53:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-11-28T16:53:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-11-28T16:53:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dc395x.ko 2023-11-28T16:53:51.339 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler 2023-11-28T16:53:51.340 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-11-28T16:53:51.340 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-11-28T16:53:51.340 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-11-28T16:53:51.340 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-11-28T16:53:51.340 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/dmx3191d.ko 2023-11-28T16:53:51.340 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r 2023-11-28T16:53:51.340 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esas2r/esas2r.ko 2023-11-28T16:53:51.340 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/esp_scsi.ko 2023-11-28T16:53:51.340 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe 2023-11-28T16:53:51.340 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/fcoe.ko 2023-11-28T16:53:51.340 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-11-28T16:53:51.340 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic 2023-11-28T16:53:51.341 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/fnic/fnic.ko 2023-11-28T16:53:51.341 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hpsa.ko 2023-11-28T16:53:51.341 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hptiop.ko 2023-11-28T16:53:51.341 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/hv_storvsc.ko 2023-11-28T16:53:51.341 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/initio.ko 2023-11-28T16:53:51.341 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ipr.ko 2023-11-28T16:53:51.341 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ips.ko 2023-11-28T16:53:51.341 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci 2023-11-28T16:53:51.341 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/isci/isci.ko 2023-11-28T16:53:51.341 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-11-28T16:53:51.341 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/iscsi_tcp.ko 2023-11-28T16:53:51.341 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc 2023-11-28T16:53:51.342 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libfc/libfc.ko 2023-11-28T16:53:51.342 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi.ko 2023-11-28T16:53:51.342 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libiscsi_tcp.ko 2023-11-28T16:53:51.342 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas 2023-11-28T16:53:51.342 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/libsas/libsas.ko 2023-11-28T16:53:51.342 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc 2023-11-28T16:53:51.342 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/lpfc/lpfc.ko 2023-11-28T16:53:51.342 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid 2023-11-28T16:53:51.342 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid.ko 2023-11-28T16:53:51.342 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-11-28T16:53:51.342 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-11-28T16:53:51.343 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-11-28T16:53:51.343 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas 2023-11-28T16:53:51.343 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-11-28T16:53:51.343 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas 2023-11-28T16:53:51.343 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvsas/mvsas.ko 2023-11-28T16:53:51.343 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/mvumi.ko 2023-11-28T16:53:51.343 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001 2023-11-28T16:53:51.343 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-11-28T16:53:51.343 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/pmcraid.ko 2023-11-28T16:53:51.343 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla1280.ko 2023-11-28T16:53:51.343 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx 2023-11-28T16:53:51.343 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-11-28T16:53:51.344 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-11-28T16:53:51.344 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx 2023-11-28T16:53:51.344 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-11-28T16:53:51.344 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/raid_class.ko 2023-11-28T16:53:51.344 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_debug.ko 2023-11-28T16:53:51.344 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_fc.ko 2023-11-28T16:53:51.344 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-11-28T16:53:51.344 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_sas.ko 2023-11-28T16:53:51.344 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_spi.ko 2023-11-28T16:53:51.344 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/scsi_transport_srp.ko 2023-11-28T16:53:51.345 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/ses.ko 2023-11-28T16:53:51.345 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/st.ko 2023-11-28T16:53:51.345 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/stex.ko 2023-11-28T16:53:51.345 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2 2023-11-28T16:53:51.345 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-11-28T16:53:51.345 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/virtio_scsi.ko 2023-11-28T16:53:51.345 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/vmw_pvscsi.ko 2023-11-28T16:53:51.345 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/scsi/wd719x.ko 2023-11-28T16:53:51.345 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb 2023-11-28T16:53:51.345 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ssb/ssb.ko 2023-11-28T16:53:51.345 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target 2023-11-28T16:53:51.345 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi 2023-11-28T16:53:51.346 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-11-28T16:53:51.346 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback 2023-11-28T16:53:51.346 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/loopback/tcm_loop.ko 2023-11-28T16:53:51.346 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_file.ko 2023-11-28T16:53:51.346 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_iblock.ko 2023-11-28T16:53:51.346 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_mod.ko 2023-11-28T16:53:51.346 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_pscsi.ko 2023-11-28T16:53:51.346 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/target_core_user.ko 2023-11-28T16:53:51.346 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc 2023-11-28T16:53:51.346 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-11-28T16:53:51.346 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal 2023-11-28T16:53:51.347 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel 2023-11-28T16:53:51.347 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal 2023-11-28T16:53:51.347 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-11-28T16:53:51.347 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-11-28T16:53:51.347 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-11-28T16:53:51.347 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-11-28T16:53:51.347 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-11-28T16:53:51.347 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-11-28T16:53:51.347 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-11-28T16:53:51.347 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-11-28T16:53:51.347 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-11-28T16:53:51.348 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-11-28T16:53:51.348 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-11-28T16:53:51.348 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-11-28T16:53:51.348 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-11-28T16:53:51.348 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-11-28T16:53:51.348 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-11-28T16:53:51.348 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty 2023-11-28T16:53:51.348 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless 2023-11-28T16:53:51.348 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-11-28T16:53:51.348 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_gsm.ko 2023-11-28T16:53:51.348 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/n_hdlc.ko 2023-11-28T16:53:51.348 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/nozomi.ko 2023-11-28T16:53:51.349 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial 2023-11-28T16:53:51.349 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250 2023-11-28T16:53:51.349 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-11-28T16:53:51.349 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/arc_uart.ko 2023-11-28T16:53:51.349 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm 2023-11-28T16:53:51.349 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/tty/serial/jsm/jsm.ko 2023-11-28T16:53:51.349 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs 2023-11-28T16:53:51.349 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core 2023-11-28T16:53:51.349 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/core/ufshcd-core.ko 2023-11-28T16:53:51.349 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host 2023-11-28T16:53:51.349 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-11-28T16:53:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio 2023-11-28T16:53:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio.ko 2023-11-28T16:53:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_aec.ko 2023-11-28T16:53:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_cif.ko 2023-11-28T16:53:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_pci_generic.ko 2023-11-28T16:53:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/uio/uio_sercos3.ko 2023-11-28T16:53:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb 2023-11-28T16:53:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class 2023-11-28T16:53:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-acm.ko 2023-11-28T16:53:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/cdc-wdm.ko 2023-11-28T16:53:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usblp.ko 2023-11-28T16:53:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/class/usbtmc.ko 2023-11-28T16:53:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host 2023-11-28T16:53:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/host/sl811-hcd.ko 2023-11-28T16:53:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image 2023-11-28T16:53:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/mdc800.ko 2023-11-28T16:53:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/image/microtek.ko 2023-11-28T16:53:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc 2023-11-28T16:53:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/adutux.ko 2023-11-28T16:53:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/appledisplay.ko 2023-11-28T16:53:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi26.ko 2023-11-28T16:53:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/emi62.ko 2023-11-28T16:53:51.351 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ezusb.ko 2023-11-28T16:53:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/idmouse.ko 2023-11-28T16:53:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/iowarrior.ko 2023-11-28T16:53:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/isight_firmware.ko 2023-11-28T16:53:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/ldusb.ko 2023-11-28T16:53:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/legousbtower.ko 2023-11-28T16:53:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga 2023-11-28T16:53:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-11-28T16:53:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/trancevibrator.ko 2023-11-28T16:53:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usb3503.ko 2023-11-28T16:53:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usblcd.ko 2023-11-28T16:53:51.352 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/usbsevseg.ko 2023-11-28T16:53:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/uss720.ko 2023-11-28T16:53:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/misc/yurex.ko 2023-11-28T16:53:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy 2023-11-28T16:53:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/phy/phy-generic.ko 2023-11-28T16:53:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial 2023-11-28T16:53:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/aircable.ko 2023-11-28T16:53:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ark3116.ko 2023-11-28T16:53:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/belkin_sa.ko 2023-11-28T16:53:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ch341.ko 2023-11-28T16:53:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cp210x.ko 2023-11-28T16:53:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cyberjack.ko 2023-11-28T16:53:51.353 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/cypress_m8.ko 2023-11-28T16:53:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/digi_acceleport.ko 2023-11-28T16:53:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/empeg.ko 2023-11-28T16:53:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ftdi_sio.ko 2023-11-28T16:53:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/garmin_gps.ko 2023-11-28T16:53:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_edgeport.ko 2023-11-28T16:53:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/io_ti.ko 2023-11-28T16:53:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipaq.ko 2023-11-28T16:53:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ipw.ko 2023-11-28T16:53:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ir-usb.ko 2023-11-28T16:53:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-11-28T16:53:51.354 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan.ko 2023-11-28T16:53:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/keyspan_pda.ko 2023-11-28T16:53:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kl5kusb105.ko 2023-11-28T16:53:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/kobil_sct.ko 2023-11-28T16:53:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mct_u232.ko 2023-11-28T16:53:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7720.ko 2023-11-28T16:53:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/mos7840.ko 2023-11-28T16:53:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/navman.ko 2023-11-28T16:53:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/omninet.ko 2023-11-28T16:53:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/opticon.ko 2023-11-28T16:53:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/option.ko 2023-11-28T16:53:51.355 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/oti6858.ko 2023-11-28T16:53:51.356 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/pl2303.ko 2023-11-28T16:53:51.356 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcaux.ko 2023-11-28T16:53:51.356 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/qcserial.ko 2023-11-28T16:53:51.356 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/quatech2.ko 2023-11-28T16:53:51.356 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/safe_serial.ko 2023-11-28T16:53:51.356 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/sierra.ko 2023-11-28T16:53:51.356 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/spcp8x5.ko 2023-11-28T16:53:51.356 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ssu100.ko 2023-11-28T16:53:51.356 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/symbolserial.ko 2023-11-28T16:53:51.356 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-11-28T16:53:51.356 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-11-28T16:53:51.356 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_debug.ko 2023-11-28T16:53:51.357 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/usb_wwan.ko 2023-11-28T16:53:51.357 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/visor.ko 2023-11-28T16:53:51.357 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/whiteheat.ko 2023-11-28T16:53:51.357 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/serial/xsens_mt.ko 2023-11-28T16:53:51.357 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage 2023-11-28T16:53:51.357 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/uas.ko 2023-11-28T16:53:51.357 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-alauda.ko 2023-11-28T16:53:51.357 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-cypress.ko 2023-11-28T16:53:51.357 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-datafab.ko 2023-11-28T16:53:51.357 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-11-28T16:53:51.357 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-freecom.ko 2023-11-28T16:53:51.357 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-isd200.ko 2023-11-28T16:53:51.358 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-11-28T16:53:51.358 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-karma.ko 2023-11-28T16:53:51.358 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-onetouch.ko 2023-11-28T16:53:51.358 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-realtek.ko 2023-11-28T16:53:51.358 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr09.ko 2023-11-28T16:53:51.358 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-sddr55.ko 2023-11-28T16:53:51.358 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/ums-usbat.ko 2023-11-28T16:53:51.358 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/storage/usb-storage.ko 2023-11-28T16:53:51.358 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip 2023-11-28T16:53:51.358 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-core.ko 2023-11-28T16:53:51.358 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/usbip-host.ko 2023-11-28T16:53:51.359 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-11-28T16:53:51.359 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio 2023-11-28T16:53:51.359 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci 2023-11-28T16:53:51.359 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-11-28T16:53:51.359 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/pci/vfio-pci.ko 2023-11-28T16:53:51.359 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio.ko 2023-11-28T16:53:51.359 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-11-28T16:53:51.359 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost 2023-11-28T16:53:51.359 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost.ko 2023-11-28T16:53:51.359 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_iotlb.ko 2023-11-28T16:53:51.359 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_net.ko 2023-11-28T16:53:51.359 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/vhost/vhost_scsi.ko 2023-11-28T16:53:51.360 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video 2023-11-28T16:53:51.360 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight 2023-11-28T16:53:51.360 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/apple_bl.ko 2023-11-28T16:53:51.360 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/lcd.ko 2023-11-28T16:53:51.360 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/video/backlight/platform_lcd.ko 2023-11-28T16:53:51.360 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio 2023-11-28T16:53:51.360 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio.ko 2023-11-28T16:53:51.360 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_balloon.ko 2023-11-28T16:53:51.360 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_mmio.ko 2023-11-28T16:53:51.360 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci.ko 2023-11-28T16:53:51.360 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-11-28T16:53:51.360 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-11-28T16:53:51.361 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/virtio/virtio_ring.ko 2023-11-28T16:53:51.361 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1 2023-11-28T16:53:51.361 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters 2023-11-28T16:53:51.361 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2482.ko 2023-11-28T16:53:51.361 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/masters/ds2490.ko 2023-11-28T16:53:51.361 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves 2023-11-28T16:53:51.361 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-11-28T16:53:51.361 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-11-28T16:53:51.361 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-11-28T16:53:51.361 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-11-28T16:53:51.361 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-11-28T16:53:51.362 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-11-28T16:53:51.362 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-11-28T16:53:51.362 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-11-28T16:53:51.362 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-11-28T16:53:51.362 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_smem.ko 2023-11-28T16:53:51.362 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/slaves/w1_therm.ko 2023-11-28T16:53:51.362 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/w1/wire.ko 2023-11-28T16:53:51.362 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog 2023-11-28T16:53:51.362 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim1535_wdt.ko 2023-11-28T16:53:51.362 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/alim7101_wdt.ko 2023-11-28T16:53:51.362 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/f71808e_wdt.ko 2023-11-28T16:53:51.362 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/hpwdt.ko 2023-11-28T16:53:51.363 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/i6300esb.ko 2023-11-28T16:53:51.363 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-11-28T16:53:51.363 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/iTCO_wdt.ko 2023-11-28T16:53:51.363 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ib700wdt.ko 2023-11-28T16:53:51.363 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ibmasr.ko 2023-11-28T16:53:51.363 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-11-28T16:53:51.363 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it8712f_wdt.ko 2023-11-28T16:53:51.363 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/it87_wdt.ko 2023-11-28T16:53:51.363 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/machzwd.ko 2023-11-28T16:53:51.363 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/nv_tco.ko 2023-11-28T16:53:51.363 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_pci.ko 2023-11-28T16:53:51.364 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/pcwd_usb.ko 2023-11-28T16:53:51.364 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-11-28T16:53:51.364 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sch311x_wdt.ko 2023-11-28T16:53:51.364 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/softdog.ko 2023-11-28T16:53:51.364 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/sp5100_tco.ko 2023-11-28T16:53:51.364 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/via_wdt.ko 2023-11-28T16:53:51.364 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-11-28T16:53:51.364 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83877f_wdt.ko 2023-11-28T16:53:51.364 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/w83977f_wdt.ko 2023-11-28T16:53:51.364 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/wdt_pci.ko 2023-11-28T16:53:51.364 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/watchdog/xen_wdt.ko 2023-11-28T16:53:51.364 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen 2023-11-28T16:53:51.365 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-acpi-processor.ko 2023-11-28T16:53:51.365 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-evtchn.ko 2023-11-28T16:53:51.365 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntalloc.ko 2023-11-28T16:53:51.365 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-gntdev.ko 2023-11-28T16:53:51.365 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback 2023-11-28T16:53:51.365 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-11-28T16:53:51.365 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xen-privcmd.ko 2023-11-28T16:53:51.365 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs 2023-11-28T16:53:51.365 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/drivers/xen/xenfs/xenfs.ko 2023-11-28T16:53:51.365 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs 2023-11-28T16:53:51.365 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p 2023-11-28T16:53:51.366 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/9p/9p.ko 2023-11-28T16:53:51.366 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs 2023-11-28T16:53:51.366 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/affs/affs.ko 2023-11-28T16:53:51.366 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs 2023-11-28T16:53:51.366 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/befs/befs.ko 2023-11-28T16:53:51.366 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/binfmt_misc.ko 2023-11-28T16:53:51.366 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs 2023-11-28T16:53:51.366 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/btrfs/btrfs.ko 2023-11-28T16:53:51.366 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles 2023-11-28T16:53:51.366 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cachefiles/cachefiles.ko 2023-11-28T16:53:51.366 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph 2023-11-28T16:53:51.366 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ceph/ceph.ko 2023-11-28T16:53:51.367 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda 2023-11-28T16:53:51.367 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/coda/coda.ko 2023-11-28T16:53:51.367 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs 2023-11-28T16:53:51.367 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/cramfs/cramfs.ko 2023-11-28T16:53:51.367 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm 2023-11-28T16:53:51.367 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/dlm/dlm.ko 2023-11-28T16:53:51.367 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs 2023-11-28T16:53:51.367 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ecryptfs/ecryptfs.ko 2023-11-28T16:53:51.367 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs 2023-11-28T16:53:51.367 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/f2fs/f2fs.ko 2023-11-28T16:53:51.367 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat 2023-11-28T16:53:51.368 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/fat.ko 2023-11-28T16:53:51.368 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/msdos.ko 2023-11-28T16:53:51.368 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fat/vfat.ko 2023-11-28T16:53:51.368 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache 2023-11-28T16:53:51.368 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fscache/fscache.ko 2023-11-28T16:53:51.368 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse 2023-11-28T16:53:51.368 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/cuse.ko 2023-11-28T16:53:51.368 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/fuse/fuse.ko 2023-11-28T16:53:51.368 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2 2023-11-28T16:53:51.368 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/gfs2/gfs2.ko 2023-11-28T16:53:51.368 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs 2023-11-28T16:53:51.368 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfs/hfs.ko 2023-11-28T16:53:51.369 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus 2023-11-28T16:53:51.369 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/hfsplus/hfsplus.ko 2023-11-28T16:53:51.369 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs 2023-11-28T16:53:51.369 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/isofs/isofs.ko 2023-11-28T16:53:51.369 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs 2023-11-28T16:53:51.369 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/jfs/jfs.ko 2023-11-28T16:53:51.369 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd 2023-11-28T16:53:51.369 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/lockd/lockd.ko 2023-11-28T16:53:51.369 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix 2023-11-28T16:53:51.369 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/minix/minix.ko 2023-11-28T16:53:51.369 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs 2023-11-28T16:53:51.369 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/netfs/netfs.ko 2023-11-28T16:53:51.370 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs 2023-11-28T16:53:51.370 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout 2023-11-28T16:53:51.370 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-11-28T16:53:51.370 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout 2023-11-28T16:53:51.370 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-11-28T16:53:51.370 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout 2023-11-28T16:53:51.370 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-11-28T16:53:51.370 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfs.ko 2023-11-28T16:53:51.370 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv3.ko 2023-11-28T16:53:51.370 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs/nfsv4.ko 2023-11-28T16:53:51.370 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common 2023-11-28T16:53:51.371 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/grace.ko 2023-11-28T16:53:51.371 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfs_common/nfs_acl.ko 2023-11-28T16:53:51.371 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd 2023-11-28T16:53:51.371 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nfsd/nfsd.ko 2023-11-28T16:53:51.371 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2 2023-11-28T16:53:51.371 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nilfs2/nilfs2.ko 2023-11-28T16:53:51.371 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls 2023-11-28T16:53:51.371 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-celtic.ko 2023-11-28T16:53:51.371 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-centeuro.ko 2023-11-28T16:53:51.371 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-croatian.ko 2023-11-28T16:53:51.371 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-cyrillic.ko 2023-11-28T16:53:51.371 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-gaelic.ko 2023-11-28T16:53:51.372 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-greek.ko 2023-11-28T16:53:51.372 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-iceland.ko 2023-11-28T16:53:51.372 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-inuit.ko 2023-11-28T16:53:51.372 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-roman.ko 2023-11-28T16:53:51.372 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-romanian.ko 2023-11-28T16:53:51.372 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/mac-turkish.ko 2023-11-28T16:53:51.372 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1250.ko 2023-11-28T16:53:51.372 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1251.ko 2023-11-28T16:53:51.372 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp1255.ko 2023-11-28T16:53:51.372 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp737.ko 2023-11-28T16:53:51.372 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp775.ko 2023-11-28T16:53:51.373 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp850.ko 2023-11-28T16:53:51.373 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp852.ko 2023-11-28T16:53:51.373 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp855.ko 2023-11-28T16:53:51.373 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp857.ko 2023-11-28T16:53:51.373 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp860.ko 2023-11-28T16:53:51.373 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp861.ko 2023-11-28T16:53:51.373 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp862.ko 2023-11-28T16:53:51.373 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp863.ko 2023-11-28T16:53:51.373 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp864.ko 2023-11-28T16:53:51.373 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp865.ko 2023-11-28T16:53:51.373 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp866.ko 2023-11-28T16:53:51.373 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp869.ko 2023-11-28T16:53:51.374 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp874.ko 2023-11-28T16:53:51.374 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp932.ko 2023-11-28T16:53:51.374 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp936.ko 2023-11-28T16:53:51.374 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp949.ko 2023-11-28T16:53:51.374 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_cp950.ko 2023-11-28T16:53:51.374 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_euc-jp.ko 2023-11-28T16:53:51.374 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-1.ko 2023-11-28T16:53:51.374 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-13.ko 2023-11-28T16:53:51.374 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-14.ko 2023-11-28T16:53:51.374 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-15.ko 2023-11-28T16:53:51.374 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-2.ko 2023-11-28T16:53:51.375 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-3.ko 2023-11-28T16:53:51.375 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-4.ko 2023-11-28T16:53:51.375 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-5.ko 2023-11-28T16:53:51.375 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-6.ko 2023-11-28T16:53:51.375 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-7.ko 2023-11-28T16:53:51.375 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_iso8859-9.ko 2023-11-28T16:53:51.375 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-r.ko 2023-11-28T16:53:51.375 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-ru.ko 2023-11-28T16:53:51.375 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_koi8-u.ko 2023-11-28T16:53:51.375 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_ucs2_utils.ko 2023-11-28T16:53:51.375 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/nls/nls_utf8.ko 2023-11-28T16:53:51.375 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2 2023-11-28T16:53:51.376 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster 2023-11-28T16:53:51.376 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-11-28T16:53:51.376 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm 2023-11-28T16:53:51.376 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-11-28T16:53:51.376 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs 2023-11-28T16:53:51.376 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-11-28T16:53:51.376 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2.ko 2023-11-28T16:53:51.376 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-11-28T16:53:51.376 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-11-28T16:53:51.376 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-11-28T16:53:51.376 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs 2023-11-28T16:53:51.377 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/overlayfs/overlay.ko 2023-11-28T16:53:51.377 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore 2023-11-28T16:53:51.377 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/pstore/ramoops.ko 2023-11-28T16:53:51.377 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs 2023-11-28T16:53:51.377 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/reiserfs/reiserfs.ko 2023-11-28T16:53:51.377 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs 2023-11-28T16:53:51.377 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/romfs/romfs.ko 2023-11-28T16:53:51.377 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb 2023-11-28T16:53:51.377 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client 2023-11-28T16:53:51.377 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/client/cifs.ko 2023-11-28T16:53:51.377 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common 2023-11-28T16:53:51.378 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_arc4.ko 2023-11-28T16:53:51.378 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/smb/common/cifs_md4.ko 2023-11-28T16:53:51.378 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs 2023-11-28T16:53:51.378 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/squashfs/squashfs.ko 2023-11-28T16:53:51.378 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv 2023-11-28T16:53:51.378 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/sysv/sysv.ko 2023-11-28T16:53:51.378 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf 2023-11-28T16:53:51.378 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/udf/udf.ko 2023-11-28T16:53:51.378 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs 2023-11-28T16:53:51.378 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/ufs/ufs.ko 2023-11-28T16:53:51.378 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs 2023-11-28T16:53:51.378 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/fs/xfs/xfs.ko 2023-11-28T16:53:51.379 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel 2023-11-28T16:53:51.379 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu 2023-11-28T16:53:51.379 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/rcu/rcutorture.ko 2023-11-28T16:53:51.379 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/torture.ko 2023-11-28T16:53:51.379 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace 2023-11-28T16:53:51.379 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-11-28T16:53:51.379 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib 2023-11-28T16:53:51.379 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/asn1_encoder.ko 2023-11-28T16:53:51.379 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc-itu-t.ko 2023-11-28T16:53:51.379 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crc8.ko 2023-11-28T16:53:51.379 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto 2023-11-28T16:53:51.380 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libarc4.ko 2023-11-28T16:53:51.380 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/crypto/libdes.ko 2023-11-28T16:53:51.380 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/libcrc32c.ko 2023-11-28T16:53:51.380 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lru_cache.ko 2023-11-28T16:53:51.380 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4 2023-11-28T16:53:51.380 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4_compress.ko 2023-11-28T16:53:51.380 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/lz4/lz4hc_compress.ko 2023-11-28T16:53:51.380 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math 2023-11-28T16:53:51.380 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/math/cordic.ko 2023-11-28T16:53:51.380 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6 2023-11-28T16:53:51.380 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/raid6/raid6_pq.ko 2023-11-28T16:53:51.381 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon 2023-11-28T16:53:51.381 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/reed_solomon/reed_solomon.ko 2023-11-28T16:53:51.381 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_bm.ko 2023-11-28T16:53:51.381 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_fsm.ko 2023-11-28T16:53:51.381 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/ts_kmp.ko 2023-11-28T16:53:51.381 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd 2023-11-28T16:53:51.381 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/lib/zstd/zstd_compress.ko 2023-11-28T16:53:51.381 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm 2023-11-28T16:53:51.381 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/mm/hwpoison-inject.ko 2023-11-28T16:53:51.381 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net 2023-11-28T16:53:51.381 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan 2023-11-28T16:53:51.381 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/6lowpan.ko 2023-11-28T16:53:51.382 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_dest.ko 2023-11-28T16:53:51.382 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_fragment.ko 2023-11-28T16:53:51.382 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_hop.ko 2023-11-28T16:53:51.382 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_ipv6.ko 2023-11-28T16:53:51.382 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_mobility.ko 2023-11-28T16:53:51.382 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_routing.ko 2023-11-28T16:53:51.382 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/6lowpan/nhc_udp.ko 2023-11-28T16:53:51.382 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802 2023-11-28T16:53:51.382 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/garp.ko 2023-11-28T16:53:51.382 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/mrp.ko 2023-11-28T16:53:51.382 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/p8022.ko 2023-11-28T16:53:51.383 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/psnap.ko 2023-11-28T16:53:51.383 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/802/stp.ko 2023-11-28T16:53:51.383 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q 2023-11-28T16:53:51.383 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/8021q/8021q.ko 2023-11-28T16:53:51.383 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p 2023-11-28T16:53:51.383 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet.ko 2023-11-28T16:53:51.383 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_fd.ko 2023-11-28T16:53:51.383 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_rdma.ko 2023-11-28T16:53:51.383 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/9p/9pnet_virtio.ko 2023-11-28T16:53:51.383 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv 2023-11-28T16:53:51.383 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/batman-adv/batman-adv.ko 2023-11-28T16:53:51.383 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge 2023-11-28T16:53:51.384 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/br_netfilter.ko 2023-11-28T16:53:51.384 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/bridge.ko 2023-11-28T16:53:51.384 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter 2023-11-28T16:53:51.384 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-11-28T16:53:51.384 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_among.ko 2023-11-28T16:53:51.384 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arp.ko 2023-11-28T16:53:51.384 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-11-28T16:53:51.384 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-11-28T16:53:51.384 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip.ko 2023-11-28T16:53:51.384 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-11-28T16:53:51.384 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_limit.ko 2023-11-28T16:53:51.385 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_log.ko 2023-11-28T16:53:51.385 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark.ko 2023-11-28T16:53:51.385 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-11-28T16:53:51.385 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-11-28T16:53:51.385 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-11-28T16:53:51.385 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-11-28T16:53:51.385 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_snat.ko 2023-11-28T16:53:51.385 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_stp.ko 2023-11-28T16:53:51.385 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-11-28T16:53:51.385 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-11-28T16:53:51.385 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-11-28T16:53:51.385 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-11-28T16:53:51.386 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/ebtables.ko 2023-11-28T16:53:51.386 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-11-28T16:53:51.386 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-11-28T16:53:51.386 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-11-28T16:53:51.386 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph 2023-11-28T16:53:51.386 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ceph/libceph.ko 2023-11-28T16:53:51.386 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core 2023-11-28T16:53:51.386 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/failover.ko 2023-11-28T16:53:51.386 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/core/pktgen.ko 2023-11-28T16:53:51.386 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp 2023-11-28T16:53:51.386 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp.ko 2023-11-28T16:53:51.387 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_diag.ko 2023-11-28T16:53:51.387 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv4.ko 2023-11-28T16:53:51.387 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dccp/dccp_ipv6.ko 2023-11-28T16:53:51.387 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver 2023-11-28T16:53:51.387 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dns_resolver/dns_resolver.ko 2023-11-28T16:53:51.387 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa 2023-11-28T16:53:51.387 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/dsa_core.ko 2023-11-28T16:53:51.387 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_brcm.ko 2023-11-28T16:53:51.387 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_dsa.ko 2023-11-28T16:53:51.387 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_none.ko 2023-11-28T16:53:51.387 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/dsa/tag_trailer.ko 2023-11-28T16:53:51.388 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154 2023-11-28T16:53:51.388 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan 2023-11-28T16:53:51.388 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-11-28T16:53:51.388 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154.ko 2023-11-28T16:53:51.388 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ieee802154/ieee802154_socket.ko 2023-11-28T16:53:51.388 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4 2023-11-28T16:53:51.388 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ah4.ko 2023-11-28T16:53:51.388 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/esp4.ko 2023-11-28T16:53:51.388 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/fou.ko 2023-11-28T16:53:51.388 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/gre.ko 2023-11-28T16:53:51.388 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/inet_diag.ko 2023-11-28T16:53:51.388 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_gre.ko 2023-11-28T16:53:51.389 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_tunnel.ko 2023-11-28T16:53:51.389 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ip_vti.ko 2023-11-28T16:53:51.389 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipcomp.ko 2023-11-28T16:53:51.389 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/ipip.ko 2023-11-28T16:53:51.389 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter 2023-11-28T16:53:51.389 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arp_tables.ko 2023-11-28T16:53:51.389 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-11-28T16:53:51.389 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-11-28T16:53:51.389 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-11-28T16:53:51.389 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-11-28T16:53:51.389 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-11-28T16:53:51.390 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-11-28T16:53:51.390 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-11-28T16:53:51.390 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-11-28T16:53:51.390 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-11-28T16:53:51.390 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/iptable_security.ko 2023-11-28T16:53:51.390 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-11-28T16:53:51.390 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-11-28T16:53:51.390 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-11-28T16:53:51.390 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-11-28T16:53:51.390 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-11-28T16:53:51.390 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-11-28T16:53:51.390 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-11-28T16:53:51.391 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-11-28T16:53:51.391 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-11-28T16:53:51.391 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-11-28T16:53:51.391 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_bic.ko 2023-11-28T16:53:51.391 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_dctcp.ko 2023-11-28T16:53:51.391 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_diag.ko 2023-11-28T16:53:51.391 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_highspeed.ko 2023-11-28T16:53:51.391 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_htcp.ko 2023-11-28T16:53:51.391 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_hybla.ko 2023-11-28T16:53:51.391 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_illinois.ko 2023-11-28T16:53:51.391 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_lp.ko 2023-11-28T16:53:51.392 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_scalable.ko 2023-11-28T16:53:51.392 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_vegas.ko 2023-11-28T16:53:51.392 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_veno.ko 2023-11-28T16:53:51.392 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_westwood.ko 2023-11-28T16:53:51.392 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tcp_yeah.ko 2023-11-28T16:53:51.392 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/tunnel4.ko 2023-11-28T16:53:51.392 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_diag.ko 2023-11-28T16:53:51.392 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/udp_tunnel.ko 2023-11-28T16:53:51.392 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv4/xfrm4_tunnel.ko 2023-11-28T16:53:51.392 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6 2023-11-28T16:53:51.392 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ah6.ko 2023-11-28T16:53:51.393 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/esp6.ko 2023-11-28T16:53:51.393 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/fou6.ko 2023-11-28T16:53:51.393 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_tunnel.ko 2023-11-28T16:53:51.393 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-11-28T16:53:51.393 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ip6_vti.ko 2023-11-28T16:53:51.393 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/ipcomp6.ko 2023-11-28T16:53:51.393 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter 2023-11-28T16:53:51.393 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-11-28T16:53:51.393 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-11-28T16:53:51.393 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-11-28T16:53:51.393 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-11-28T16:53:51.393 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-11-28T16:53:51.394 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-11-28T16:53:51.394 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-11-28T16:53:51.394 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-11-28T16:53:51.394 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-11-28T16:53:51.394 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-11-28T16:53:51.394 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-11-28T16:53:51.394 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-11-28T16:53:51.394 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-11-28T16:53:51.394 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-11-28T16:53:51.394 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-11-28T16:53:51.394 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-11-28T16:53:51.394 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-11-28T16:53:51.395 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-11-28T16:53:51.395 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-11-28T16:53:51.395 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-11-28T16:53:51.395 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-11-28T16:53:51.395 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-11-28T16:53:51.395 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-11-28T16:53:51.395 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-11-28T16:53:51.395 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/sit.ko 2023-11-28T16:53:51.395 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/tunnel6.ko 2023-11-28T16:53:51.395 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/ipv6/xfrm6_tunnel.ko 2023-11-28T16:53:51.395 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key 2023-11-28T16:53:51.395 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/key/af_key.ko 2023-11-28T16:53:51.396 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp 2023-11-28T16:53:51.396 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_core.ko 2023-11-28T16:53:51.396 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_debugfs.ko 2023-11-28T16:53:51.396 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_eth.ko 2023-11-28T16:53:51.396 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip.ko 2023-11-28T16:53:51.396 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ip6.ko 2023-11-28T16:53:51.396 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_netlink.ko 2023-11-28T16:53:51.396 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/l2tp/l2tp_ppp.ko 2023-11-28T16:53:51.396 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc 2023-11-28T16:53:51.396 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/llc/llc.ko 2023-11-28T16:53:51.396 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154 2023-11-28T16:53:51.397 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mac802154/mac802154.ko 2023-11-28T16:53:51.397 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls 2023-11-28T16:53:51.397 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/mpls/mpls_gso.ko 2023-11-28T16:53:51.397 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter 2023-11-28T16:53:51.397 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset 2023-11-28T16:53:51.397 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set.ko 2023-11-28T16:53:51.397 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-11-28T16:53:51.397 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-11-28T16:53:51.397 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-11-28T16:53:51.397 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-11-28T16:53:51.397 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-11-28T16:53:51.397 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-11-28T16:53:51.398 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-11-28T16:53:51.398 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-11-28T16:53:51.398 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-11-28T16:53:51.398 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-11-28T16:53:51.398 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-11-28T16:53:51.398 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-11-28T16:53:51.398 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-11-28T16:53:51.398 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-11-28T16:53:51.398 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-11-28T16:53:51.398 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs 2023-11-28T16:53:51.398 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs.ko 2023-11-28T16:53:51.399 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-11-28T16:53:51.399 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-11-28T16:53:51.399 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-11-28T16:53:51.399 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-11-28T16:53:51.399 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-11-28T16:53:51.399 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-11-28T16:53:51.399 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-11-28T16:53:51.399 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-11-28T16:53:51.399 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-11-28T16:53:51.399 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-11-28T16:53:51.399 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-11-28T16:53:51.400 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-11-28T16:53:51.400 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-11-28T16:53:51.400 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conncount.ko 2023-11-28T16:53:51.400 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack.ko 2023-11-28T16:53:51.400 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-11-28T16:53:51.400 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-11-28T16:53:51.400 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-11-28T16:53:51.400 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_h323.ko 2023-11-28T16:53:51.400 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_irc.ko 2023-11-28T16:53:51.400 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-11-28T16:53:51.400 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-11-28T16:53:51.400 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-11-28T16:53:51.401 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sane.ko 2023-11-28T16:53:51.401 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_sip.ko 2023-11-28T16:53:51.401 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-11-28T16:53:51.401 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-11-28T16:53:51.401 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_dup_netdev.ko 2023-11-28T16:53:51.401 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table.ko 2023-11-28T16:53:51.401 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_flow_table_inet.ko 2023-11-28T16:53:51.401 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_log_syslog.ko 2023-11-28T16:53:51.401 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat.ko 2023-11-28T16:53:51.401 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_amanda.ko 2023-11-28T16:53:51.401 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_ftp.ko 2023-11-28T16:53:51.402 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_irc.ko 2023-11-28T16:53:51.402 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_sip.ko 2023-11-28T16:53:51.402 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_nat_tftp.ko 2023-11-28T16:53:51.402 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_synproxy_core.ko 2023-11-28T16:53:51.402 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nf_tables.ko 2023-11-28T16:53:51.402 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink.ko 2023-11-28T16:53:51.402 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_acct.ko 2023-11-28T16:53:51.402 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_log.ko 2023-11-28T16:53:51.402 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_osf.ko 2023-11-28T16:53:51.402 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nfnetlink_queue.ko 2023-11-28T16:53:51.402 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_chain_nat.ko 2023-11-28T16:53:51.402 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_compat.ko 2023-11-28T16:53:51.403 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_ct.ko 2023-11-28T16:53:51.403 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_dup_netdev.ko 2023-11-28T16:53:51.403 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fib.ko 2023-11-28T16:53:51.403 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_fwd_netdev.ko 2023-11-28T16:53:51.403 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_hash.ko 2023-11-28T16:53:51.403 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_limit.ko 2023-11-28T16:53:51.403 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_log.ko 2023-11-28T16:53:51.403 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_masq.ko 2023-11-28T16:53:51.403 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_nat.ko 2023-11-28T16:53:51.403 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_numgen.ko 2023-11-28T16:53:51.403 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_queue.ko 2023-11-28T16:53:51.404 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_quota.ko 2023-11-28T16:53:51.404 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_redir.ko 2023-11-28T16:53:51.404 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject.ko 2023-11-28T16:53:51.404 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_reject_inet.ko 2023-11-28T16:53:51.404 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_socket.ko 2023-11-28T16:53:51.404 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/nft_tunnel.ko 2023-11-28T16:53:51.404 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_AUDIT.ko 2023-11-28T16:53:51.404 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CHECKSUM.ko 2023-11-28T16:53:51.404 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CLASSIFY.ko 2023-11-28T16:53:51.404 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-11-28T16:53:51.404 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_CT.ko 2023-11-28T16:53:51.404 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_DSCP.ko 2023-11-28T16:53:51.405 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HL.ko 2023-11-28T16:53:51.405 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_HMARK.ko 2023-11-28T16:53:51.405 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_IDLETIMER.ko 2023-11-28T16:53:51.405 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LED.ko 2023-11-28T16:53:51.405 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_LOG.ko 2023-11-28T16:53:51.405 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_MASQUERADE.ko 2023-11-28T16:53:51.405 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NETMAP.ko 2023-11-28T16:53:51.405 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFLOG.ko 2023-11-28T16:53:51.405 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_NFQUEUE.ko 2023-11-28T16:53:51.405 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_RATEEST.ko 2023-11-28T16:53:51.405 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_REDIRECT.ko 2023-11-28T16:53:51.406 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_SECMARK.ko 2023-11-28T16:53:51.406 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPMSS.ko 2023-11-28T16:53:51.406 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-11-28T16:53:51.406 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TEE.ko 2023-11-28T16:53:51.406 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TPROXY.ko 2023-11-28T16:53:51.406 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_TRACE.ko 2023-11-28T16:53:51.406 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_addrtype.ko 2023-11-28T16:53:51.406 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_bpf.ko 2023-11-28T16:53:51.406 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cgroup.ko 2023-11-28T16:53:51.406 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cluster.ko 2023-11-28T16:53:51.406 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_comment.ko 2023-11-28T16:53:51.406 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connbytes.ko 2023-11-28T16:53:51.407 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlabel.ko 2023-11-28T16:53:51.407 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connlimit.ko 2023-11-28T16:53:51.407 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_connmark.ko 2023-11-28T16:53:51.407 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_conntrack.ko 2023-11-28T16:53:51.407 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_cpu.ko 2023-11-28T16:53:51.407 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dccp.ko 2023-11-28T16:53:51.407 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_devgroup.ko 2023-11-28T16:53:51.407 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_dscp.ko 2023-11-28T16:53:51.407 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ecn.ko 2023-11-28T16:53:51.407 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_esp.ko 2023-11-28T16:53:51.407 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hashlimit.ko 2023-11-28T16:53:51.408 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_helper.ko 2023-11-28T16:53:51.408 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_hl.ko 2023-11-28T16:53:51.408 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipcomp.ko 2023-11-28T16:53:51.408 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_iprange.ko 2023-11-28T16:53:51.408 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_ipvs.ko 2023-11-28T16:53:51.408 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_l2tp.ko 2023-11-28T16:53:51.408 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_length.ko 2023-11-28T16:53:51.408 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_limit.ko 2023-11-28T16:53:51.408 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mac.ko 2023-11-28T16:53:51.408 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_mark.ko 2023-11-28T16:53:51.408 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_multiport.ko 2023-11-28T16:53:51.408 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nat.ko 2023-11-28T16:53:51.409 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_nfacct.ko 2023-11-28T16:53:51.409 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_osf.ko 2023-11-28T16:53:51.409 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_owner.ko 2023-11-28T16:53:51.409 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_physdev.ko 2023-11-28T16:53:51.409 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_pkttype.ko 2023-11-28T16:53:51.409 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_policy.ko 2023-11-28T16:53:51.409 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_quota.ko 2023-11-28T16:53:51.409 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_rateest.ko 2023-11-28T16:53:51.409 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_realm.ko 2023-11-28T16:53:51.409 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_recent.ko 2023-11-28T16:53:51.409 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_sctp.ko 2023-11-28T16:53:51.409 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_set.ko 2023-11-28T16:53:51.410 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_socket.ko 2023-11-28T16:53:51.410 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_state.ko 2023-11-28T16:53:51.410 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_statistic.ko 2023-11-28T16:53:51.410 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_string.ko 2023-11-28T16:53:51.410 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_tcpmss.ko 2023-11-28T16:53:51.410 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_time.ko 2023-11-28T16:53:51.410 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netfilter/xt_u32.ko 2023-11-28T16:53:51.410 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink 2023-11-28T16:53:51.410 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/netlink/netlink_diag.ko 2023-11-28T16:53:51.410 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh 2023-11-28T16:53:51.410 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/nsh/nsh.ko 2023-11-28T16:53:51.410 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch 2023-11-28T16:53:51.411 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/openvswitch.ko 2023-11-28T16:53:51.411 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-geneve.ko 2023-11-28T16:53:51.411 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-gre.ko 2023-11-28T16:53:51.411 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/openvswitch/vport-vxlan.ko 2023-11-28T16:53:51.411 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet 2023-11-28T16:53:51.411 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/packet/af_packet_diag.ko 2023-11-28T16:53:51.411 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds 2023-11-28T16:53:51.411 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds.ko 2023-11-28T16:53:51.411 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_rdma.ko 2023-11-28T16:53:51.411 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/rds/rds_tcp.ko 2023-11-28T16:53:51.411 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched 2023-11-28T16:53:51.411 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_csum.ko 2023-11-28T16:53:51.412 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_gact.ko 2023-11-28T16:53:51.412 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_ipt.ko 2023-11-28T16:53:51.412 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_mirred.ko 2023-11-28T16:53:51.412 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_nat.ko 2023-11-28T16:53:51.412 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_pedit.ko 2023-11-28T16:53:51.412 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_police.ko 2023-11-28T16:53:51.412 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_simple.ko 2023-11-28T16:53:51.412 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_skbedit.ko 2023-11-28T16:53:51.412 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/act_vlan.ko 2023-11-28T16:53:51.412 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_basic.ko 2023-11-28T16:53:51.412 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_bpf.ko 2023-11-28T16:53:51.412 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_flow.ko 2023-11-28T16:53:51.413 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_fw.ko 2023-11-28T16:53:51.413 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_route.ko 2023-11-28T16:53:51.413 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/cls_u32.ko 2023-11-28T16:53:51.413 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_cmp.ko 2023-11-28T16:53:51.413 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_ipset.ko 2023-11-28T16:53:51.413 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_meta.ko 2023-11-28T16:53:51.413 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_nbyte.ko 2023-11-28T16:53:51.413 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_text.ko 2023-11-28T16:53:51.413 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/em_u32.ko 2023-11-28T16:53:51.413 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_choke.ko 2023-11-28T16:53:51.413 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_codel.ko 2023-11-28T16:53:51.413 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_drr.ko 2023-11-28T16:53:51.414 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_fq.ko 2023-11-28T16:53:51.414 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_gred.ko 2023-11-28T16:53:51.414 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hfsc.ko 2023-11-28T16:53:51.414 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_hhf.ko 2023-11-28T16:53:51.414 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_htb.ko 2023-11-28T16:53:51.414 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_ingress.ko 2023-11-28T16:53:51.414 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio.ko 2023-11-28T16:53:51.414 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_mqprio_lib.ko 2023-11-28T16:53:51.414 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_multiq.ko 2023-11-28T16:53:51.414 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_netem.ko 2023-11-28T16:53:51.414 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_pie.ko 2023-11-28T16:53:51.415 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_plug.ko 2023-11-28T16:53:51.415 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_prio.ko 2023-11-28T16:53:51.415 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_qfq.ko 2023-11-28T16:53:51.415 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_red.ko 2023-11-28T16:53:51.415 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfb.ko 2023-11-28T16:53:51.415 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_sfq.ko 2023-11-28T16:53:51.415 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_tbf.ko 2023-11-28T16:53:51.415 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sched/sch_teql.ko 2023-11-28T16:53:51.415 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp 2023-11-28T16:53:51.415 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp.ko 2023-11-28T16:53:51.415 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sctp/sctp_diag.ko 2023-11-28T16:53:51.415 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc 2023-11-28T16:53:51.415 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss 2023-11-28T16:53:51.416 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-11-28T16:53:51.416 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-11-28T16:53:51.416 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/sunrpc.ko 2023-11-28T16:53:51.416 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma 2023-11-28T16:53:51.416 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-11-28T16:53:51.416 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc 2023-11-28T16:53:51.416 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/diag.ko 2023-11-28T16:53:51.416 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/tipc/tipc.ko 2023-11-28T16:53:51.416 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix 2023-11-28T16:53:51.416 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/unix/unix_diag.ko 2023-11-28T16:53:51.416 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock 2023-11-28T16:53:51.416 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-11-28T16:53:51.417 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-11-28T16:53:51.417 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock.ko 2023-11-28T16:53:51.417 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_diag.ko 2023-11-28T16:53:51.417 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/vmw_vsock/vsock_loopback.ko 2023-11-28T16:53:51.417 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm 2023-11-28T16:53:51.417 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/net/xfrm/xfrm_ipcomp.ko 2023-11-28T16:53:51.417 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security 2023-11-28T16:53:51.417 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys 2023-11-28T16:53:51.417 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys 2023-11-28T16:53:51.417 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-11-28T16:53:51.417 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys 2023-11-28T16:53:51.418 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/security/keys/trusted-keys/trusted.ko 2023-11-28T16:53:51.418 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt 2023-11-28T16:53:51.418 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib 2023-11-28T16:53:51.418 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/kernel/virt/lib/irqbypass.ko 2023-11-28T16:53:51.418 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias 2023-11-28T16:53:51.418 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.alias.bin 2023-11-28T16:53:51.418 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin 2023-11-28T16:53:51.418 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.alias.bin 2023-11-28T16:53:51.418 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.bin 2023-11-28T16:53:51.418 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.builtin.modinfo 2023-11-28T16:53:51.418 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep 2023-11-28T16:53:51.418 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.dep.bin 2023-11-28T16:53:51.419 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.devname 2023-11-28T16:53:51.419 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.order 2023-11-28T16:53:51.419 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.softdep 2023-11-28T16:53:51.419 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols 2023-11-28T16:53:51.419 INFO:teuthology.orchestra.run.smithi116.stdout:/lib/modules/6.6.0-g8247bbb5b470/modules.symbols.bin 2023-11-28T16:53:51.419 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-g8247bbb5b470 2023-11-28T16:53:51.420 DEBUG:teuthology.orchestra.run.smithi116:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-11-28T16:53:51.442 INFO:teuthology.orchestra.run.smithi116.stdout:postinstall scriptlet (using /bin/sh): 2023-11-28T16:53:51.442 INFO:teuthology.orchestra.run.smithi116.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-g8247bbb5b470 -a -r /boot/System.map-6.6.0-g8247bbb5b470 ]; then 2023-11-28T16:53:51.442 INFO:teuthology.orchestra.run.smithi116.stdout:cp /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm 2023-11-28T16:53:51.443 INFO:teuthology.orchestra.run.smithi116.stdout:cp /boot/System.map-6.6.0-g8247bbb5b470 /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T16:53:51.443 INFO:teuthology.orchestra.run.smithi116.stdout:rm -f /boot/vmlinuz-6.6.0-g8247bbb5b470 /boot/System.map-6.6.0-g8247bbb5b470 2023-11-28T16:53:51.443 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/installkernel 6.6.0-g8247bbb5b470 /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T16:53:51.443 INFO:teuthology.orchestra.run.smithi116.stdout:rm -f /boot/.vmlinuz-6.6.0-g8247bbb5b470-rpm /boot/.System.map-6.6.0-g8247bbb5b470-rpm 2023-11-28T16:53:51.443 INFO:teuthology.orchestra.run.smithi116.stdout:fi 2023-11-28T16:53:51.443 INFO:teuthology.orchestra.run.smithi116.stdout:preuninstall scriptlet (using /bin/sh): 2023-11-28T16:53:51.443 INFO:teuthology.orchestra.run.smithi116.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-11-28T16:53:51.443 INFO:teuthology.orchestra.run.smithi116.stdout:new-kernel-pkg --remove 6.6.0-g8247bbb5b470 --rminitrd --initrdfile=/boot/initramfs-6.6.0-g8247bbb5b470.img 2023-11-28T16:53:51.443 INFO:teuthology.orchestra.run.smithi116.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-11-28T16:53:51.443 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-install remove 6.6.0-g8247bbb5b470 2023-11-28T16:53:51.443 INFO:teuthology.orchestra.run.smithi116.stdout:fi 2023-11-28T16:53:51.443 INFO:teuthology.orchestra.run.smithi116.stdout:postuninstall scriptlet (using /bin/sh): 2023-11-28T16:53:51.444 INFO:teuthology.orchestra.run.smithi116.stdout:if [ -x /sbin/update-bootloader ]; then 2023-11-28T16:53:51.444 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/update-bootloader --remove 6.6.0-g8247bbb5b470 2023-11-28T16:53:51.444 INFO:teuthology.orchestra.run.smithi116.stdout:fi 2023-11-28T16:53:51.445 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2023-11-28T16:53:51.523 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2023-11-28T16:53:51.523 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2023-11-28T16:53:51.523 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2023-11-28T16:53:51.523 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 68.el9 2023-11-28T16:53:51.523 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2023-11-28T16:53:51.523 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-28T16:53:51.523 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2023-11-28T16:53:51.523 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8264254 2023-11-28T16:53:51.523 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2023-11-28T16:53:51.524 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-28T16:53:51.524 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-28T16:53:51.524 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-28T16:53:51.524 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-28T16:53:51.524 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2023-11-28T16:53:51.524 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2023-11-28T16:53:51.524 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-28T16:53:51.524 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2023-11-28T16:53:51.524 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2023-11-28T16:53:51.524 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T16:53:51.524 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-28T16:53:51.524 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-28T16:53:51.525 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-28T16:53:51.525 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2023-11-28T16:53:51.525 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T16:53:51.525 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2023-11-28T16:53:51.525 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-28T16:53:51.525 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 6.6.0-g8247bbb5b470 2023-11-28T16:53:51.525 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-28T16:53:52.105 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2023-11-28T16:53:53.535 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-28T16:53:53.572 INFO:teuthology.orchestra.run.smithi116.stderr:done 2023-11-28T16:53:53.583 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2023-11-28T16:53:53.599 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.tcuZGUPE9x 2023-11-28T16:53:53.599 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tcuZGUPE9x 2023-11-28T16:53:53.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.tcuZGUPE9x 2023-11-28T16:53:53.783 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.tcuZGUPE9x is 6KB 2023-11-28T16:53:53.833 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.tcuZGUPE9x 2023-11-28T16:53:53.848 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2023-11-28T16:53:53.912 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-28T16:53:53.913 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-28T16:53:53.913 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-28T16:53:53.913 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470.conf 2023-11-28T16:53:53.914 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.6.0-g8247bbb5b470 2023-11-28T16:53:54.077 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2023-11-28T16:54:15.062 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-11-28T16:54:15.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-28T16:54:15.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T16:54:24.088 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-28T16:54:24.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-28T16:54:24.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T16:54:49.698 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-11-28T16:54:58.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-28T16:54:58.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T16:55:17.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-11-28T16:55:24.098 DEBUG:teuthology.orchestra.remote:timed out 2023-11-28T16:55:29.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-28T16:55:29.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T16:55:32.162 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2023-11-28T16:55:33.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-28T16:55:33.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T16:55:40.785 DEBUG:teuthology.orchestra.run.smithi116:> true 2023-11-28T16:55:41.422 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-28T16:55:41.422 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.6.0-g8247bbb5b470"... 2023-11-28T16:55:41.423 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2023-11-28T16:55:41.464 INFO:teuthology.orchestra.run.smithi116.stdout:6.6.0-g8247bbb5b470 2023-11-28T16:55:41.464 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-g8247bbb5b470 vs 6.6.0-g8247bbb5b470 2023-11-28T16:55:41.464 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-28T16:55:41.464 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-28T16:55:42.465 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-28T16:55:42.465 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-28T16:55:42.582 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2023-11-28T16:55:42.634 DEBUG:teuthology.parallel:result is None 2023-11-28T16:55:47.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-28T16:55:47.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T16:55:47.574 DEBUG:teuthology.orchestra.run.smithi156:> true 2023-11-28T16:55:48.167 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-28T16:55:48.167 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.6.0-g8247bbb5b470"... 2023-11-28T16:55:48.167 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2023-11-28T16:55:48.207 INFO:teuthology.orchestra.run.smithi156.stdout:6.6.0-g8247bbb5b470 2023-11-28T16:55:48.207 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-g8247bbb5b470 vs 6.6.0-g8247bbb5b470 2023-11-28T16:55:48.207 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-28T16:55:48.207 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-28T16:55:49.208 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-11-28T16:55:49.208 DEBUG:teuthology.orchestra.run.smithi156:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-28T16:55:49.343 INFO:teuthology.orchestra.run.smithi156.stdout:ttyS1 2023-11-28T16:55:49.393 DEBUG:teuthology.parallel:result is None 2023-11-28T16:55:49.393 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-28T16:55:49.400 INFO:teuthology.task.internal:Creating test directory... 2023-11-28T16:55:49.400 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T16:55:49.405 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T16:55:49.444 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-28T16:55:49.452 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-28T16:55:49.461 INFO:teuthology.task.internal:Creating archive directory... 2023-11-28T16:55:49.462 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T16:55:49.489 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T16:55:49.548 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-28T16:55:49.555 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-28T16:55:49.555 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T16:55:49.577 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T16:55:49.653 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T16:55:49.661 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T16:55:49.691 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T16:55:49.696 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T16:55:49.700 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-28T16:55:49.707 INFO:teuthology.task.internal:Configuring sudo... 2023-11-28T16:55:49.707 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T16:55:49.712 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T16:55:49.792 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-28T16:55:49.801 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-28T16:55:49.801 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T16:55:49.806 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T16:55:49.855 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T16:55:49.952 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T16:55:50.038 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T16:55:50.107 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T16:55:50.194 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T16:55:50.194 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T16:55:50.264 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T16:55:50.343 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T16:55:50.488 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T16:55:50.570 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T16:55:50.660 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T16:55:50.660 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T16:55:50.728 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2023-11-28T16:55:50.733 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2023-11-28T16:55:50.829 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T16:55:50.926 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T16:55:51.323 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-28T16:55:51.332 INFO:teuthology.task.internal:Starting timer... 2023-11-28T16:55:51.332 INFO:teuthology.run_tasks:Running task pcp... 2023-11-28T16:55:51.357 INFO:teuthology.run_tasks:Running task selinux... 2023-11-28T16:55:51.370 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2023-11-28T16:55:51.509 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2023-11-28T16:55:51.517 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2023-11-28T16:55:51.721 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2023-11-28T16:55:51.728 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-28T16:55:51.729 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2023-11-28T16:55:51.783 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2023-11-28T16:55:51.783 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2023-11-28T16:55:51.872 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2023-11-28T16:55:51.873 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi156.front.sepia.ceph.com': 'permissive'} 2023-11-28T16:55:51.873 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T16:55:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T16:55:51.956 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T16:55:52.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T16:55:52.101 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-28T16:55:52.101 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2023-11-28T16:55:52.187 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2023-11-28T16:55:52.333 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-28T16:55:52.343 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-28T16:55:52.343 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-28T16:55:52.360 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-28T16:55:52.362 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com,smithi156.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-28T17:05:50.541 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi156.front.sepia.ceph.com')] 2023-11-28T17:05:50.542 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-28T17:05:50.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T17:05:50.683 DEBUG:teuthology.orchestra.run.smithi116:> true 2023-11-28T17:05:50.833 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-11-28T17:05:50.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-28T17:05:50.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T17:05:50.976 DEBUG:teuthology.orchestra.run.smithi156:> true 2023-11-28T17:05:51.136 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-11-28T17:05:51.137 INFO:teuthology.run_tasks:Running task clock... 2023-11-28T17:05:51.149 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-28T17:05:51.150 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T17:05:51.150 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T17:05:51.160 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T17:05:51.161 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T17:05:51.261 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-28T17:05:51.262 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-28T17:05:51.311 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-28T17:05:51.314 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-28T17:05:51.399 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2023-11-28T17:05:51.407 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2023-11-28T17:05:51.439 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2023-11-28T17:05:51.448 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2023-11-28T17:05:51.491 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-28T17:05:51.499 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-28T17:05:51.544 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-28T17:05:51.554 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-28T17:05:51.670 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2023-11-28T17:05:51.673 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2023-11-28T17:05:51.680 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T17:05:51.680 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2023-11-28T17:05:51.680 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:05:51.680 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:05:51.681 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:05:51.681 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:05:51.681 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T17:05:51.682 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2023-11-28T17:05:51.682 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:05:51.682 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:05:51.682 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:05:51.682 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:05:51.684 INFO:teuthology.run_tasks:Running task install... 2023-11-28T17:05:51.693 DEBUG:teuthology.task.install:project ceph 2023-11-28T17:05:51.693 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}} 2023-11-28T17:05:51.694 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'} 2023-11-28T17:05:51.694 INFO:teuthology.task.install:Using flavor: default 2023-11-28T17:05:51.701 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-28T17:05:51.701 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-11-28T17:05:51.701 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'tag': None, 'wait_for_package': False} 2023-11-28T17:05:51.702 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T17:05:51.704 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'tag': None, 'wait_for_package': False} 2023-11-28T17:05:51.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T17:05:51.852 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/ 2023-11-28T17:05:51.852 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7531.g47c1c547 2023-11-28T17:05:51.921 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/ 2023-11-28T17:05:51.921 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7531.g47c1c547 2023-11-28T17:05:51.971 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-28T17:05:51.971 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:05:51.972 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-28T17:05:52.074 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-28T17:05:52.074 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/47c1c547372ed5bff7984647dc200bac0e019caf/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-28T17:05:52.089 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231127.102654/47c1c547372ed5bff7984647dc200bac0e019caf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-28T17:05:52.090 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:05:52.090 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-28T17:05:52.192 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-28T17:05:52.193 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/47c1c547372ed5bff7984647dc200bac0e019caf/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-28T17:05:52.215 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-28T17:05:52.331 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-28T17:05:52.342 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-28T17:05:52.431 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2023-11-28T17:05:52.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2023-11-28T17:05:52.461 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-28T17:05:52.553 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2023-11-28T17:05:52.556 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2023-11-28T17:05:52.787 INFO:teuthology.orchestra.run.smithi116.stdout:73 files removed 2023-11-28T17:05:52.825 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-28T17:05:52.912 INFO:teuthology.orchestra.run.smithi156.stdout:73 files removed 2023-11-28T17:05:52.950 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-28T17:05:53.831 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 210 kB/s | 78 kB 00:00 2023-11-28T17:05:53.917 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 134 kB/s | 78 kB 00:00 2023-11-28T17:05:54.425 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 23 kB/s | 9.5 kB 00:00 2023-11-28T17:05:54.447 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 18 kB/s | 9.5 kB 00:00 2023-11-28T17:05:54.918 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2023-11-28T17:05:54.967 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2023-11-28T17:05:55.638 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 9.9 MB/s | 6.4 MB 00:00 2023-11-28T17:05:55.687 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - AppStream 9.8 MB/s | 6.4 MB 00:00 2023-11-28T17:05:57.297 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2023-11-28T17:05:57.513 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2023-11-28T17:05:58.643 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 8.1 MB/s | 7.9 MB 00:00 2023-11-28T17:05:59.359 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 7.9 MB 00:01 2023-11-28T17:06:01.694 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 10 MB/s | 18 MB 00:01 2023-11-28T17:06:02.918 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 7.9 MB/s | 18 MB 00:02 2023-11-28T17:06:07.927 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 5.8 MB 00:01 2023-11-28T17:06:09.353 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 5.8 MB 00:04 2023-11-28T17:06:09.587 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 15 kB 00:00 2023-11-28T17:06:10.621 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 24 MB/s | 20 MB 00:00 2023-11-28T17:06:11.104 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2023-11-28T17:06:14.836 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 5.6 MB/s | 20 MB 00:03 2023-11-28T17:06:16.187 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2023-11-28T17:06:19.078 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-28T17:06:19.080 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-28T17:06:19.101 INFO:teuthology.orchestra.run.smithi156.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-11-28T17:06:19.102 INFO:teuthology.orchestra.run.smithi156.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-11-28T17:06:19.104 INFO:teuthology.orchestra.run.smithi156.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-11-28T17:06:19.105 INFO:teuthology.orchestra.run.smithi156.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-11-28T17:06:19.108 INFO:teuthology.orchestra.run.smithi156.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-11-28T17:06:19.112 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-11-28T17:06:19.113 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-11-28T17:06:19.193 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-11-28T17:06:19.207 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2023-11-28T17:06:19.207 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-11-28T17:06:19.207 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2023-11-28T17:06:19.208 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-11-28T17:06:19.208 INFO:teuthology.orchestra.run.smithi156.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-11-28T17:06:19.208 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 6.6 k 2023-11-28T17:06:19.208 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 5.2 M 2023-11-28T17:06:19.208 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 859 k 2023-11-28T17:06:19.208 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 141 k 2023-11-28T17:06:19.208 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 1.5 M 2023-11-28T17:06:19.208 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 133 k 2023-11-28T17:06:19.208 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 1.8 M 2023-11-28T17:06:19.208 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 7.4 M 2023-11-28T17:06:19.208 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 50 k 2023-11-28T17:06:19.209 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 8.7 M 2023-11-28T17:06:19.209 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 41 M 2023-11-28T17:06:19.209 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 262 k 2023-11-28T17:06:19.209 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 773 k 2023-11-28T17:06:19.209 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 217 k 2023-11-28T17:06:19.209 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 18 k 2023-11-28T17:06:19.209 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel x86_64 0.190-1.el9 CentOS-AppStream 21 k 2023-11-28T17:06:19.209 INFO:teuthology.orchestra.run.smithi156.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-11-28T17:06:19.209 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-11-28T17:06:19.209 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 31 k 2023-11-28T17:06:19.210 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 719 k 2023-11-28T17:06:19.210 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 126 k 2023-11-28T17:06:19.210 INFO:teuthology.orchestra.run.smithi156.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-11-28T17:06:19.210 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2023-11-28T17:06:19.210 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel x86_64 1:3.0.7-25.el9 appstream 4.1 M 2023-11-28T17:06:19.210 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-11-28T17:06:19.210 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-11-28T17:06:19.210 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-11-28T17:06:19.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 162 k 2023-11-28T17:06:19.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 321 k 2023-11-28T17:06:19.210 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 298 k 2023-11-28T17:06:19.211 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 99 k 2023-11-28T17:06:19.211 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 85 k 2023-11-28T17:06:19.211 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 3.0 M 2023-11-28T17:06:19.211 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 170 k 2023-11-28T17:06:19.211 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 CentOS-AppStream 85 k 2023-11-28T17:06:19.211 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2023-11-28T17:06:19.211 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager x86_64 1:1.45.6-1.el9 CentOS-BaseOS 2.2 M 2023-11-28T17:06:19.211 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-libnm x86_64 1:1.45.6-1.el9 CentOS-BaseOS 1.8 M 2023-11-28T17:06:19.211 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-team x86_64 1:1.45.6-1.el9 CentOS-BaseOS 42 k 2023-11-28T17:06:19.211 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-tui x86_64 1:1.45.6-1.el9 CentOS-BaseOS 245 k 2023-11-28T17:06:19.212 INFO:teuthology.orchestra.run.smithi156.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-11-28T17:06:19.212 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-debuginfod-client x86_64 0.190-1.el9 CentOS-BaseOS 36 k 2023-11-28T17:06:19.212 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf x86_64 0.190-1.el9 CentOS-BaseOS 192 k 2023-11-28T17:06:19.212 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libs x86_64 0.190-1.el9 CentOS-BaseOS 255 k 2023-11-28T17:06:19.212 INFO:teuthology.orchestra.run.smithi156.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-11-28T17:06:19.212 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-11-28T17:06:19.212 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 3.3 M 2023-11-28T17:06:19.212 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 3.0 M 2023-11-28T17:06:19.212 INFO:teuthology.orchestra.run.smithi156.stdout: openssl x86_64 1:3.0.7-25.el9 baseos 1.2 M 2023-11-28T17:06:19.212 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs x86_64 1:3.0.7-25.el9 baseos 2.2 M 2023-11-28T17:06:19.213 INFO:teuthology.orchestra.run.smithi156.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-11-28T17:06:19.213 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-11-28T17:06:19.213 INFO:teuthology.orchestra.run.smithi156.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2023-11-28T17:06:19.213 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-28T17:06:19.213 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 19 M 2023-11-28T17:06:19.213 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 28 k 2023-11-28T17:06:19.213 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 2.1 M 2023-11-28T17:06:19.213 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 243 k 2023-11-28T17:06:19.213 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 4.4 M 2023-11-28T17:06:19.213 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 18 M 2023-11-28T17:06:19.214 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 15 k 2023-11-28T17:06:19.214 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 24 k 2023-11-28T17:06:19.215 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-28T17:06:19.215 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-28T17:06:19.215 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-28T17:06:19.215 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-28T17:06:19.215 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-28T17:06:19.215 INFO:teuthology.orchestra.run.smithi156.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-28T17:06:19.215 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-28T17:06:19.216 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-28T17:06:19.216 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 163 k 2023-11-28T17:06:19.216 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-28T17:06:19.216 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-28T17:06:19.216 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-28T17:06:19.216 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-28T17:06:19.216 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-28T17:06:19.216 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 476 k 2023-11-28T17:06:19.216 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-28T17:06:19.216 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 4.8 M 2023-11-28T17:06:19.216 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-28T17:06:19.216 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-11-28T17:06:19.216 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-28T17:06:19.217 INFO:teuthology.orchestra.run.smithi156.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-11-28T17:06:19.217 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-11-28T17:06:19.217 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-11-28T17:06:19.217 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-28T17:06:19.217 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-28T17:06:19.217 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-28T17:06:19.217 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-28T17:06:19.218 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-11-28T17:06:19.218 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-28T17:06:19.218 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-28T17:06:19.218 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-28T17:06:19.218 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-28T17:06:19.218 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 45 k 2023-11-28T17:06:19.218 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 129 k 2023-11-28T17:06:19.218 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-28T17:06:19.218 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-28T17:06:19.219 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-28T17:06:19.219 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-28T17:06:19.219 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-28T17:06:19.219 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-28T17:06:19.219 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-28T17:06:19.219 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-28T17:06:19.219 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-28T17:06:19.219 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-28T17:06:19.219 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-28T17:06:19.219 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-28T17:06:19.219 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-28T17:06:19.219 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-28T17:06:19.220 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-28T17:06:19.220 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-28T17:06:19.220 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-28T17:06:19.220 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-28T17:06:19.220 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-28T17:06:19.220 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-28T17:06:19.220 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-28T17:06:19.220 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-28T17:06:19.220 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-28T17:06:19.220 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-28T17:06:19.220 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-28T17:06:19.221 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-28T17:06:19.221 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-28T17:06:19.221 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-28T17:06:19.221 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-28T17:06:19.221 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-28T17:06:19.221 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-28T17:06:19.221 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-28T17:06:19.221 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-28T17:06:19.221 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-28T17:06:19.222 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-28T17:06:19.222 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-28T17:06:19.222 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-28T17:06:19.222 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-28T17:06:19.222 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-28T17:06:19.222 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-28T17:06:19.222 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-28T17:06:19.222 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-28T17:06:19.222 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-28T17:06:19.222 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-28T17:06:19.222 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-28T17:06:19.223 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-28T17:06:19.223 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-28T17:06:19.223 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-28T17:06:19.223 INFO:teuthology.orchestra.run.smithi156.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-28T17:06:19.223 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-28T17:06:19.223 INFO:teuthology.orchestra.run.smithi156.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-28T17:06:19.223 INFO:teuthology.orchestra.run.smithi156.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-11-28T17:06:19.223 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-28T17:06:19.223 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2023-11-28T17:06:19.223 INFO:teuthology.orchestra.run.smithi156.stdout:Installing weak dependencies: 2023-11-28T17:06:19.223 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-28T17:06:19.224 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:06:19.224 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-11-28T17:06:19.224 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2023-11-28T17:06:19.224 INFO:teuthology.orchestra.run.smithi156.stdout:Install 135 Packages 2023-11-28T17:06:19.224 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 15 Packages 2023-11-28T17:06:19.224 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:06:19.224 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 205 M 2023-11-28T17:06:19.225 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-11-28T17:06:20.272 INFO:teuthology.orchestra.run.smithi156.stdout:(1/150): ceph-18.0.0-7531.g47c1c547.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2023-11-28T17:06:20.422 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2023-11-28T17:06:20.839 INFO:teuthology.orchestra.run.smithi156.stdout:(2/150): ceph-base-18.0.0-7531.g47c1c547.el9.x8 5.8 MB/s | 5.2 MB 00:00 2023-11-28T17:06:20.931 INFO:teuthology.orchestra.run.smithi156.stdout:(3/150): ceph-immutable-object-cache-18.0.0-753 1.5 MB/s | 141 kB 00:00 2023-11-28T17:06:21.014 INFO:teuthology.orchestra.run.smithi156.stdout:(4/150): ceph-fuse-18.0.0-7531.g47c1c547.el9.x8 1.1 MB/s | 859 kB 00:00 2023-11-28T17:06:21.215 INFO:teuthology.orchestra.run.smithi156.stdout:(5/150): ceph-mds-18.0.0-7531.g47c1c547.el9.x86 7.4 MB/s | 2.1 MB 00:00 2023-11-28T17:06:21.465 INFO:teuthology.orchestra.run.smithi156.stdout:(6/150): ceph-common-18.0.0-7531.g47c1c547.el9. 12 MB/s | 19 MB 00:01 2023-11-28T17:06:21.665 INFO:teuthology.orchestra.run.smithi156.stdout:(7/150): ceph-mon-18.0.0-7531.g47c1c547.el9.x86 9.8 MB/s | 4.4 MB 00:00 2023-11-28T17:06:21.757 INFO:teuthology.orchestra.run.smithi156.stdout:(8/150): ceph-mgr-18.0.0-7531.g47c1c547.el9.x86 2.0 MB/s | 1.5 MB 00:00 2023-11-28T17:06:21.874 INFO:teuthology.orchestra.run.smithi156.stdout:(9/150): ceph-selinux-18.0.0-7531.g47c1c547.el9 208 kB/s | 24 kB 00:00 2023-11-28T17:06:22.583 INFO:teuthology.orchestra.run.smithi156.stdout:(10/150): ceph-radosgw-18.0.0-7531.g47c1c547.el 9.5 MB/s | 8.7 MB 00:00 2023-11-28T17:06:22.667 INFO:teuthology.orchestra.run.smithi156.stdout:(11/150): cephfs-mirror-18.0.0-7531.g47c1c547.e 2.6 MB/s | 217 kB 00:00 2023-11-28T17:06:22.909 INFO:teuthology.orchestra.run.smithi156.stdout:(12/150): ceph-osd-18.0.0-7531.g47c1c547.el9.x8 12 MB/s | 18 MB 00:01 2023-11-28T17:06:22.934 INFO:teuthology.orchestra.run.smithi156.stdout:(13/150): libcephfs-devel-18.0.0-7531.g47c1c547 117 kB/s | 31 kB 00:00 2023-11-28T17:06:22.984 INFO:teuthology.orchestra.run.smithi156.stdout:(14/150): libcephsqlite-18.0.0-7531.g47c1c547.e 3.3 MB/s | 163 kB 00:00 2023-11-28T17:06:23.026 INFO:teuthology.orchestra.run.smithi156.stdout:(15/150): libcephfs2-18.0.0-7531.g47c1c547.el9. 6.1 MB/s | 719 kB 00:00 2023-11-28T17:06:23.059 INFO:teuthology.orchestra.run.smithi156.stdout:(16/150): librados-devel-18.0.0-7531.g47c1c547. 1.7 MB/s | 126 kB 00:00 2023-11-28T17:06:23.101 INFO:teuthology.orchestra.run.smithi156.stdout:(17/150): libradosstriper1-18.0.0-7531.g47c1c54 6.3 MB/s | 476 kB 00:00 2023-11-28T17:06:23.168 INFO:teuthology.orchestra.run.smithi156.stdout:(18/150): python3-ceph-argparse-18.0.0-7531.g47 678 kB/s | 45 kB 00:00 2023-11-28T17:06:23.244 INFO:teuthology.orchestra.run.smithi156.stdout:(19/150): python3-ceph-common-18.0.0-7531.g47c1 1.7 MB/s | 129 kB 00:00 2023-11-28T17:06:23.305 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-28T17:06:23.307 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-11-28T17:06:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:(20/150): python3-cephfs-18.0.0-7531.g47c1c547. 2.1 MB/s | 162 kB 00:00 2023-11-28T17:06:23.328 INFO:teuthology.orchestra.run.smithi116.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-11-28T17:06:23.329 INFO:teuthology.orchestra.run.smithi116.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-11-28T17:06:23.330 INFO:teuthology.orchestra.run.smithi116.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-11-28T17:06:23.332 INFO:teuthology.orchestra.run.smithi116.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-11-28T17:06:23.335 INFO:teuthology.orchestra.run.smithi116.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-11-28T17:06:23.339 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-11-28T17:06:23.340 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-11-28T17:06:23.418 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-11-28T17:06:23.437 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2023-11-28T17:06:23.437 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-11-28T17:06:23.437 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2023-11-28T17:06:23.437 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-11-28T17:06:23.437 INFO:teuthology.orchestra.run.smithi116.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-11-28T17:06:23.437 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 6.6 k 2023-11-28T17:06:23.437 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 5.2 M 2023-11-28T17:06:23.437 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 859 k 2023-11-28T17:06:23.437 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 141 k 2023-11-28T17:06:23.437 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 1.5 M 2023-11-28T17:06:23.438 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 133 k 2023-11-28T17:06:23.438 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 1.8 M 2023-11-28T17:06:23.438 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 7.4 M 2023-11-28T17:06:23.438 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 50 k 2023-11-28T17:06:23.438 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 8.7 M 2023-11-28T17:06:23.438 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 41 M 2023-11-28T17:06:23.438 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 262 k 2023-11-28T17:06:23.438 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 773 k 2023-11-28T17:06:23.438 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 217 k 2023-11-28T17:06:23.438 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 18 k 2023-11-28T17:06:23.439 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel x86_64 0.190-1.el9 CentOS-AppStream 21 k 2023-11-28T17:06:23.439 INFO:teuthology.orchestra.run.smithi116.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-11-28T17:06:23.439 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-11-28T17:06:23.439 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 31 k 2023-11-28T17:06:23.439 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 719 k 2023-11-28T17:06:23.439 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 126 k 2023-11-28T17:06:23.439 INFO:teuthology.orchestra.run.smithi116.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-11-28T17:06:23.439 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2023-11-28T17:06:23.439 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel x86_64 1:3.0.7-25.el9 appstream 4.1 M 2023-11-28T17:06:23.439 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-11-28T17:06:23.439 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-11-28T17:06:23.439 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-11-28T17:06:23.440 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 162 k 2023-11-28T17:06:23.440 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 321 k 2023-11-28T17:06:23.440 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 298 k 2023-11-28T17:06:23.440 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 99 k 2023-11-28T17:06:23.440 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 85 k 2023-11-28T17:06:23.440 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 3.0 M 2023-11-28T17:06:23.440 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 170 k 2023-11-28T17:06:23.440 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 CentOS-AppStream 85 k 2023-11-28T17:06:23.440 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2023-11-28T17:06:23.440 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager x86_64 1:1.45.6-1.el9 CentOS-BaseOS 2.2 M 2023-11-28T17:06:23.441 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm x86_64 1:1.45.6-1.el9 CentOS-BaseOS 1.8 M 2023-11-28T17:06:23.441 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team x86_64 1:1.45.6-1.el9 CentOS-BaseOS 42 k 2023-11-28T17:06:23.441 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui x86_64 1:1.45.6-1.el9 CentOS-BaseOS 245 k 2023-11-28T17:06:23.441 INFO:teuthology.orchestra.run.smithi116.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-11-28T17:06:23.441 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client x86_64 0.190-1.el9 CentOS-BaseOS 36 k 2023-11-28T17:06:23.441 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf x86_64 0.190-1.el9 CentOS-BaseOS 192 k 2023-11-28T17:06:23.441 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs x86_64 0.190-1.el9 CentOS-BaseOS 255 k 2023-11-28T17:06:23.441 INFO:teuthology.orchestra.run.smithi116.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-11-28T17:06:23.441 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-11-28T17:06:23.441 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 3.3 M 2023-11-28T17:06:23.441 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 3.0 M 2023-11-28T17:06:23.441 INFO:teuthology.orchestra.run.smithi116.stdout: openssl x86_64 1:3.0.7-25.el9 baseos 1.2 M 2023-11-28T17:06:23.442 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs x86_64 1:3.0.7-25.el9 baseos 2.2 M 2023-11-28T17:06:23.442 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-11-28T17:06:23.442 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-11-28T17:06:23.442 INFO:teuthology.orchestra.run.smithi116.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2023-11-28T17:06:23.442 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-28T17:06:23.442 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 19 M 2023-11-28T17:06:23.442 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 28 k 2023-11-28T17:06:23.442 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 2.1 M 2023-11-28T17:06:23.442 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 243 k 2023-11-28T17:06:23.442 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 4.4 M 2023-11-28T17:06:23.443 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 18 M 2023-11-28T17:06:23.443 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7531.g47c1c547.el9 ceph-noarch 15 k 2023-11-28T17:06:23.443 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 24 k 2023-11-28T17:06:23.443 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-28T17:06:23.443 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-28T17:06:23.443 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-28T17:06:23.443 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-28T17:06:23.443 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-28T17:06:23.443 INFO:teuthology.orchestra.run.smithi116.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-28T17:06:23.444 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-28T17:06:23.444 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-28T17:06:23.444 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 163 k 2023-11-28T17:06:23.444 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-28T17:06:23.444 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-28T17:06:23.444 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-28T17:06:23.444 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-28T17:06:23.445 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-28T17:06:23.445 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 476 k 2023-11-28T17:06:23.445 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-28T17:06:23.445 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 4.8 M 2023-11-28T17:06:23.445 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-28T17:06:23.445 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-11-28T17:06:23.445 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-28T17:06:23.445 INFO:teuthology.orchestra.run.smithi116.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-11-28T17:06:23.445 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-11-28T17:06:23.445 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-11-28T17:06:23.445 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-28T17:06:23.445 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-28T17:06:23.446 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-28T17:06:23.446 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-28T17:06:23.446 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-11-28T17:06:23.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-28T17:06:23.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-28T17:06:23.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-28T17:06:23.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-28T17:06:23.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 45 k 2023-11-28T17:06:23.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:18.0.0-7531.g47c1c547.el9 ceph 129 k 2023-11-28T17:06:23.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-28T17:06:23.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-28T17:06:23.447 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-28T17:06:23.447 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-28T17:06:23.447 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-28T17:06:23.447 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-28T17:06:23.447 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-28T17:06:23.447 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-28T17:06:23.447 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-28T17:06:23.447 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-28T17:06:23.447 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-28T17:06:23.447 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-28T17:06:23.447 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-28T17:06:23.448 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-28T17:06:23.448 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-28T17:06:23.448 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-28T17:06:23.448 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-28T17:06:23.448 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-28T17:06:23.448 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-28T17:06:23.448 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-28T17:06:23.448 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-28T17:06:23.448 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-28T17:06:23.448 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-28T17:06:23.449 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-28T17:06:23.449 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-28T17:06:23.449 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-28T17:06:23.449 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-28T17:06:23.449 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-28T17:06:23.449 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-28T17:06:23.449 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-28T17:06:23.449 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-28T17:06:23.449 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-28T17:06:23.449 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-28T17:06:23.449 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-28T17:06:23.449 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-28T17:06:23.450 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-28T17:06:23.450 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-28T17:06:23.450 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-28T17:06:23.450 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-28T17:06:23.450 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-28T17:06:23.450 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-28T17:06:23.450 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-28T17:06:23.450 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-28T17:06:23.450 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-28T17:06:23.450 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-28T17:06:23.451 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-28T17:06:23.451 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-28T17:06:23.451 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-28T17:06:23.451 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-28T17:06:23.451 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-28T17:06:23.451 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-28T17:06:23.451 INFO:teuthology.orchestra.run.smithi116.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-11-28T17:06:23.451 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-28T17:06:23.451 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2023-11-28T17:06:23.451 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2023-11-28T17:06:23.451 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-28T17:06:23.451 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:06:23.452 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-11-28T17:06:23.452 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2023-11-28T17:06:23.452 INFO:teuthology.orchestra.run.smithi116.stdout:Install 135 Packages 2023-11-28T17:06:23.452 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 15 Packages 2023-11-28T17:06:23.452 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:06:23.452 INFO:teuthology.orchestra.run.smithi156.stdout:(21/150): librgw2-18.0.0-7531.g47c1c547.el9.x86 12 MB/s | 4.8 MB 00:00 2023-11-28T17:06:23.453 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 205 M 2023-11-28T17:06:23.453 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-11-28T17:06:23.487 INFO:teuthology.orchestra.run.smithi156.stdout:(22/150): python3-rados-18.0.0-7531.g47c1c547.e 1.9 MB/s | 321 kB 00:00 2023-11-28T17:06:23.520 INFO:teuthology.orchestra.run.smithi156.stdout:(23/150): python3-rbd-18.0.0-7531.g47c1c547.el9 3.9 MB/s | 298 kB 00:00 2023-11-28T17:06:23.553 INFO:teuthology.orchestra.run.smithi156.stdout:(24/150): python3-rgw-18.0.0-7531.g47c1c547.el9 1.5 MB/s | 99 kB 00:00 2023-11-28T17:06:23.579 INFO:teuthology.orchestra.run.smithi156.stdout:(25/150): rbd-fuse-18.0.0-7531.g47c1c547.el9.x8 1.4 MB/s | 85 kB 00:00 2023-11-28T17:06:23.677 INFO:teuthology.orchestra.run.smithi156.stdout:(26/150): rbd-nbd-18.0.0-7531.g47c1c547.el9.x86 1.7 MB/s | 170 kB 00:00 2023-11-28T17:06:23.771 INFO:teuthology.orchestra.run.smithi156.stdout:(27/150): ceph-grafana-dashboards-18.0.0-7531.g 304 kB/s | 28 kB 00:00 2023-11-28T17:06:23.846 INFO:teuthology.orchestra.run.smithi156.stdout:(28/150): rbd-mirror-18.0.0-7531.g47c1c547.el9. 10 MB/s | 3.0 MB 00:00 2023-11-28T17:06:23.871 INFO:teuthology.orchestra.run.smithi156.stdout:(29/150): ceph-mgr-cephadm-18.0.0-7531.g47c1c54 1.3 MB/s | 133 kB 00:00 2023-11-28T17:06:24.188 INFO:teuthology.orchestra.run.smithi156.stdout:(30/150): ceph-mgr-dashboard-18.0.0-7531.g47c1c 5.2 MB/s | 1.8 MB 00:00 2023-11-28T17:06:24.272 INFO:teuthology.orchestra.run.smithi156.stdout:(31/150): ceph-mgr-modules-core-18.0.0-7531.g47 2.9 MB/s | 243 kB 00:00 2023-11-28T17:06:24.339 INFO:teuthology.orchestra.run.smithi156.stdout:(32/150): ceph-mgr-rook-18.0.0-7531.g47c1c547.e 765 kB/s | 50 kB 00:00 2023-11-28T17:06:24.359 INFO:teuthology.orchestra.run.smithi116.stdout:(1/150): ceph-18.0.0-7531.g47c1c547.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2023-11-28T17:06:24.489 INFO:teuthology.orchestra.run.smithi156.stdout:(33/150): ceph-mgr-diskprediction-local-18.0.0- 12 MB/s | 7.4 MB 00:00 2023-11-28T17:06:24.506 INFO:teuthology.orchestra.run.smithi156.stdout:(34/150): ceph-prometheus-alerts-18.0.0-7531.g4 91 kB/s | 15 kB 00:00 2023-11-28T17:06:24.759 INFO:teuthology.orchestra.run.smithi116.stdout:(2/150): ceph-fuse-18.0.0-7531.g47c1c547.el9.x8 2.1 MB/s | 859 kB 00:00 2023-11-28T17:06:24.843 INFO:teuthology.orchestra.run.smithi116.stdout:(3/150): ceph-immutable-object-cache-18.0.0-753 1.7 MB/s | 141 kB 00:00 2023-11-28T17:06:25.098 INFO:teuthology.orchestra.run.smithi156.stdout:(35/150): ceph-volume-18.0.0-7531.g47c1c547.el9 444 kB/s | 262 kB 00:00 2023-11-28T17:06:25.173 INFO:teuthology.orchestra.run.smithi156.stdout:(36/150): cephfs-top-18.0.0-7531.g47c1c547.el9. 246 kB/s | 18 kB 00:00 2023-11-28T17:06:25.373 INFO:teuthology.orchestra.run.smithi156.stdout:(37/150): automake-1.16.2-8.el9.noarch.rpm 3.3 MB/s | 662 kB 00:00 2023-11-28T17:06:25.440 INFO:teuthology.orchestra.run.smithi156.stdout:(38/150): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 920 kB 00:00 2023-11-28T17:06:25.482 INFO:teuthology.orchestra.run.smithi156.stdout:(39/150): cephadm-18.0.0-7531.g47c1c547.el9.noa 794 kB/s | 773 kB 00:00 2023-11-28T17:06:25.516 INFO:teuthology.orchestra.run.smithi156.stdout:(40/150): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 106 kB 00:00 2023-11-28T17:06:25.541 INFO:teuthology.orchestra.run.smithi156.stdout:(41/150): elfutils-libelf-devel-0.190-1.el9.x86 379 kB/s | 21 kB 00:00 2023-11-28T17:06:25.575 INFO:teuthology.orchestra.run.smithi156.stdout:(42/150): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2023-11-28T17:06:25.600 INFO:teuthology.orchestra.run.smithi156.stdout:(43/150): flexiblas-3.0.4-8.el9.x86_64.rpm 574 kB/s | 32 kB 00:00 2023-11-28T17:06:25.693 INFO:teuthology.orchestra.run.smithi116.stdout:(4/150): ceph-mds-18.0.0-7531.g47c1c547.el9.x86 2.5 MB/s | 2.1 MB 00:00 2023-11-28T17:06:25.750 INFO:teuthology.orchestra.run.smithi156.stdout:(44/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2023-11-28T17:06:25.776 INFO:teuthology.orchestra.run.smithi156.stdout:(45/150): flexiblas-openblas-openmp-3.0.4-8.el9 106 kB/s | 17 kB 00:00 2023-11-28T17:06:25.809 INFO:teuthology.orchestra.run.smithi156.stdout:(46/150): jq-1.6-16.el9.x86_64.rpm 3.2 MB/s | 186 kB 00:00 2023-11-28T17:06:25.834 INFO:teuthology.orchestra.run.smithi156.stdout:(47/150): libaio-devel-0.3.111-13.el9.x86_64.rp 176 kB/s | 10 kB 00:00 2023-11-28T17:06:25.868 INFO:teuthology.orchestra.run.smithi156.stdout:(48/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2023-11-28T17:06:25.894 INFO:teuthology.orchestra.run.smithi156.stdout:(49/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 782 kB/s | 44 kB 00:00 2023-11-28T17:06:25.935 INFO:teuthology.orchestra.run.smithi156.stdout:(50/150): librdkafka-1.6.1-102.el9.x86_64.rpm 9.9 MB/s | 661 kB 00:00 2023-11-28T17:06:25.947 INFO:teuthology.orchestra.run.smithi116.stdout:(5/150): ceph-common-18.0.0-7531.g47c1c547.el9. 9.8 MB/s | 19 MB 00:01 2023-11-28T17:06:25.978 INFO:teuthology.orchestra.run.smithi156.stdout:(51/150): libtool-2.4.6-45.el9.x86_64.rpm 7.3 MB/s | 578 kB 00:00 2023-11-28T17:06:26.003 INFO:teuthology.orchestra.run.smithi156.stdout:(52/150): libuuid-devel-2.37.4-15.el9.x86_64.rp 398 kB/s | 24 kB 00:00 2023-11-28T17:06:26.028 INFO:teuthology.orchestra.run.smithi156.stdout:(53/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 945 kB/s | 46 kB 00:00 2023-11-28T17:06:26.062 INFO:teuthology.orchestra.run.smithi156.stdout:(54/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-11-28T17:06:26.069 INFO:teuthology.orchestra.run.smithi116.stdout:(6/150): ceph-base-18.0.0-7531.g47c1c547.el9.x8 2.5 MB/s | 5.2 MB 00:02 2023-11-28T17:06:26.095 INFO:teuthology.orchestra.run.smithi156.stdout:(55/150): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2023-11-28T17:06:26.129 INFO:teuthology.orchestra.run.smithi156.stdout:(56/150): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2023-11-28T17:06:26.154 INFO:teuthology.orchestra.run.smithi156.stdout:(57/150): openblas-0.3.21-2.el9.x86_64.rpm 613 kB/s | 34 kB 00:00 2023-11-28T17:06:26.194 INFO:teuthology.orchestra.run.smithi116.stdout:(7/150): ceph-mgr-18.0.0-7531.g47c1c547.el9.x86 2.9 MB/s | 1.5 MB 00:00 2023-11-28T17:06:26.294 INFO:teuthology.orchestra.run.smithi116.stdout:(8/150): ceph-mon-18.0.0-7531.g47c1c547.el9.x86 13 MB/s | 4.4 MB 00:00 2023-11-28T17:06:26.321 INFO:teuthology.orchestra.run.smithi156.stdout:(58/150): openblas-openmp-0.3.21-2.el9.x86_64.r 25 MB/s | 4.7 MB 00:00 2023-11-28T17:06:26.362 INFO:teuthology.orchestra.run.smithi116.stdout:(9/150): ceph-selinux-18.0.0-7531.g47c1c547.el9 367 kB/s | 24 kB 00:00 2023-11-28T17:06:26.380 INFO:teuthology.orchestra.run.smithi156.stdout:(59/150): postgresql-13.11-1.el9.x86_64.rpm 7.3 MB/s | 1.5 MB 00:00 2023-11-28T17:06:26.422 INFO:teuthology.orchestra.run.smithi156.stdout:(60/150): postgresql-contrib-13.11-1.el9.x86_64 9.2 MB/s | 811 kB 00:00 2023-11-28T17:06:26.447 INFO:teuthology.orchestra.run.smithi156.stdout:(61/150): postgresql-private-libs-13.11-1.el9.x 2.2 MB/s | 134 kB 00:00 2023-11-28T17:06:26.647 INFO:teuthology.orchestra.run.smithi156.stdout:(62/150): postgresql-server-13.11-1.el9.x86_64. 25 MB/s | 5.7 MB 00:00 2023-11-28T17:06:26.789 INFO:teuthology.orchestra.run.smithi156.stdout:(63/150): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2023-11-28T17:06:26.815 INFO:teuthology.orchestra.run.smithi156.stdout:(64/150): python3-jinja2-2.11.3-4.el9.noarch.rp 1.5 MB/s | 228 kB 00:00 2023-11-28T17:06:26.840 INFO:teuthology.orchestra.run.smithi156.stdout:(65/150): python3-jsonpatch-1.21-16.el9.noarch. 518 kB/s | 24 kB 00:00 2023-11-28T17:06:26.873 INFO:teuthology.orchestra.run.smithi156.stdout:(66/150): python3-jsonpointer-2.0-4.el9.noarch. 312 kB/s | 18 kB 00:00 2023-11-28T17:06:26.915 INFO:teuthology.orchestra.run.smithi156.stdout:(67/150): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2023-11-28T17:06:26.932 INFO:teuthology.orchestra.run.smithi156.stdout:(68/150): python3-markupsafe-1.1.1-12.el9.x86_6 545 kB/s | 32 kB 00:00 2023-11-28T17:06:27.183 INFO:teuthology.orchestra.run.smithi156.stdout:(69/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2023-11-28T17:06:27.208 INFO:teuthology.orchestra.run.smithi156.stdout:(70/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 792 kB/s | 217 kB 00:00 2023-11-28T17:06:27.241 INFO:teuthology.orchestra.run.smithi156.stdout:(71/150): python3-oauthlib-3.1.1-5.el9.noarch.r 3.0 MB/s | 174 kB 00:00 2023-11-28T17:06:27.258 INFO:teuthology.orchestra.run.smithi156.stdout:(72/150): python3-prettytable-0.7.2-27.el9.noar 861 kB/s | 41 kB 00:00 2023-11-28T17:06:27.284 INFO:teuthology.orchestra.run.smithi156.stdout:(73/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.2 MB/s | 132 kB 00:00 2023-11-28T17:06:27.317 INFO:teuthology.orchestra.run.smithi156.stdout:(74/150): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 211 kB 00:00 2023-11-28T17:06:27.342 INFO:teuthology.orchestra.run.smithi156.stdout:(75/150): python3-pytz-2021.1-5.el9.noarch.rpm 846 kB/s | 47 kB 00:00 2023-11-28T17:06:27.369 INFO:teuthology.orchestra.run.smithi156.stdout:(76/150): python3-requests-oauthlib-1.3.0-12.el 877 kB/s | 43 kB 00:00 2023-11-28T17:06:27.576 INFO:teuthology.orchestra.run.smithi156.stdout:(77/150): python3-toml-0.10.2-6.el9.noarch.rpm 198 kB/s | 39 kB 00:00 2023-11-28T17:06:27.879 INFO:teuthology.orchestra.run.smithi156.stdout:(78/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2023-11-28T17:06:27.912 INFO:teuthology.orchestra.run.smithi156.stdout:(79/150): socat-1.7.4.1-5.el9.x86_64.rpm 923 kB/s | 300 kB 00:00 2023-11-28T17:06:27.938 INFO:teuthology.orchestra.run.smithi156.stdout:(80/150): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 56 kB 00:00 2023-11-28T17:06:27.963 INFO:teuthology.orchestra.run.smithi156.stdout:(81/150): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 1.7 MB/s | 85 kB 00:00 2023-11-28T17:06:27.988 INFO:teuthology.orchestra.run.smithi156.stdout:(82/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2023-11-28T17:06:28.106 INFO:teuthology.orchestra.run.smithi156.stdout:(83/150): zlib-devel-1.2.11-40.el9.x86_64.rpm 311 kB/s | 44 kB 00:00 2023-11-28T17:06:28.139 INFO:teuthology.orchestra.run.smithi156.stdout:(84/150): fuse-2.9.9-15.el9.x86_64.rpm 526 kB/s | 79 kB 00:00 2023-11-28T17:06:28.181 INFO:teuthology.orchestra.run.smithi156.stdout:(85/150): libgfortran-11.4.1-2.3.el9.x86_64.rpm 11 MB/s | 803 kB 00:00 2023-11-28T17:06:28.215 INFO:teuthology.orchestra.run.smithi156.stdout:(86/150): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.8 MB/s | 195 kB 00:00 2023-11-28T17:06:28.248 INFO:teuthology.orchestra.run.smithi156.stdout:(87/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 241 kB 00:00 2023-11-28T17:06:28.307 INFO:teuthology.orchestra.run.smithi156.stdout:(88/150): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.1 MB 00:00 2023-11-28T17:06:28.324 INFO:teuthology.orchestra.run.smithi156.stdout:(89/150): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2023-11-28T17:06:28.349 INFO:teuthology.orchestra.run.smithi156.stdout:(90/150): python3-pycparser-2.20-6.el9.noarch.r 3.0 MB/s | 124 kB 00:00 2023-11-28T17:06:28.366 INFO:teuthology.orchestra.run.smithi156.stdout:(91/150): python3-requests-2.25.1-7.el9.noarch. 2.8 MB/s | 113 kB 00:00 2023-11-28T17:06:28.803 INFO:teuthology.orchestra.run.smithi116.stdout:(10/150): ceph-radosgw-18.0.0-7531.g47c1c547.el 3.3 MB/s | 8.7 MB 00:02 2023-11-28T17:06:28.851 INFO:teuthology.orchestra.run.smithi156.stdout:(92/150): ceph-test-18.0.0-7531.g47c1c547.el9.x 5.9 MB/s | 41 MB 00:06 2023-11-28T17:06:28.861 INFO:teuthology.orchestra.run.smithi116.stdout:(11/150): cephfs-mirror-18.0.0-7531.g47c1c547.e 3.7 MB/s | 217 kB 00:00 2023-11-28T17:06:28.883 INFO:teuthology.orchestra.run.smithi156.stdout:(93/150): python3-urllib3-1.26.5-4.el9.noarch.r 351 kB/s | 187 kB 00:00 2023-11-28T17:06:28.937 INFO:teuthology.orchestra.run.smithi116.stdout:(12/150): libcephfs-devel-18.0.0-7531.g47c1c547 419 kB/s | 31 kB 00:00 2023-11-28T17:06:28.959 INFO:teuthology.orchestra.run.smithi156.stdout:(94/150): lua-devel-5.4.4-4.el9.x86_64.rpm 299 kB/s | 22 kB 00:00 2023-11-28T17:06:29.301 INFO:teuthology.orchestra.run.smithi156.stdout:(95/150): gperftools-libs-2.9.1-2.el9.x86_64.rp 906 kB/s | 309 kB 00:00 2023-11-28T17:06:29.395 INFO:teuthology.orchestra.run.smithi116.stdout:(13/150): ceph-test-18.0.0-7531.g47c1c547.el9.x 14 MB/s | 41 MB 00:03 2023-11-28T17:06:29.445 INFO:teuthology.orchestra.run.smithi116.stdout:(14/150): libcephfs2-18.0.0-7531.g47c1c547.el9. 1.4 MB/s | 719 kB 00:00 2023-11-28T17:06:29.459 INFO:teuthology.orchestra.run.smithi156.stdout:(96/150): python3-devel-3.9.17-2.el9.x86_64.rpm 404 kB/s | 245 kB 00:00 2023-11-28T17:06:29.479 INFO:teuthology.orchestra.run.smithi116.stdout:(15/150): libcephsqlite-18.0.0-7531.g47c1c547.e 1.9 MB/s | 163 kB 00:00 2023-11-28T17:06:29.513 INFO:teuthology.orchestra.run.smithi116.stdout:(16/150): librados-devel-18.0.0-7531.g47c1c547. 1.9 MB/s | 126 kB 00:00 2023-11-28T17:06:29.601 INFO:teuthology.orchestra.run.smithi156.stdout:(97/150): libarrow-doc-9.0.0-9.el9.noarch.rpm 173 kB/s | 24 kB 00:00 2023-11-28T17:06:29.685 INFO:teuthology.orchestra.run.smithi156.stdout:(98/150): liboath-2.6.7-2.el9.x86_64.rpm 588 kB/s | 49 kB 00:00 2023-11-28T17:06:29.747 INFO:teuthology.orchestra.run.smithi116.stdout:(17/150): ceph-osd-18.0.0-7531.g47c1c547.el9.x8 4.8 MB/s | 18 MB 00:03 2023-11-28T17:06:29.780 INFO:teuthology.orchestra.run.smithi116.stdout:(18/150): libradosstriper1-18.0.0-7531.g47c1c54 1.6 MB/s | 476 kB 00:00 2023-11-28T17:06:29.793 INFO:teuthology.orchestra.run.smithi156.stdout:(99/150): openssl-devel-3.0.7-25.el9.x86_64.rpm 2.9 MB/s | 4.1 MB 00:01 2023-11-28T17:06:29.819 INFO:teuthology.orchestra.run.smithi156.stdout:(100/150): libunwind-1.6.2-1.el9.x86_64.rpm 507 kB/s | 67 kB 00:00 2023-11-28T17:06:29.847 INFO:teuthology.orchestra.run.smithi116.stdout:(19/150): python3-ceph-argparse-18.0.0-7531.g47 455 kB/s | 45 kB 00:00 2023-11-28T17:06:29.881 INFO:teuthology.orchestra.run.smithi116.stdout:(20/150): python3-ceph-common-18.0.0-7531.g47c1 1.3 MB/s | 129 kB 00:00 2023-11-28T17:06:29.939 INFO:teuthology.orchestra.run.smithi116.stdout:(21/150): python3-rados-18.0.0-7531.g47c1c547.e 5.4 MB/s | 321 kB 00:00 2023-11-28T17:06:29.965 INFO:teuthology.orchestra.run.smithi116.stdout:(22/150): python3-cephfs-18.0.0-7531.g47c1c547. 1.4 MB/s | 162 kB 00:00 2023-11-28T17:06:29.998 INFO:teuthology.orchestra.run.smithi116.stdout:(23/150): python3-rbd-18.0.0-7531.g47c1c547.el9 5.0 MB/s | 298 kB 00:00 2023-11-28T17:06:30.010 INFO:teuthology.orchestra.run.smithi156.stdout:(101/150): luarocks-3.9.2-1.el9.noarch.rpm 700 kB/s | 151 kB 00:00 2023-11-28T17:06:30.052 INFO:teuthology.orchestra.run.smithi156.stdout:(102/150): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.5 MB/s | 837 kB 00:00 2023-11-28T17:06:30.074 INFO:teuthology.orchestra.run.smithi116.stdout:(24/150): rbd-fuse-18.0.0-7531.g47c1c547.el9.x8 1.1 MB/s | 85 kB 00:00 2023-11-28T17:06:30.099 INFO:teuthology.orchestra.run.smithi116.stdout:(25/150): python3-rgw-18.0.0-7531.g47c1c547.el9 746 kB/s | 99 kB 00:00 2023-11-28T17:06:30.119 INFO:teuthology.orchestra.run.smithi156.stdout:(103/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 660 kB/s | 43 kB 00:00 2023-11-28T17:06:30.153 INFO:teuthology.orchestra.run.smithi156.stdout:(104/150): python3-asyncssh-2.13.2-3.el9.noarch 3.8 MB/s | 548 kB 00:00 2023-11-28T17:06:30.187 INFO:teuthology.orchestra.run.smithi156.stdout:(105/150): python3-cachetools-4.2.4-1.el9.noarc 483 kB/s | 32 kB 00:00 2023-11-28T17:06:30.208 INFO:teuthology.orchestra.run.smithi116.stdout:(26/150): rbd-nbd-18.0.0-7531.g47c1c547.el9.x86 1.5 MB/s | 170 kB 00:00 2023-11-28T17:06:30.212 INFO:teuthology.orchestra.run.smithi156.stdout:(106/150): python3-certifi-2021.10.8-2.el9.noar 258 kB/s | 15 kB 00:00 2023-11-28T17:06:30.292 INFO:teuthology.orchestra.run.smithi116.stdout:(27/150): rbd-mirror-18.0.0-7531.g47c1c547.el9. 14 MB/s | 3.0 MB 00:00 2023-11-28T17:06:30.312 INFO:teuthology.orchestra.run.smithi156.stdout:(107/150): libarrow-9.0.0-9.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2023-11-28T17:06:30.346 INFO:teuthology.orchestra.run.smithi156.stdout:(108/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2023-11-28T17:06:30.350 INFO:teuthology.orchestra.run.smithi116.stdout:(28/150): ceph-grafana-dashboards-18.0.0-7531.g 199 kB/s | 28 kB 00:00 2023-11-28T17:06:30.380 INFO:teuthology.orchestra.run.smithi156.stdout:(109/150): python3-cherrypy-18.6.1-2.el9.noarch 2.1 MB/s | 358 kB 00:00 2023-11-28T17:06:30.383 INFO:teuthology.orchestra.run.smithi116.stdout:(29/150): ceph-mgr-cephadm-18.0.0-7531.g47c1c54 1.4 MB/s | 133 kB 00:00 2023-11-28T17:06:30.405 INFO:teuthology.orchestra.run.smithi156.stdout:(110/150): python3-google-auth-2.23.4-1.el9.noa 2.1 MB/s | 197 kB 00:00 2023-11-28T17:06:30.430 INFO:teuthology.orchestra.run.smithi156.stdout:(111/150): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2023-11-28T17:06:30.456 INFO:teuthology.orchestra.run.smithi156.stdout:(112/150): python3-jaraco-classes-3.2.1-5.el9.n 236 kB/s | 18 kB 00:00 2023-11-28T17:06:30.481 INFO:teuthology.orchestra.run.smithi156.stdout:(113/150): python3-jaraco-collections-3.0.0-8.e 309 kB/s | 23 kB 00:00 2023-11-28T17:06:30.484 INFO:teuthology.orchestra.run.smithi116.stdout:(30/150): librgw2-18.0.0-7531.g47c1c547.el9.x86 4.9 MB/s | 4.8 MB 00:00 2023-11-28T17:06:30.506 INFO:teuthology.orchestra.run.smithi156.stdout:(114/150): python3-jaraco-functools-3.5.0-2.el9 259 kB/s | 19 kB 00:00 2023-11-28T17:06:30.531 INFO:teuthology.orchestra.run.smithi156.stdout:(115/150): python3-jaraco-text-3.2.0-6.el9.noar 263 kB/s | 20 kB 00:00 2023-11-28T17:06:30.557 INFO:teuthology.orchestra.run.smithi156.stdout:(116/150): python3-jwt+crypto-2.4.0-1.el9.noarc 121 kB/s | 9.0 kB 00:00 2023-11-28T17:06:30.582 INFO:teuthology.orchestra.run.smithi156.stdout:(117/150): python3-jwt-2.4.0-1.el9.noarch.rpm 546 kB/s | 41 kB 00:00 2023-11-28T17:06:30.633 INFO:teuthology.orchestra.run.smithi156.stdout:(118/150): python3-logutils-0.3.5-21.el9.noarch 621 kB/s | 46 kB 00:00 2023-11-28T17:06:30.642 INFO:teuthology.orchestra.run.smithi116.stdout:(31/150): ceph-mgr-modules-core-18.0.0-7531.g47 1.5 MB/s | 243 kB 00:00 2023-11-28T17:06:30.649 INFO:teuthology.orchestra.run.smithi156.stdout:(119/150): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2023-11-28T17:06:30.743 INFO:teuthology.orchestra.run.smithi116.stdout:(32/150): ceph-mgr-rook-18.0.0-7531.g47c1c547.e 507 kB/s | 50 kB 00:00 2023-11-28T17:06:30.841 INFO:teuthology.orchestra.run.smithi156.stdout:(120/150): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2023-11-28T17:06:30.885 INFO:teuthology.orchestra.run.smithi116.stdout:(33/150): ceph-mgr-diskprediction-local-18.0.0- 15 MB/s | 7.4 MB 00:00 2023-11-28T17:06:30.925 INFO:teuthology.orchestra.run.smithi156.stdout:(121/150): python3-portend-3.1.0-2.el9.noarch.r 198 kB/s | 16 kB 00:00 2023-11-28T17:06:30.926 INFO:teuthology.orchestra.run.smithi116.stdout:(34/150): ceph-prometheus-alerts-18.0.0-7531.g4 83 kB/s | 15 kB 00:00 2023-11-28T17:06:31.000 INFO:teuthology.orchestra.run.smithi156.stdout:(122/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2023-11-28T17:06:31.002 INFO:teuthology.orchestra.run.smithi116.stdout:(35/150): ceph-volume-18.0.0-7531.g47c1c547.el9 2.2 MB/s | 262 kB 00:00 2023-11-28T17:06:31.042 INFO:teuthology.orchestra.run.smithi156.stdout:(123/150): python3-kubernetes-26.1.0-2.el9.noar 2.0 MB/s | 1.0 MB 00:00 2023-11-28T17:06:31.060 INFO:teuthology.orchestra.run.smithi116.stdout:(36/150): cephadm-18.0.0-7531.g47c1c547.el9.noa 5.7 MB/s | 773 kB 00:00 2023-11-28T17:06:31.067 INFO:teuthology.orchestra.run.smithi156.stdout:(124/150): python3-natsort-7.1.1-5.el9.noarch.r 133 kB/s | 58 kB 00:00 2023-11-28T17:06:31.093 INFO:teuthology.orchestra.run.smithi156.stdout:(125/150): python3-repoze-lru-0.7-16.el9.noarch 336 kB/s | 31 kB 00:00 2023-11-28T17:06:31.102 INFO:teuthology.orchestra.run.smithi116.stdout:(37/150): cephfs-top-18.0.0-7531.g47c1c547.el9. 183 kB/s | 18 kB 00:00 2023-11-28T17:06:31.126 INFO:teuthology.orchestra.run.smithi156.stdout:(126/150): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2023-11-28T17:06:31.151 INFO:teuthology.orchestra.run.smithi156.stdout:(127/150): python3-tempora-5.0.0-2.el9.noarch.r 618 kB/s | 36 kB 00:00 2023-11-28T17:06:31.215 INFO:teuthology.orchestra.run.smithi116.stdout:(38/150): ceph-mgr-dashboard-18.0.0-7531.g47c1c 2.2 MB/s | 1.8 MB 00:00 2023-11-28T17:06:31.215 INFO:teuthology.orchestra.run.smithi156.stdout:(128/150): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2023-11-28T17:06:31.215 INFO:teuthology.orchestra.run.smithi156.stdout:(129/150): python3-typing-extensions-4.4.0-2.el 684 kB/s | 51 kB 00:00 2023-11-28T17:06:31.236 INFO:teuthology.orchestra.run.smithi156.stdout:(130/150): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2023-11-28T17:06:31.269 INFO:teuthology.orchestra.run.smithi156.stdout:(131/150): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2023-11-28T17:06:31.303 INFO:teuthology.orchestra.run.smithi156.stdout:(132/150): python3-zc-lockfile-2.0-10.el9.noarc 301 kB/s | 20 kB 00:00 2023-11-28T17:06:31.336 INFO:teuthology.orchestra.run.smithi156.stdout:(133/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2023-11-28T17:06:31.387 INFO:teuthology.orchestra.run.smithi156.stdout:(134/150): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2023-11-28T17:06:31.503 INFO:teuthology.orchestra.run.smithi116.stdout:(39/150): boost-program-options-1.75.0-8.el9.x8 443 kB/s | 106 kB 00:00 2023-11-28T17:06:31.562 INFO:teuthology.orchestra.run.smithi156.stdout:(135/150): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2023-11-28T17:06:31.620 INFO:teuthology.orchestra.run.smithi116.stdout:(40/150): bison-3.7.4-5.el9.x86_64.rpm 1.7 MB/s | 920 kB 00:00 2023-11-28T17:06:31.637 INFO:teuthology.orchestra.run.smithi156.stdout:(136/150): libacl-devel-2.3.1-4.el9.x86_64.rpm 913 kB/s | 68 kB 00:00 2023-11-28T17:06:31.712 INFO:teuthology.orchestra.run.smithi116.stdout:(41/150): automake-1.16.2-8.el9.noarch.rpm 1.0 MB/s | 662 kB 00:00 2023-11-28T17:06:31.738 INFO:teuthology.orchestra.run.smithi156.stdout:(137/150): librbd1-18.0.0-7531.g47c1c547.el9.x8 8.6 MB/s | 3.0 MB 00:00 2023-11-28T17:06:31.754 INFO:teuthology.orchestra.run.smithi116.stdout:(42/150): elfutils-libelf-devel-0.190-1.el9.x86 87 kB/s | 21 kB 00:00 2023-11-28T17:06:31.829 INFO:teuthology.orchestra.run.smithi116.stdout:(43/150): flex-2.6.4-9.el9.x86_64.rpm 1.4 MB/s | 307 kB 00:00 2023-11-28T17:06:31.871 INFO:teuthology.orchestra.run.smithi116.stdout:(44/150): flexiblas-3.0.4-8.el9.x86_64.rpm 201 kB/s | 32 kB 00:00 2023-11-28T17:06:31.896 INFO:teuthology.orchestra.run.smithi156.stdout:(138/150): NetworkManager-1.45.6-1.el9.x86_64.r 8.7 MB/s | 2.2 MB 00:00 2023-11-28T17:06:32.097 INFO:teuthology.orchestra.run.smithi156.stdout:(139/150): NetworkManager-libnm-1.45.6-1.el9.x8 5.1 MB/s | 1.8 MB 00:00 2023-11-28T17:06:32.180 INFO:teuthology.orchestra.run.smithi156.stdout:(140/150): NetworkManager-team-1.45.6-1.el9.x86 152 kB/s | 42 kB 00:00 2023-11-28T17:06:32.205 INFO:teuthology.orchestra.run.smithi156.stdout:(141/150): NetworkManager-tui-1.45.6-1.el9.x86_ 2.2 MB/s | 245 kB 00:00 2023-11-28T17:06:32.229 INFO:teuthology.orchestra.run.smithi116.stdout:(45/150): flexiblas-openblas-openmp-3.0.4-8.el9 43 kB/s | 17 kB 00:00 2023-11-28T17:06:32.289 INFO:teuthology.orchestra.run.smithi156.stdout:(142/150): librados2-18.0.0-7531.g47c1c547.el9. 3.5 MB/s | 3.3 MB 00:00 2023-11-28T17:06:32.314 INFO:teuthology.orchestra.run.smithi156.stdout:(143/150): acl-2.3.1-4.el9.x86_64.rpm 640 kB/s | 69 kB 00:00 2023-11-28T17:06:32.339 INFO:teuthology.orchestra.run.smithi156.stdout:(144/150): elfutils-debuginfod-client-0.190-1.e 275 kB/s | 36 kB 00:00 2023-11-28T17:06:32.373 INFO:teuthology.orchestra.run.smithi156.stdout:(145/150): elfutils-libelf-0.190-1.el9.x86_64.r 2.3 MB/s | 192 kB 00:00 2023-11-28T17:06:32.380 INFO:teuthology.orchestra.run.smithi116.stdout:(46/150): jq-1.6-16.el9.x86_64.rpm 367 kB/s | 186 kB 00:00 2023-11-28T17:06:32.398 INFO:teuthology.orchestra.run.smithi156.stdout:(146/150): libacl-2.3.1-4.el9.x86_64.rpm 395 kB/s | 22 kB 00:00 2023-11-28T17:06:32.432 INFO:teuthology.orchestra.run.smithi156.stdout:(147/150): elfutils-libs-0.190-1.el9.x86_64.rpm 2.8 MB/s | 255 kB 00:00 2023-11-28T17:06:32.474 INFO:teuthology.orchestra.run.smithi156.stdout:(148/150): xfsdump-3.1.12-4.el9.x86_64.rpm 3.2 MB/s | 321 kB 00:00 2023-11-28T17:06:32.572 INFO:teuthology.orchestra.run.smithi116.stdout:(47/150): libaio-devel-0.3.111-13.el9.x86_64.rp 30 kB/s | 10 kB 00:00 2023-11-28T17:06:32.674 INFO:teuthology.orchestra.run.smithi156.stdout:(149/150): openssl-3.0.7-25.el9.x86_64.rpm 4.4 MB/s | 1.2 MB 00:00 2023-11-28T17:06:32.747 INFO:teuthology.orchestra.run.smithi116.stdout:(48/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.0 MB/s | 3.0 MB 00:00 2023-11-28T17:06:32.749 INFO:teuthology.orchestra.run.smithi156.stdout:(150/150): openssl-libs-3.0.7-25.el9.x86_64.rpm 6.9 MB/s | 2.2 MB 00:00 2023-11-28T17:06:32.758 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-11-28T17:06:32.759 INFO:teuthology.orchestra.run.smithi156.stdout:Total 15 MB/s | 205 MB 00:13 2023-11-28T17:06:32.772 INFO:teuthology.orchestra.run.smithi116.stdout:(49/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 408 kB/s | 159 kB 00:00 2023-11-28T17:06:32.846 INFO:teuthology.orchestra.run.smithi116.stdout:(50/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 163 kB/s | 44 kB 00:00 2023-11-28T17:06:32.939 INFO:teuthology.orchestra.run.smithi116.stdout:(51/150): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 661 kB 00:00 2023-11-28T17:06:32.990 INFO:teuthology.orchestra.run.smithi116.stdout:(52/150): libuuid-devel-2.37.4-15.el9.x86_64.rp 175 kB/s | 24 kB 00:00 2023-11-28T17:06:33.023 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-11-28T17:06:33.031 INFO:teuthology.orchestra.run.smithi116.stdout:(53/150): libtool-2.4.6-45.el9.x86_64.rpm 3.1 MB/s | 578 kB 00:00 2023-11-28T17:06:33.082 INFO:teuthology.orchestra.run.smithi116.stdout:(54/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 327 kB/s | 46 kB 00:00 2023-11-28T17:06:33.191 INFO:teuthology.orchestra.run.smithi116.stdout:(55/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 282 kB 00:00 2023-11-28T17:06:33.224 INFO:teuthology.orchestra.run.smithi116.stdout:(56/150): lua-5.4.4-4.el9.x86_64.rpm 981 kB/s | 187 kB 00:00 2023-11-28T17:06:33.249 INFO:teuthology.orchestra.run.smithi116.stdout:(57/150): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2023-11-28T17:06:33.302 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-11-28T17:06:33.303 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-11-28T17:06:33.317 INFO:teuthology.orchestra.run.smithi116.stdout:(58/150): openblas-0.3.21-2.el9.x86_64.rpm 277 kB/s | 34 kB 00:00 2023-11-28T17:06:33.900 INFO:teuthology.orchestra.run.smithi116.stdout:(59/150): postgresql-13.11-1.el9.x86_64.rpm 2.3 MB/s | 1.5 MB 00:00 2023-11-28T17:06:33.942 INFO:teuthology.orchestra.run.smithi116.stdout:(60/150): postgresql-contrib-13.11-1.el9.x86_64 1.3 MB/s | 811 kB 00:00 2023-11-28T17:06:34.176 INFO:teuthology.orchestra.run.smithi116.stdout:(61/150): postgresql-private-libs-13.11-1.el9.x 488 kB/s | 134 kB 00:00 2023-11-28T17:06:34.317 INFO:teuthology.orchestra.run.smithi116.stdout:(62/150): openblas-openmp-0.3.21-2.el9.x86_64.r 4.4 MB/s | 4.7 MB 00:01 2023-11-28T17:06:34.659 INFO:teuthology.orchestra.run.smithi116.stdout:(63/150): python3-jinja2-2.11.3-4.el9.noarch.rp 693 kB/s | 228 kB 00:00 2023-11-28T17:06:35.051 INFO:teuthology.orchestra.run.smithi116.stdout:(64/150): postgresql-server-13.11-1.el9.x86_64. 5.1 MB/s | 5.7 MB 00:01 2023-11-28T17:06:35.077 INFO:teuthology.orchestra.run.smithi116.stdout:(65/150): python3-jsonpatch-1.21-16.el9.noarch. 59 kB/s | 24 kB 00:00 2023-11-28T17:06:35.327 INFO:teuthology.orchestra.run.smithi116.stdout:(66/150): python3-babel-2.9.1-2.el9.noarch.rpm 5.0 MB/s | 5.8 MB 00:01 2023-11-28T17:06:35.352 INFO:teuthology.orchestra.run.smithi116.stdout:(67/150): python3-jsonpointer-2.0-4.el9.noarch. 69 kB/s | 18 kB 00:00 2023-11-28T17:06:35.386 INFO:teuthology.orchestra.run.smithi116.stdout:(68/150): python3-mako-1.1.4-6.el9.noarch.rpm 491 kB/s | 144 kB 00:00 2023-11-28T17:06:35.411 INFO:teuthology.orchestra.run.smithi116.stdout:(69/150): python3-markupsafe-1.1.1-12.el9.x86_6 381 kB/s | 32 kB 00:00 2023-11-28T17:06:35.505 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-11-28T17:06:35.506 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-11-28T17:06:35.511 INFO:teuthology.orchestra.run.smithi116.stdout:(70/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2023-11-28T17:06:35.836 INFO:teuthology.orchestra.run.smithi116.stdout:(71/150): python3-oauthlib-3.1.1-5.el9.noarch.r 411 kB/s | 174 kB 00:00 2023-11-28T17:06:35.862 INFO:teuthology.orchestra.run.smithi116.stdout:(72/150): python3-prettytable-0.7.2-27.el9.noar 119 kB/s | 41 kB 00:00 2023-11-28T17:06:35.970 INFO:teuthology.orchestra.run.smithi116.stdout:(73/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.2 MB/s | 4.8 MB 00:00 2023-11-28T17:06:36.004 INFO:teuthology.orchestra.run.smithi116.stdout:(74/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 796 kB/s | 132 kB 00:00 2023-11-28T17:06:36.029 INFO:teuthology.orchestra.run.smithi116.stdout:(75/150): python3-pyasn1-modules-0.4.8-6.el9.no 1.2 MB/s | 211 kB 00:00 2023-11-28T17:06:36.096 INFO:teuthology.orchestra.run.smithi116.stdout:(76/150): python3-pytz-2021.1-5.el9.noarch.rpm 387 kB/s | 47 kB 00:00 2023-11-28T17:06:36.146 INFO:teuthology.orchestra.run.smithi116.stdout:(77/150): python3-requests-oauthlib-1.3.0-12.el 371 kB/s | 43 kB 00:00 2023-11-28T17:06:36.446 INFO:teuthology.orchestra.run.smithi116.stdout:(78/150): python3-toml-0.10.2-6.el9.noarch.rpm 110 kB/s | 39 kB 00:00 2023-11-28T17:06:36.480 INFO:teuthology.orchestra.run.smithi116.stdout:(79/150): socat-1.7.4.1-5.el9.x86_64.rpm 910 kB/s | 300 kB 00:00 2023-11-28T17:06:36.747 INFO:teuthology.orchestra.run.smithi116.stdout:(80/150): uuid-1.6.2-55.el9.x86_64.rpm 190 kB/s | 56 kB 00:00 2023-11-28T17:06:36.772 INFO:teuthology.orchestra.run.smithi116.stdout:(81/150): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 297 kB/s | 85 kB 00:00 2023-11-28T17:06:37.039 INFO:teuthology.orchestra.run.smithi116.stdout:(82/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 215 kB/s | 63 kB 00:00 2023-11-28T17:06:37.081 INFO:teuthology.orchestra.run.smithi116.stdout:(83/150): zlib-devel-1.2.11-40.el9.x86_64.rpm 146 kB/s | 44 kB 00:00 2023-11-28T17:06:37.307 INFO:teuthology.orchestra.run.smithi116.stdout:(84/150): fuse-2.9.9-15.el9.x86_64.rpm 295 kB/s | 79 kB 00:00 2023-11-28T17:06:37.540 INFO:teuthology.orchestra.run.smithi116.stdout:(85/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2023-11-28T17:06:37.582 INFO:teuthology.orchestra.run.smithi116.stdout:(86/150): libgfortran-11.4.1-2.3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2023-11-28T17:06:37.607 INFO:teuthology.orchestra.run.smithi116.stdout:(87/150): libquadmath-11.4.1-2.3.el9.x86_64.rpm 656 kB/s | 195 kB 00:00 2023-11-28T17:06:37.641 INFO:teuthology.orchestra.run.smithi116.stdout:(88/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 241 kB 00:00 2023-11-28T17:06:37.691 INFO:teuthology.orchestra.run.smithi116.stdout:(89/150): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2023-11-28T17:06:37.808 INFO:teuthology.orchestra.run.smithi116.stdout:(90/150): python3-cryptography-36.0.1-4.el9.x86 5.8 MB/s | 1.1 MB 00:00 2023-11-28T17:06:37.833 INFO:teuthology.orchestra.run.smithi116.stdout:(91/150): python3-pycparser-2.20-6.el9.noarch.r 647 kB/s | 124 kB 00:00 2023-11-28T17:06:37.859 INFO:teuthology.orchestra.run.smithi116.stdout:(92/150): python3-requests-2.25.1-7.el9.noarch. 692 kB/s | 113 kB 00:00 2023-11-28T17:06:37.892 INFO:teuthology.orchestra.run.smithi116.stdout:(93/150): python3-urllib3-1.26.5-4.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2023-11-28T17:06:38.143 INFO:teuthology.orchestra.run.smithi116.stdout:(94/150): lua-devel-5.4.4-4.el9.x86_64.rpm 89 kB/s | 22 kB 00:00 2023-11-28T17:06:38.318 INFO:teuthology.orchestra.run.smithi116.stdout:(95/150): python3-devel-3.9.17-2.el9.x86_64.rpm 538 kB/s | 245 kB 00:00 2023-11-28T17:06:39.026 INFO:teuthology.orchestra.run.smithi116.stdout:(96/150): gperftools-libs-2.9.1-2.el9.x86_64.rp 350 kB/s | 309 kB 00:00 2023-11-28T17:06:39.202 INFO:teuthology.orchestra.run.smithi116.stdout:(97/150): libarrow-doc-9.0.0-9.el9.noarch.rpm 140 kB/s | 24 kB 00:00 2023-11-28T17:06:39.402 INFO:teuthology.orchestra.run.smithi116.stdout:(98/150): liboath-2.6.7-2.el9.x86_64.rpm 244 kB/s | 49 kB 00:00 2023-11-28T17:06:39.552 INFO:teuthology.orchestra.run.smithi116.stdout:(99/150): libunwind-1.6.2-1.el9.x86_64.rpm 451 kB/s | 67 kB 00:00 2023-11-28T17:06:39.653 INFO:teuthology.orchestra.run.smithi116.stdout:(100/150): libarrow-9.0.0-9.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2023-11-28T17:06:39.719 INFO:teuthology.orchestra.run.smithi116.stdout:(101/150): luarocks-3.9.2-1.el9.noarch.rpm 911 kB/s | 151 kB 00:00 2023-11-28T17:06:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:(102/150): parquet-libs-9.0.0-9.el9.x86_64.rpm 4.7 MB/s | 837 kB 00:00 2023-11-28T17:06:39.936 INFO:teuthology.orchestra.run.smithi116.stdout:(103/150): openssl-devel-3.0.7-25.el9.x86_64.rp 2.0 MB/s | 4.1 MB 00:02 2023-11-28T17:06:39.962 INFO:teuthology.orchestra.run.smithi116.stdout:(104/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 327 kB/s | 43 kB 00:00 2023-11-28T17:06:40.029 INFO:teuthology.orchestra.run.smithi116.stdout:(105/150): python3-asyncssh-2.13.2-3.el9.noarch 1.7 MB/s | 548 kB 00:00 2023-11-28T17:06:40.087 INFO:teuthology.orchestra.run.smithi116.stdout:(106/150): python3-certifi-2021.10.8-2.el9.noar 120 kB/s | 15 kB 00:00 2023-11-28T17:06:40.188 INFO:teuthology.orchestra.run.smithi116.stdout:(107/150): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2023-11-28T17:06:40.238 INFO:teuthology.orchestra.run.smithi116.stdout:(108/150): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2023-11-28T17:06:40.297 INFO:teuthology.orchestra.run.smithi116.stdout:(109/150): python3-cachetools-4.2.4-1.el9.noarc 90 kB/s | 32 kB 00:00 2023-11-28T17:06:40.330 INFO:teuthology.orchestra.run.smithi116.stdout:(110/150): python3-google-auth-2.23.4-1.el9.noa 1.4 MB/s | 197 kB 00:00 2023-11-28T17:06:40.355 INFO:teuthology.orchestra.run.smithi116.stdout:(111/150): python3-jaraco-8.2.1-3.el9.noarch.rp 92 kB/s | 11 kB 00:00 2023-11-28T17:06:40.439 INFO:teuthology.orchestra.run.smithi116.stdout:(112/150): python3-jaraco-classes-3.2.1-5.el9.n 125 kB/s | 18 kB 00:00 2023-11-28T17:06:40.473 INFO:teuthology.orchestra.run.smithi116.stdout:(113/150): python3-jaraco-collections-3.0.0-8.e 163 kB/s | 23 kB 00:00 2023-11-28T17:06:40.498 INFO:teuthology.orchestra.run.smithi116.stdout:(114/150): python3-jaraco-functools-3.5.0-2.el9 137 kB/s | 19 kB 00:00 2023-11-28T17:06:40.582 INFO:teuthology.orchestra.run.smithi116.stdout:(115/150): python3-jaraco-text-3.2.0-6.el9.noar 139 kB/s | 20 kB 00:00 2023-11-28T17:06:40.607 INFO:teuthology.orchestra.run.smithi116.stdout:(116/150): python3-jwt+crypto-2.4.0-1.el9.noarc 84 kB/s | 9.0 kB 00:00 2023-11-28T17:06:40.632 INFO:teuthology.orchestra.run.smithi116.stdout:(117/150): python3-jwt-2.4.0-1.el9.noarch.rpm 308 kB/s | 41 kB 00:00 2023-11-28T17:06:40.757 INFO:teuthology.orchestra.run.smithi116.stdout:(118/150): python3-more-itertools-8.12.0-2.el9. 633 kB/s | 79 kB 00:00 2023-11-28T17:06:40.816 INFO:teuthology.orchestra.run.smithi116.stdout:(119/150): python3-logutils-0.3.5-21.el9.noarch 222 kB/s | 46 kB 00:00 2023-11-28T17:06:40.858 INFO:teuthology.orchestra.run.smithi116.stdout:(120/150): python3-kubernetes-26.1.0-2.el9.noar 3.7 MB/s | 1.0 MB 00:00 2023-11-28T17:06:40.883 INFO:teuthology.orchestra.run.smithi116.stdout:(121/150): python3-natsort-7.1.1-5.el9.noarch.r 463 kB/s | 58 kB 00:00 2023-11-28T17:06:40.975 INFO:teuthology.orchestra.run.smithi116.stdout:(122/150): python3-portend-3.1.0-2.el9.noarch.r 141 kB/s | 16 kB 00:00 2023-11-28T17:06:41.000 INFO:teuthology.orchestra.run.smithi116.stdout:(123/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 773 kB/s | 90 kB 00:00 2023-11-28T17:06:41.084 INFO:teuthology.orchestra.run.smithi116.stdout:(124/150): python3-repoze-lru-0.7-16.el9.noarch 285 kB/s | 31 kB 00:00 2023-11-28T17:06:41.118 INFO:teuthology.orchestra.run.smithi116.stdout:(125/150): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2023-11-28T17:06:41.151 INFO:teuthology.orchestra.run.smithi116.stdout:(126/150): python3-pecan-1.4.2-3.el9.noarch.rpm 815 kB/s | 272 kB 00:00 2023-11-28T17:06:41.210 INFO:teuthology.orchestra.run.smithi116.stdout:(127/150): python3-rsa-4.9-2.el9.noarch.rpm 474 kB/s | 59 kB 00:00 2023-11-28T17:06:41.235 INFO:teuthology.orchestra.run.smithi116.stdout:(128/150): python3-tempora-5.0.0-2.el9.noarch.r 306 kB/s | 36 kB 00:00 2023-11-28T17:06:41.269 INFO:teuthology.orchestra.run.smithi116.stdout:(129/150): python3-typing-extensions-4.4.0-2.el 437 kB/s | 51 kB 00:00 2023-11-28T17:06:41.352 INFO:teuthology.orchestra.run.smithi116.stdout:(130/150): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2023-11-28T17:06:41.386 INFO:teuthology.orchestra.run.smithi116.stdout:(131/150): python3-websocket-client-1.2.3-2.el9 598 kB/s | 90 kB 00:00 2023-11-28T17:06:41.478 INFO:teuthology.orchestra.run.smithi116.stdout:(132/150): python3-zc-lockfile-2.0-10.el9.noarc 160 kB/s | 20 kB 00:00 2023-11-28T17:06:41.520 INFO:teuthology.orchestra.run.smithi116.stdout:(133/150): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2023-11-28T17:06:41.553 INFO:teuthology.orchestra.run.smithi116.stdout:(134/150): python3-werkzeug-2.0.3-3.el9.1.noarc 1.5 MB/s | 427 kB 00:00 2023-11-28T17:06:41.720 INFO:teuthology.orchestra.run.smithi116.stdout:(135/150): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2023-11-28T17:06:41.779 INFO:teuthology.orchestra.run.smithi116.stdout:(136/150): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 68 kB 00:00 2023-11-28T17:06:42.021 INFO:teuthology.orchestra.run.smithi116.stdout:(137/150): librados2-18.0.0-7531.g47c1c547.el9. 6.6 MB/s | 3.3 MB 00:00 2023-11-28T17:06:42.021 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-11-28T17:06:42.110 INFO:teuthology.orchestra.run.smithi116.stdout:(138/150): NetworkManager-1.45.6-1.el9.x86_64.r 6.8 MB/s | 2.2 MB 00:00 2023-11-28T17:06:42.152 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/165 2023-11-28T17:06:42.198 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/165 2023-11-28T17:06:42.271 INFO:teuthology.orchestra.run.smithi116.stdout:(139/150): NetworkManager-libnm-1.45.6-1.el9.x8 7.1 MB/s | 1.8 MB 00:00 2023-11-28T17:06:42.293 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/165 2023-11-28T17:06:42.297 INFO:teuthology.orchestra.run.smithi116.stdout:(140/150): NetworkManager-team-1.45.6-1.el9.x86 235 kB/s | 42 kB 00:00 2023-11-28T17:06:42.355 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/165 2023-11-28T17:06:42.380 INFO:teuthology.orchestra.run.smithi116.stdout:(141/150): librbd1-18.0.0-7531.g47c1c547.el9.x8 3.7 MB/s | 3.0 MB 00:00 2023-11-28T17:06:42.391 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2023-11-28T17:06:42.414 INFO:teuthology.orchestra.run.smithi116.stdout:(142/150): NetworkManager-tui-1.45.6-1.el9.x86_ 1.7 MB/s | 245 kB 00:00 2023-11-28T17:06:42.439 INFO:teuthology.orchestra.run.smithi116.stdout:(143/150): acl-2.3.1-4.el9.x86_64.rpm 488 kB/s | 69 kB 00:00 2023-11-28T17:06:42.465 INFO:teuthology.orchestra.run.smithi116.stdout:(144/150): elfutils-debuginfod-client-0.190-1.e 443 kB/s | 36 kB 00:00 2023-11-28T17:06:42.490 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2023-11-28T17:06:42.506 INFO:teuthology.orchestra.run.smithi116.stdout:(145/150): elfutils-libelf-0.190-1.el9.x86_64.r 2.9 MB/s | 192 kB 00:00 2023-11-28T17:06:42.540 INFO:teuthology.orchestra.run.smithi116.stdout:(146/150): elfutils-libs-0.190-1.el9.x86_64.rpm 2.5 MB/s | 255 kB 00:00 2023-11-28T17:06:42.590 INFO:teuthology.orchestra.run.smithi116.stdout:(147/150): libacl-2.3.1-4.el9.x86_64.rpm 176 kB/s | 22 kB 00:00 2023-11-28T17:06:42.624 INFO:teuthology.orchestra.run.smithi116.stdout:(148/150): xfsdump-3.1.12-4.el9.x86_64.rpm 2.7 MB/s | 321 kB 00:00 2023-11-28T17:06:42.923 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2023-11-28T17:06:42.964 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 8/165 2023-11-28T17:06:43.120 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 8/165 2023-11-28T17:06:43.148 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 9/165 2023-11-28T17:06:43.267 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 9/165 2023-11-28T17:06:43.300 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:18.0.0-7531.g47c1c547.el9.x86_64 10/165 2023-11-28T17:06:43.364 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/165 2023-11-28T17:06:43.641 INFO:teuthology.orchestra.run.smithi116.stdout:(149/150): openssl-3.0.7-25.el9.x86_64.rpm 1.2 MB/s | 1.2 MB 00:01 2023-11-28T17:06:43.643 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-libelf-0.190-1.el9.x86_64 12/165 2023-11-28T17:06:43.712 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-13.11-1.el9.x86_64 13/165 2023-11-28T17:06:43.737 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 14/165 2023-11-28T17:06:43.793 INFO:teuthology.orchestra.run.smithi116.stdout:(150/150): openssl-libs-3.0.7-25.el9.x86_64.rpm 1.8 MB/s | 2.2 MB 00:01 2023-11-28T17:06:43.805 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-11-28T17:06:43.805 INFO:teuthology.orchestra.run.smithi116.stdout:Total 10 MB/s | 205 MB 00:20 2023-11-28T17:06:43.895 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 14/165 2023-11-28T17:06:43.921 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86 15/165 2023-11-28T17:06:44.004 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86 15/165 2023-11-28T17:06:44.073 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-11-28T17:06:44.315 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 16/165 2023-11-28T17:06:44.354 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-11-28T17:06:44.354 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-11-28T17:06:44.501 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-libnm-1:1.45.6-1.el9.x86_64 17/165 2023-11-28T17:06:44.581 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.6-1.el9.x86_64 17/165 2023-11-28T17:06:44.803 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.45.6-1.el9.x86_64 18/165 2023-11-28T17:06:44.855 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-1:1.45.6-1.el9.x86_64 18/165 2023-11-28T17:06:45.319 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.45.6-1.el9.x86_64 18/165 2023-11-28T17:06:45.360 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/165 2023-11-28T17:06:45.477 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/165 2023-11-28T17:06:45.546 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 21/165 2023-11-28T17:06:45.582 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 22/165 2023-11-28T17:06:45.639 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 23/165 2023-11-28T17:06:45.693 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 24/165 2023-11-28T17:06:45.813 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 25/165 2023-11-28T17:06:45.858 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 26/165 2023-11-28T17:06:45.904 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/165 2023-11-28T17:06:45.935 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2023-11-28T17:06:46.045 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2023-11-28T17:06:46.082 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:18.0.0-7531.g47c1c547.el9. 30/165 2023-11-28T17:06:46.132 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:18.0.0-7531.g47c1c547.el 31/165 2023-11-28T17:06:46.163 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:18.0.0-7531.g47c1c547.el9.x86_6 32/165 2023-11-28T17:06:46.274 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/165 2023-11-28T17:06:46.389 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/165 2023-11-28T17:06:46.546 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-11-28T17:06:46.547 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-11-28T17:06:46.556 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 35/165 2023-11-28T17:06:46.636 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 36/165 2023-11-28T17:06:47.323 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.9-2.el9.noarch 37/165 2023-11-28T17:06:47.415 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 38/165 2023-11-28T17:06:47.459 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 39/165 2023-11-28T17:06:47.507 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-libs-0.190-1.el9.x86_64 40/165 2023-11-28T17:06:47.546 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:18.0.0-7531.g47c1c547.el9.x86_6 41/165 2023-11-28T17:06:47.608 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 42/165 2023-11-28T17:06:47.796 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 43/165 2023-11-28T17:06:47.843 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 44/165 2023-11-28T17:06:47.979 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : re2-1:20211101-3.el9.x86_64 45/165 2023-11-28T17:06:48.348 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 46/165 2023-11-28T17:06:48.434 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/165 2023-11-28T17:06:48.539 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/165 2023-11-28T17:06:48.578 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/165 2023-11-28T17:06:48.618 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/165 2023-11-28T17:06:48.687 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/165 2023-11-28T17:06:48.749 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/165 2023-11-28T17:06:48.810 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 53/165 2023-11-28T17:06:48.998 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 54/165 2023-11-28T17:06:49.043 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 55/165 2023-11-28T17:06:49.088 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 56/165 2023-11-28T17:06:49.214 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 57/165 2023-11-28T17:06:49.260 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 58/165 2023-11-28T17:06:49.309 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 59/165 2023-11-28T17:06:49.366 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 60/165 2023-11-28T17:06:49.523 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 61/165 2023-11-28T17:06:49.572 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 62/165 2023-11-28T17:06:49.640 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ply-3.11-14.el9.noarch 63/165 2023-11-28T17:06:49.722 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 64/165 2023-11-28T17:06:49.978 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 65/165 2023-11-28T17:06:50.052 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 66/165 2023-11-28T17:06:50.232 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 67/165 2023-11-28T17:06:50.286 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 68/165 2023-11-28T17:06:50.312 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 69/165 2023-11-28T17:06:50.371 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 70/165 2023-11-28T17:06:50.419 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 71/165 2023-11-28T17:06:50.457 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.9-15.el9.x86_64 72/165 2023-11-28T17:06:50.500 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 73/165 2023-11-28T17:06:50.578 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 74/165 2023-11-28T17:06:50.642 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : uuid-1.6.2-55.el9.x86_64 75/165 2023-11-28T17:06:50.691 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/165 2023-11-28T17:06:50.829 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/165 2023-11-28T17:06:50.868 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 78/165 2023-11-28T17:06:50.902 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 79/165 2023-11-28T17:06:50.955 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 80/165 2023-11-28T17:06:50.997 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 81/165 2023-11-28T17:06:51.138 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 82/165 2023-11-28T17:06:51.180 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 83/165 2023-11-28T17:06:51.319 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/165 2023-11-28T17:06:51.415 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/165 2023-11-28T17:06:52.388 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/165 2023-11-28T17:06:52.443 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/165 2023-11-28T17:06:52.477 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/165 2023-11-28T17:06:52.510 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/165 2023-11-28T17:06:52.754 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.21-2.el9.x86_64 90/165 2023-11-28T17:06:52.781 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 91/165 2023-11-28T17:06:52.881 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 92/165 2023-11-28T17:06:53.262 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-11-28T17:06:53.490 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/165 2023-11-28T17:06:53.524 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/165 2023-11-28T17:06:53.602 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/165 2023-11-28T17:06:53.672 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 93/165 2023-11-28T17:06:53.675 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/165 2023-11-28T17:06:53.708 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2023-11-28T17:06:53.801 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 94/165 2023-11-28T17:06:53.810 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2023-11-28T17:06:54.219 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2023-11-28T17:06:54.247 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 8/165 2023-11-28T17:06:54.402 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 8/165 2023-11-28T17:06:54.423 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 9/165 2023-11-28T17:06:54.540 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 9/165 2023-11-28T17:06:54.577 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:18.0.0-7531.g47c1c547.el9.x86_64 10/165 2023-11-28T17:06:54.637 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/165 2023-11-28T17:06:54.919 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libelf-0.190-1.el9.x86_64 12/165 2023-11-28T17:06:54.980 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-13.11-1.el9.x86_64 13/165 2023-11-28T17:06:55.011 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 14/165 2023-11-28T17:06:55.150 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 14/165 2023-11-28T17:06:55.179 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86 15/165 2023-11-28T17:06:55.261 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86 15/165 2023-11-28T17:06:55.336 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 95/165 2023-11-28T17:06:55.420 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 96/165 2023-11-28T17:06:55.460 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 97/165 2023-11-28T17:06:55.513 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : jq-1.6-16.el9.x86_64 98/165 2023-11-28T17:06:55.557 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2023-11-28T17:06:55.589 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 16/165 2023-11-28T17:06:55.599 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 100/165 2023-11-28T17:06:55.776 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-libnm-1:1.45.6-1.el9.x86_64 17/165 2023-11-28T17:06:55.864 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.6-1.el9.x86_64 17/165 2023-11-28T17:06:55.964 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2023-11-28T17:06:55.994 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 102/165 2023-11-28T17:06:56.072 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.45.6-1.el9.x86_64 18/165 2023-11-28T17:06:56.098 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 102/165 2023-11-28T17:06:56.134 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:18.0.0-7531.g47c1c547.el9.x86_64 103/165 2023-11-28T17:06:56.137 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-1:1.45.6-1.el9.x86_64 18/165 2023-11-28T17:06:56.197 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2023-11-28T17:06:56.586 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.45.6-1.el9.x86_64 18/165 2023-11-28T17:06:56.625 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/165 2023-11-28T17:06:56.673 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 105/165 2023-11-28T17:06:56.706 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 106/165 2023-11-28T17:06:56.740 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/165 2023-11-28T17:06:56.798 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 106/165 2023-11-28T17:06:56.805 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 21/165 2023-11-28T17:06:56.824 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:18.0.0-7531.g47c1c547.el9.x86_64 107/165 2023-11-28T17:06:56.853 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 22/165 2023-11-28T17:06:56.910 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 23/165 2023-11-28T17:06:56.966 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 24/165 2023-11-28T17:06:57.070 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 25/165 2023-11-28T17:06:57.131 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 26/165 2023-11-28T17:06:57.187 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/165 2023-11-28T17:06:57.226 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2023-11-28T17:06:57.345 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2023-11-28T17:06:57.387 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:18.0.0-7531.g47c1c547.el9. 30/165 2023-11-28T17:06:57.439 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:18.0.0-7531.g47c1c547.el 31/165 2023-11-28T17:06:57.477 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:18.0.0-7531.g47c1c547.el9.x86_6 32/165 2023-11-28T17:06:57.596 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/165 2023-11-28T17:06:57.702 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/165 2023-11-28T17:06:57.904 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 35/165 2023-11-28T17:06:57.975 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 36/165 2023-11-28T17:06:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 37/165 2023-11-28T17:06:58.847 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 38/165 2023-11-28T17:06:58.900 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 39/165 2023-11-28T17:06:58.948 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libs-0.190-1.el9.x86_64 40/165 2023-11-28T17:06:58.995 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:18.0.0-7531.g47c1c547.el9.x86_6 41/165 2023-11-28T17:06:59.057 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 42/165 2023-11-28T17:06:59.271 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 43/165 2023-11-28T17:06:59.285 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 108/165 2023-11-28T17:06:59.321 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 108/165 2023-11-28T17:06:59.348 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 44/165 2023-11-28T17:06:59.490 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-3.el9.x86_64 45/165 2023-11-28T17:06:59.535 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 46/165 2023-11-28T17:06:59.599 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/165 2023-11-28T17:06:59.697 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/165 2023-11-28T17:06:59.753 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/165 2023-11-28T17:06:59.792 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/165 2023-11-28T17:06:59.861 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/165 2023-11-28T17:06:59.918 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/165 2023-11-28T17:06:59.937 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 108/165 2023-11-28T17:06:59.975 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 53/165 2023-11-28T17:06:59.992 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 109/165 2023-11-28T17:07:00.157 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 54/165 2023-11-28T17:07:00.179 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 109/165 2023-11-28T17:07:00.179 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-28T17:07:00.179 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-28T17:07:00.180 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:00.201 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 55/165 2023-11-28T17:07:00.226 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 110/165 2023-11-28T17:07:00.245 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 56/165 2023-11-28T17:07:00.371 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 57/165 2023-11-28T17:07:00.416 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 58/165 2023-11-28T17:07:00.466 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 59/165 2023-11-28T17:07:00.533 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 60/165 2023-11-28T17:07:00.695 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 61/165 2023-11-28T17:07:00.755 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 62/165 2023-11-28T17:07:00.826 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 63/165 2023-11-28T17:07:00.907 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 64/165 2023-11-28T17:07:01.199 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 65/165 2023-11-28T17:07:01.259 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 66/165 2023-11-28T17:07:01.435 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 67/165 2023-11-28T17:07:01.494 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 68/165 2023-11-28T17:07:01.519 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 69/165 2023-11-28T17:07:01.557 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 70/165 2023-11-28T17:07:01.601 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 71/165 2023-11-28T17:07:01.637 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-15.el9.x86_64 72/165 2023-11-28T17:07:01.680 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 73/165 2023-11-28T17:07:01.745 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 74/165 2023-11-28T17:07:01.813 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : uuid-1.6.2-55.el9.x86_64 75/165 2023-11-28T17:07:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/165 2023-11-28T17:07:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/165 2023-11-28T17:07:02.042 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 78/165 2023-11-28T17:07:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 79/165 2023-11-28T17:07:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 80/165 2023-11-28T17:07:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 81/165 2023-11-28T17:07:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 82/165 2023-11-28T17:07:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 83/165 2023-11-28T17:07:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/165 2023-11-28T17:07:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/165 2023-11-28T17:07:03.550 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/165 2023-11-28T17:07:03.599 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/165 2023-11-28T17:07:03.633 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/165 2023-11-28T17:07:03.666 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/165 2023-11-28T17:07:03.933 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.21-2.el9.x86_64 90/165 2023-11-28T17:07:03.962 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 91/165 2023-11-28T17:07:04.060 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 92/165 2023-11-28T17:07:04.852 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 93/165 2023-11-28T17:07:04.983 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 94/165 2023-11-28T17:07:06.571 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 95/165 2023-11-28T17:07:06.657 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 96/165 2023-11-28T17:07:06.696 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 97/165 2023-11-28T17:07:06.726 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jq-1.6-16.el9.x86_64 98/165 2023-11-28T17:07:06.769 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2023-11-28T17:07:06.812 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 100/165 2023-11-28T17:07:07.173 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2023-11-28T17:07:07.199 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 102/165 2023-11-28T17:07:07.319 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 102/165 2023-11-28T17:07:07.347 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:18.0.0-7531.g47c1c547.el9.x86_64 103/165 2023-11-28T17:07:07.433 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2023-11-28T17:07:07.923 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 105/165 2023-11-28T17:07:07.960 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 106/165 2023-11-28T17:07:08.051 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 106/165 2023-11-28T17:07:08.079 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:18.0.0-7531.g47c1c547.el9.x86_64 107/165 2023-11-28T17:07:10.508 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 108/165 2023-11-28T17:07:10.559 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 108/165 2023-11-28T17:07:11.176 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 108/165 2023-11-28T17:07:11.229 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 109/165 2023-11-28T17:07:11.400 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 109/165 2023-11-28T17:07:11.400 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-28T17:07:11.401 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-28T17:07:11.401 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:11.447 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 110/165 2023-11-28T17:07:13.163 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 110/165 2023-11-28T17:07:13.163 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2023-11-28T17:07:13.164 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2023-11-28T17:07:13.164 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2023-11-28T17:07:13.164 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2023-11-28T17:07:13.164 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2023-11-28T17:07:13.164 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /root 2023-11-28T17:07:13.164 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2023-11-28T17:07:13.164 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:13.384 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 111/165 2023-11-28T17:07:13.431 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 111/165 2023-11-28T17:07:13.431 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:13.431 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-28T17:07:13.431 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:15.086 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 112/165 2023-11-28T17:07:15.170 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 112/165 2023-11-28T17:07:15.170 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:15.171 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-28T17:07:15.171 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T17:07:15.171 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T17:07:15.171 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:15.394 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 113/165 2023-11-28T17:07:15.465 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 113/165 2023-11-28T17:07:15.465 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:15.466 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-28T17:07:15.466 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T17:07:15.466 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T17:07:15.466 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:15.954 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 114/165 2023-11-28T17:07:16.025 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 114/165 2023-11-28T17:07:16.025 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:16.025 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-28T17:07:16.025 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T17:07:16.026 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T17:07:16.026 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:16.139 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2023-11-28T17:07:16.815 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 116/165 2023-11-28T17:07:16.847 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 116/165 2023-11-28T17:07:16.892 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7531.g47c1c547.e 117/165 2023-11-28T17:07:16.984 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7531.g47c1c547. 118/165 2023-11-28T17:07:17.010 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noa 119/165 2023-11-28T17:07:17.325 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noa 119/165 2023-11-28T17:07:17.361 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.n 120/165 2023-11-28T17:07:18.479 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.n 120/165 2023-11-28T17:07:18.512 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c 121/165 2023-11-28T17:07:18.654 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c 121/165 2023-11-28T17:07:18.849 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7531.g47c1c547.el 122/165 2023-11-28T17:07:18.881 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 123/165 2023-11-28T17:07:18.970 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 123/165 2023-11-28T17:07:18.970 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:18.970 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-28T17:07:18.970 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T17:07:18.970 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T17:07:18.970 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:19.023 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 124/165 2023-11-28T17:07:19.066 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 124/165 2023-11-28T17:07:19.124 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:18.0.0-7531.g47c1c547.el9.x86_64 125/165 2023-11-28T17:07:19.966 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libtool-2.4.6-45.el9.x86_64 126/165 2023-11-28T17:07:20.000 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 127/165 2023-11-28T17:07:20.072 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 127/165 2023-11-28T17:07:20.072 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:20.072 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-28T17:07:20.072 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T17:07:20.072 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T17:07:20.072 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:20.126 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7531.g47c1c 128/165 2023-11-28T17:07:20.198 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7531.g47c1c 128/165 2023-11-28T17:07:20.198 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:20.199 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-28T17:07:20.199 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:20.277 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 129/165 2023-11-28T17:07:20.354 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 129/165 2023-11-28T17:07:20.354 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:20.354 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-11-28T17:07:20.354 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-28T17:07:20.355 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-28T17:07:20.355 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:20.710 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 130/165 2023-11-28T17:07:20.781 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 130/165 2023-11-28T17:07:20.781 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:20.781 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-28T17:07:20.782 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T17:07:20.782 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T17:07:20.782 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:24.267 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 110/165 2023-11-28T17:07:24.267 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2023-11-28T17:07:24.267 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2023-11-28T17:07:24.267 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2023-11-28T17:07:24.267 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2023-11-28T17:07:24.267 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2023-11-28T17:07:24.267 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2023-11-28T17:07:24.267 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2023-11-28T17:07:24.267 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:24.520 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 111/165 2023-11-28T17:07:24.569 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 111/165 2023-11-28T17:07:24.569 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:24.569 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-28T17:07:24.569 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:24.955 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:18.0.0-7531.g47c1c547.el9.x86_64 131/165 2023-11-28T17:07:25.017 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 132/165 2023-11-28T17:07:25.084 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:18.0.0-7531.g47c1c547.el9.x86_64 133/165 2023-11-28T17:07:25.119 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 134/165 2023-11-28T17:07:25.389 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : elfutils-libelf-devel-0.190-1.el9.x86_64 135/165 2023-11-28T17:07:25.517 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 136/165 2023-11-28T17:07:25.546 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 137/165 2023-11-28T17:07:25.571 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:18.0.0-7531.g47c1c547.el9.x86_ 138/165 2023-11-28T17:07:25.603 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-debuginfod-client-0.190-1.el9.x86_64 139/165 2023-11-28T17:07:25.639 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-team-1:1.45.6-1.el9.x86_64 140/165 2023-11-28T17:07:25.750 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-tui-1:1.45.6-1.el9.x86_64 141/165 2023-11-28T17:07:25.803 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 142/165 2023-11-28T17:07:25.849 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 143/165 2023-11-28T17:07:26.222 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 112/165 2023-11-28T17:07:26.301 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 112/165 2023-11-28T17:07:26.301 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:26.301 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-28T17:07:26.301 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T17:07:26.301 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-28T17:07:26.302 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:26.522 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 113/165 2023-11-28T17:07:26.592 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 113/165 2023-11-28T17:07:26.592 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:26.592 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-28T17:07:26.593 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T17:07:26.593 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-28T17:07:26.593 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:26.808 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T17:07:26.855 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T17:07:26.912 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T17:07:27.090 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 114/165 2023-11-28T17:07:27.169 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 114/165 2023-11-28T17:07:27.169 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:27.169 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-28T17:07:27.169 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T17:07:27.169 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-28T17:07:27.170 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:27.292 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2023-11-28T17:07:27.969 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 116/165 2023-11-28T17:07:28.001 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 116/165 2023-11-28T17:07:28.045 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7531.g47c1c547.e 117/165 2023-11-28T17:07:28.139 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7531.g47c1c547. 118/165 2023-11-28T17:07:28.188 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noa 119/165 2023-11-28T17:07:28.246 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-top-2:18.0.0-7531.g47c1c547.el9.noarch 145/165 2023-11-28T17:07:28.365 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 146/165 2023-11-28T17:07:28.422 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 147/165 2023-11-28T17:07:28.494 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/165 2023-11-28T17:07:28.508 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noa 119/165 2023-11-28T17:07:28.547 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.n 120/165 2023-11-28T17:07:28.641 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/165 2023-11-28T17:07:28.681 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/165 2023-11-28T17:07:28.715 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 151/165 2023-11-28T17:07:28.749 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 152/165 2023-11-28T17:07:28.792 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 153/165 2023-11-28T17:07:28.792 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 154/165 2023-11-28T17:07:28.924 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 154/165 2023-11-28T17:07:28.991 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 155/165 2023-11-28T17:07:29.038 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 156/165 2023-11-28T17:07:29.070 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 157/165 2023-11-28T17:07:29.070 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 158/165 2023-11-28T17:07:29.207 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 158/165 2023-11-28T17:07:29.238 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 159/165 2023-11-28T17:07:29.278 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T17:07:29.278 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T17:07:29.620 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T17:07:29.620 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2023-11-28T17:07:29.670 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.n 120/165 2023-11-28T17:07:29.706 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c 121/165 2023-11-28T17:07:29.788 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2023-11-28T17:07:29.824 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 162/165 2023-11-28T17:07:29.847 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c 121/165 2023-11-28T17:07:29.859 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 163/165 2023-11-28T17:07:29.893 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 164/165 2023-11-28T17:07:30.014 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7531.g47c1c547.el 122/165 2023-11-28T17:07:30.020 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 165/165 2023-11-28T17:07:30.050 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 123/165 2023-11-28T17:07:30.138 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 123/165 2023-11-28T17:07:30.138 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:30.138 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-28T17:07:30.138 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T17:07:30.139 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-28T17:07:30.139 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:30.201 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 124/165 2023-11-28T17:07:30.240 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 124/165 2023-11-28T17:07:30.276 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 165/165 2023-11-28T17:07:30.276 INFO:teuthology.orchestra.run.smithi156.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-11-28T17:07:30.276 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:30.332 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:18.0.0-7531.g47c1c547.el9.x86_64 125/165 2023-11-28T17:07:31.188 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtool-2.4.6-45.el9.x86_64 126/165 2023-11-28T17:07:31.219 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 127/165 2023-11-28T17:07:31.293 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 127/165 2023-11-28T17:07:31.293 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:31.293 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-28T17:07:31.293 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T17:07:31.293 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-28T17:07:31.293 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:31.345 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7531.g47c1c 128/165 2023-11-28T17:07:31.444 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7531.g47c1c 128/165 2023-11-28T17:07:31.444 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:31.444 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-28T17:07:31.445 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:31.521 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 129/165 2023-11-28T17:07:31.602 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 129/165 2023-11-28T17:07:31.603 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:31.603 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-11-28T17:07:31.603 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-28T17:07:31.603 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-11-28T17:07:31.603 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:31.981 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 130/165 2023-11-28T17:07:32.055 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 130/165 2023-11-28T17:07:32.055 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-28T17:07:32.055 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-28T17:07:32.056 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T17:07:32.056 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-28T17:07:32.056 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:34.865 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:18.0.0-7531.g47c1c547.el9.x86_64 1/165 2023-11-28T17:07:34.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 2/165 2023-11-28T17:07:34.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 3/165 2023-11-28T17:07:34.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 4/165 2023-11-28T17:07:34.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7531.g47c1c 5/165 2023-11-28T17:07:34.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 6/165 2023-11-28T17:07:34.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 7/165 2023-11-28T17:07:34.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 8/165 2023-11-28T17:07:34.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 9/165 2023-11-28T17:07:34.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 10/165 2023-11-28T17:07:34.866 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 11/165 2023-11-28T17:07:34.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:18.0.0-7531.g47c1c547.el9.x86_64 12/165 2023-11-28T17:07:34.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 13/165 2023-11-28T17:07:34.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:18.0.0-7531.g47c1c547.el9.x86_ 14/165 2023-11-28T17:07:34.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 15/165 2023-11-28T17:07:34.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 16/165 2023-11-28T17:07:34.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:18.0.0-7531.g47c1c547.el9.x86_6 17/165 2023-11-28T17:07:34.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86 18/165 2023-11-28T17:07:34.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 19/165 2023-11-28T17:07:34.867 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7531.g47c1c547.el 20/165 2023-11-28T17:07:34.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:18.0.0-7531.g47c1c547.el9. 21/165 2023-11-28T17:07:34.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:18.0.0-7531.g47c1c547.el9.x86_6 22/165 2023-11-28T17:07:34.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:18.0.0-7531.g47c1c547.el9.x86_64 23/165 2023-11-28T17:07:34.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:18.0.0-7531.g47c1c547.el9.x86_64 24/165 2023-11-28T17:07:34.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:18.0.0-7531.g47c1c547.el9.x86_64 25/165 2023-11-28T17:07:34.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 26/165 2023-11-28T17:07:34.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 27/165 2023-11-28T17:07:34.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:18.0.0-7531.g47c1c547.el9.x86_64 28/165 2023-11-28T17:07:34.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7531.g47c1c547. 29/165 2023-11-28T17:07:34.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noa 30/165 2023-11-28T17:07:34.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.n 31/165 2023-11-28T17:07:34.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c 32/165 2023-11-28T17:07:34.868 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7531.g47c1c547.el 33/165 2023-11-28T17:07:34.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 34/165 2023-11-28T17:07:34.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7531.g47c1c547.e 35/165 2023-11-28T17:07:34.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 36/165 2023-11-28T17:07:34.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 37/165 2023-11-28T17:07:34.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-top-2:18.0.0-7531.g47c1c547.el9.noarch 38/165 2023-11-28T17:07:34.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/165 2023-11-28T17:07:34.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/165 2023-11-28T17:07:34.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/165 2023-11-28T17:07:34.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-devel-0.190-1.el9.x86_64 42/165 2023-11-28T17:07:34.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/165 2023-11-28T17:07:34.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/165 2023-11-28T17:07:34.869 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/165 2023-11-28T17:07:34.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/165 2023-11-28T17:07:34.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : jq-1.6-16.el9.x86_64 47/165 2023-11-28T17:07:34.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/165 2023-11-28T17:07:34.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/165 2023-11-28T17:07:34.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/165 2023-11-28T17:07:34.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/165 2023-11-28T17:07:34.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/165 2023-11-28T17:07:34.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 53/165 2023-11-28T17:07:34.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/165 2023-11-28T17:07:34.870 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/165 2023-11-28T17:07:34.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/165 2023-11-28T17:07:34.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 57/165 2023-11-28T17:07:34.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 58/165 2023-11-28T17:07:34.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 59/165 2023-11-28T17:07:34.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/165 2023-11-28T17:07:34.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/165 2023-11-28T17:07:34.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/165 2023-11-28T17:07:34.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/165 2023-11-28T17:07:34.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/165 2023-11-28T17:07:34.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 65/165 2023-11-28T17:07:34.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 66/165 2023-11-28T17:07:34.871 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 67/165 2023-11-28T17:07:34.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 68/165 2023-11-28T17:07:34.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 69/165 2023-11-28T17:07:34.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 70/165 2023-11-28T17:07:34.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 71/165 2023-11-28T17:07:34.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 72/165 2023-11-28T17:07:34.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/165 2023-11-28T17:07:34.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 74/165 2023-11-28T17:07:34.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 75/165 2023-11-28T17:07:34.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 76/165 2023-11-28T17:07:34.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 77/165 2023-11-28T17:07:34.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 78/165 2023-11-28T17:07:34.872 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 79/165 2023-11-28T17:07:34.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 80/165 2023-11-28T17:07:34.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 81/165 2023-11-28T17:07:34.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 82/165 2023-11-28T17:07:34.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 83/165 2023-11-28T17:07:34.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 84/165 2023-11-28T17:07:34.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 85/165 2023-11-28T17:07:34.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 86/165 2023-11-28T17:07:34.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 87/165 2023-11-28T17:07:34.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 88/165 2023-11-28T17:07:34.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 89/165 2023-11-28T17:07:34.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ply-3.11-14.el9.noarch 90/165 2023-11-28T17:07:34.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 91/165 2023-11-28T17:07:34.873 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 92/165 2023-11-28T17:07:34.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 93/165 2023-11-28T17:07:34.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 94/165 2023-11-28T17:07:34.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/165 2023-11-28T17:07:34.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/165 2023-11-28T17:07:34.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/165 2023-11-28T17:07:34.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 98/165 2023-11-28T17:07:34.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 99/165 2023-11-28T17:07:34.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/165 2023-11-28T17:07:34.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/165 2023-11-28T17:07:34.874 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/165 2023-11-28T17:07:34.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 103/165 2023-11-28T17:07:34.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/165 2023-11-28T17:07:34.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/165 2023-11-28T17:07:34.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/165 2023-11-28T17:07:34.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/165 2023-11-28T17:07:34.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/165 2023-11-28T17:07:34.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/165 2023-11-28T17:07:34.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 110/165 2023-11-28T17:07:34.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/165 2023-11-28T17:07:34.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/165 2023-11-28T17:07:34.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/165 2023-11-28T17:07:34.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/165 2023-11-28T17:07:34.875 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/165 2023-11-28T17:07:34.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/165 2023-11-28T17:07:34.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/165 2023-11-28T17:07:34.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/165 2023-11-28T17:07:34.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/165 2023-11-28T17:07:34.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/165 2023-11-28T17:07:34.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/165 2023-11-28T17:07:34.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/165 2023-11-28T17:07:34.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/165 2023-11-28T17:07:34.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/165 2023-11-28T17:07:34.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/165 2023-11-28T17:07:34.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/165 2023-11-28T17:07:34.876 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/165 2023-11-28T17:07:34.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/165 2023-11-28T17:07:34.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/165 2023-11-28T17:07:34.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/165 2023-11-28T17:07:34.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/165 2023-11-28T17:07:34.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/165 2023-11-28T17:07:34.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/165 2023-11-28T17:07:34.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/165 2023-11-28T17:07:34.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/165 2023-11-28T17:07:34.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 136/165 2023-11-28T17:07:34.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/165 2023-11-28T17:07:34.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 138/165 2023-11-28T17:07:34.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/165 2023-11-28T17:07:34.877 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/165 2023-11-28T17:07:34.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/165 2023-11-28T17:07:34.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-1:1.45.6-1.el9.x86_64 142/165 2023-11-28T17:07:34.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 143/165 2023-11-28T17:07:34.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-libnm-1:1.45.6-1.el9.x86_64 144/165 2023-11-28T17:07:34.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 145/165 2023-11-28T17:07:34.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-team-1:1.45.6-1.el9.x86_64 146/165 2023-11-28T17:07:34.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/165 2023-11-28T17:07:34.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-tui-1:1.45.6-1.el9.x86_64 148/165 2023-11-28T17:07:34.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 149/165 2023-11-28T17:07:34.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : acl-2.3.1-4.el9.x86_64 150/165 2023-11-28T17:07:34.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : acl-2.3.1-3.el9.x86_64 151/165 2023-11-28T17:07:34.878 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-debuginfod-client-0.190-1.el9.x86_64 152/165 2023-11-28T17:07:34.879 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/165 2023-11-28T17:07:34.879 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-0.190-1.el9.x86_64 154/165 2023-11-28T17:07:34.879 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 155/165 2023-11-28T17:07:34.879 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libs-0.190-1.el9.x86_64 156/165 2023-11-28T17:07:34.879 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 157/165 2023-11-28T17:07:34.879 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 158/165 2023-11-28T17:07:34.879 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 159/165 2023-11-28T17:07:34.879 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 160/165 2023-11-28T17:07:34.879 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 161/165 2023-11-28T17:07:34.879 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 162/165 2023-11-28T17:07:34.879 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 163/165 2023-11-28T17:07:34.879 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 164/165 2023-11-28T17:07:36.213 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:18.0.0-7531.g47c1c547.el9.x86_64 131/165 2023-11-28T17:07:36.274 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 132/165 2023-11-28T17:07:36.333 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:18.0.0-7531.g47c1c547.el9.x86_64 133/165 2023-11-28T17:07:36.368 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 134/165 2023-11-28T17:07:36.627 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : elfutils-libelf-devel-0.190-1.el9.x86_64 135/165 2023-11-28T17:07:36.749 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 136/165 2023-11-28T17:07:36.787 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 137/165 2023-11-28T17:07:36.821 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:18.0.0-7531.g47c1c547.el9.x86_ 138/165 2023-11-28T17:07:36.884 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-debuginfod-client-0.190-1.el9.x86_64 139/165 2023-11-28T17:07:36.931 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-team-1:1.45.6-1.el9.x86_64 140/165 2023-11-28T17:07:36.983 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 165/165 2023-11-28T17:07:36.983 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:36.983 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2023-11-28T17:07:36.983 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-1:1.45.6-1.el9.x86_64 2023-11-28T17:07:36.983 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-libnm-1:1.45.6-1.el9.x86_64 2023-11-28T17:07:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-team-1:1.45.6-1.el9.x86_64 2023-11-28T17:07:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-tui-1:1.45.6-1.el9.x86_64 2023-11-28T17:07:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: acl-2.3.1-4.el9.x86_64 2023-11-28T17:07:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-debuginfod-client-0.190-1.el9.x86_64 2023-11-28T17:07:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-0.190-1.el9.x86_64 2023-11-28T17:07:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libs-0.190-1.el9.x86_64 2023-11-28T17:07:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-2.3.1-4.el9.x86_64 2023-11-28T17:07:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-11-28T17:07:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-1:3.0.7-25.el9.x86_64 2023-11-28T17:07:36.984 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2023-11-28T17:07:36.985 INFO:teuthology.orchestra.run.smithi156.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-11-28T17:07:36.985 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-11-28T17:07:36.985 INFO:teuthology.orchestra.run.smithi156.stdout: automake-1.16.2-8.el9.noarch 2023-11-28T17:07:36.985 INFO:teuthology.orchestra.run.smithi156.stdout: bison-3.7.4-5.el9.x86_64 2023-11-28T17:07:36.985 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-28T17:07:36.985 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.985 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.985 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.985 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.985 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:36.985 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.985 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.986 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.986 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:36.986 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:36.986 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:36.986 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:36.986 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:36.986 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.986 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.986 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:36.986 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.986 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.987 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.987 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:36.987 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:36.987 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.987 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:36.987 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel-0.190-1.el9.x86_64 2023-11-28T17:07:36.987 INFO:teuthology.orchestra.run.smithi156.stdout: flex-2.6.4-9.el9.x86_64 2023-11-28T17:07:36.987 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-28T17:07:36.987 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-28T17:07:36.987 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-28T17:07:36.987 INFO:teuthology.orchestra.run.smithi156.stdout: fuse-2.9.9-15.el9.x86_64 2023-11-28T17:07:36.988 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-28T17:07:36.988 INFO:teuthology.orchestra.run.smithi156.stdout: jq-1.6-16.el9.x86_64 2023-11-28T17:07:36.988 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-11-28T17:07:36.988 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-28T17:07:36.988 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-28T17:07:36.988 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.988 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.988 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.988 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-28T17:07:36.988 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-28T17:07:36.988 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-28T17:07:36.989 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-28T17:07:36.989 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-28T17:07:36.989 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.989 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.989 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-28T17:07:36.989 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.989 INFO:teuthology.orchestra.run.smithi156.stdout: libtool-2.4.6-45.el9.x86_64 2023-11-28T17:07:36.989 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-28T17:07:36.989 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-11-28T17:07:36.989 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-11-28T17:07:36.989 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-28T17:07:36.989 INFO:teuthology.orchestra.run.smithi156.stdout: lua-5.4.4-4.el9.x86_64 2023-11-28T17:07:36.990 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-11-28T17:07:36.990 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks-3.9.2-1.el9.noarch 2023-11-28T17:07:36.990 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-28T17:07:36.990 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-28T17:07:36.990 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-28T17:07:36.990 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2023-11-28T17:07:36.990 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-28T17:07:36.990 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-13.11-1.el9.x86_64 2023-11-28T17:07:36.990 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-11-28T17:07:36.990 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-11-28T17:07:36.990 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server-13.11-1.el9.x86_64 2023-11-28T17:07:36.990 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-28T17:07:36.991 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-28T17:07:36.991 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-28T17:07:36.991 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-28T17:07:36.991 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.991 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.991 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.991 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-28T17:07:36.991 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-28T17:07:36.991 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-28T17:07:36.991 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-28T17:07:36.991 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-28T17:07:36.992 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-28T17:07:36.992 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-28T17:07:36.992 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-28T17:07:36.992 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-28T17:07:36.992 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-28T17:07:36.992 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-28T17:07:36.992 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-28T17:07:36.992 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-28T17:07:36.992 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-28T17:07:36.992 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-28T17:07:36.992 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-28T17:07:36.992 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-28T17:07:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-28T17:07:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-28T17:07:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-28T17:07:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-28T17:07:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-28T17:07:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-28T17:07:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-28T17:07:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-28T17:07:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-28T17:07:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-28T17:07:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply-3.11-14.el9.noarch 2023-11-28T17:07:36.993 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-28T17:07:36.994 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-28T17:07:36.994 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-28T17:07:36.994 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-28T17:07:36.994 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-28T17:07:36.994 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-28T17:07:36.994 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-28T17:07:36.994 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.994 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.994 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-28T17:07:36.994 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-28T17:07:36.994 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-28T17:07:36.995 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.995 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-28T17:07:36.995 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-28T17:07:36.995 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-28T17:07:36.995 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-28T17:07:36.995 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-28T17:07:36.995 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-28T17:07:36.996 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-28T17:07:36.996 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-28T17:07:36.996 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-28T17:07:36.996 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-28T17:07:36.996 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-28T17:07:36.996 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.996 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.996 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:36.996 INFO:teuthology.orchestra.run.smithi156.stdout: re2-1:20211101-3.el9.x86_64 2023-11-28T17:07:36.997 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-28T17:07:36.997 INFO:teuthology.orchestra.run.smithi156.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-28T17:07:36.997 INFO:teuthology.orchestra.run.smithi156.stdout: uuid-1.6.2-55.el9.x86_64 2023-11-28T17:07:36.997 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-11-28T17:07:36.997 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-28T17:07:36.997 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-11-28T17:07:36.997 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:36.997 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-11-28T17:07:37.039 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-tui-1:1.45.6-1.el9.x86_64 141/165 2023-11-28T17:07:37.106 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 142/165 2023-11-28T17:07:37.148 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 143/165 2023-11-28T17:07:37.567 DEBUG:teuthology.parallel:result is None 2023-11-28T17:07:38.064 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T17:07:38.104 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T17:07:38.164 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 144/165 2023-11-28T17:07:39.484 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-top-2:18.0.0-7531.g47c1c547.el9.noarch 145/165 2023-11-28T17:07:39.609 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 146/165 2023-11-28T17:07:39.689 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 147/165 2023-11-28T17:07:39.776 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/165 2023-11-28T17:07:39.928 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/165 2023-11-28T17:07:39.962 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/165 2023-11-28T17:07:39.996 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 151/165 2023-11-28T17:07:40.029 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 152/165 2023-11-28T17:07:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 153/165 2023-11-28T17:07:40.071 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 154/165 2023-11-28T17:07:40.229 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 154/165 2023-11-28T17:07:40.280 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 155/165 2023-11-28T17:07:40.327 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 156/165 2023-11-28T17:07:40.359 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 157/165 2023-11-28T17:07:40.359 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 158/165 2023-11-28T17:07:40.488 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 158/165 2023-11-28T17:07:40.518 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 159/165 2023-11-28T17:07:40.558 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T17:07:40.558 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T17:07:40.893 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 160/165 2023-11-28T17:07:40.893 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2023-11-28T17:07:41.077 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 161/165 2023-11-28T17:07:41.113 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 162/165 2023-11-28T17:07:41.147 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 163/165 2023-11-28T17:07:41.189 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 164/165 2023-11-28T17:07:41.316 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 165/165 2023-11-28T17:07:41.577 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 165/165 2023-11-28T17:07:41.577 INFO:teuthology.orchestra.run.smithi116.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-11-28T17:07:41.577 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:46.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:18.0.0-7531.g47c1c547.el9.x86_64 1/165 2023-11-28T17:07:46.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 2/165 2023-11-28T17:07:46.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 3/165 2023-11-28T17:07:46.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 4/165 2023-11-28T17:07:46.154 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7531.g47c1c 5/165 2023-11-28T17:07:46.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 6/165 2023-11-28T17:07:46.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 7/165 2023-11-28T17:07:46.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 8/165 2023-11-28T17:07:46.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 9/165 2023-11-28T17:07:46.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 10/165 2023-11-28T17:07:46.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 11/165 2023-11-28T17:07:46.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:18.0.0-7531.g47c1c547.el9.x86_64 12/165 2023-11-28T17:07:46.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 13/165 2023-11-28T17:07:46.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:18.0.0-7531.g47c1c547.el9.x86_ 14/165 2023-11-28T17:07:46.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 15/165 2023-11-28T17:07:46.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 16/165 2023-11-28T17:07:46.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:18.0.0-7531.g47c1c547.el9.x86_6 17/165 2023-11-28T17:07:46.155 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86 18/165 2023-11-28T17:07:46.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 19/165 2023-11-28T17:07:46.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7531.g47c1c547.el 20/165 2023-11-28T17:07:46.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:18.0.0-7531.g47c1c547.el9. 21/165 2023-11-28T17:07:46.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:18.0.0-7531.g47c1c547.el9.x86_6 22/165 2023-11-28T17:07:46.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:18.0.0-7531.g47c1c547.el9.x86_64 23/165 2023-11-28T17:07:46.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:18.0.0-7531.g47c1c547.el9.x86_64 24/165 2023-11-28T17:07:46.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:18.0.0-7531.g47c1c547.el9.x86_64 25/165 2023-11-28T17:07:46.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 26/165 2023-11-28T17:07:46.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 27/165 2023-11-28T17:07:46.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:18.0.0-7531.g47c1c547.el9.x86_64 28/165 2023-11-28T17:07:46.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7531.g47c1c547. 29/165 2023-11-28T17:07:46.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noa 30/165 2023-11-28T17:07:46.156 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.n 31/165 2023-11-28T17:07:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c 32/165 2023-11-28T17:07:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7531.g47c1c547.el 33/165 2023-11-28T17:07:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 34/165 2023-11-28T17:07:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7531.g47c1c547.e 35/165 2023-11-28T17:07:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 36/165 2023-11-28T17:07:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 37/165 2023-11-28T17:07:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-top-2:18.0.0-7531.g47c1c547.el9.noarch 38/165 2023-11-28T17:07:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/165 2023-11-28T17:07:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/165 2023-11-28T17:07:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/165 2023-11-28T17:07:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-devel-0.190-1.el9.x86_64 42/165 2023-11-28T17:07:46.157 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/165 2023-11-28T17:07:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/165 2023-11-28T17:07:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/165 2023-11-28T17:07:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/165 2023-11-28T17:07:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jq-1.6-16.el9.x86_64 47/165 2023-11-28T17:07:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/165 2023-11-28T17:07:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/165 2023-11-28T17:07:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/165 2023-11-28T17:07:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/165 2023-11-28T17:07:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/165 2023-11-28T17:07:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 53/165 2023-11-28T17:07:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/165 2023-11-28T17:07:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/165 2023-11-28T17:07:46.158 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/165 2023-11-28T17:07:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 57/165 2023-11-28T17:07:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 58/165 2023-11-28T17:07:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 59/165 2023-11-28T17:07:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/165 2023-11-28T17:07:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/165 2023-11-28T17:07:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/165 2023-11-28T17:07:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/165 2023-11-28T17:07:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/165 2023-11-28T17:07:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 65/165 2023-11-28T17:07:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 66/165 2023-11-28T17:07:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 67/165 2023-11-28T17:07:46.159 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 68/165 2023-11-28T17:07:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 69/165 2023-11-28T17:07:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 70/165 2023-11-28T17:07:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 71/165 2023-11-28T17:07:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 72/165 2023-11-28T17:07:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/165 2023-11-28T17:07:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 74/165 2023-11-28T17:07:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 75/165 2023-11-28T17:07:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 76/165 2023-11-28T17:07:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 77/165 2023-11-28T17:07:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 78/165 2023-11-28T17:07:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 79/165 2023-11-28T17:07:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 80/165 2023-11-28T17:07:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 81/165 2023-11-28T17:07:46.160 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 82/165 2023-11-28T17:07:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 83/165 2023-11-28T17:07:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 84/165 2023-11-28T17:07:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 85/165 2023-11-28T17:07:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 86/165 2023-11-28T17:07:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 87/165 2023-11-28T17:07:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 88/165 2023-11-28T17:07:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 89/165 2023-11-28T17:07:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 90/165 2023-11-28T17:07:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 91/165 2023-11-28T17:07:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 92/165 2023-11-28T17:07:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 93/165 2023-11-28T17:07:46.161 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 94/165 2023-11-28T17:07:46.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/165 2023-11-28T17:07:46.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/165 2023-11-28T17:07:46.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/165 2023-11-28T17:07:46.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 98/165 2023-11-28T17:07:46.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 99/165 2023-11-28T17:07:46.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/165 2023-11-28T17:07:46.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/165 2023-11-28T17:07:46.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/165 2023-11-28T17:07:46.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 103/165 2023-11-28T17:07:46.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/165 2023-11-28T17:07:46.162 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/165 2023-11-28T17:07:46.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/165 2023-11-28T17:07:46.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/165 2023-11-28T17:07:46.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/165 2023-11-28T17:07:46.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/165 2023-11-28T17:07:46.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 110/165 2023-11-28T17:07:46.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/165 2023-11-28T17:07:46.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/165 2023-11-28T17:07:46.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/165 2023-11-28T17:07:46.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/165 2023-11-28T17:07:46.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/165 2023-11-28T17:07:46.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/165 2023-11-28T17:07:46.163 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/165 2023-11-28T17:07:46.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/165 2023-11-28T17:07:46.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/165 2023-11-28T17:07:46.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/165 2023-11-28T17:07:46.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/165 2023-11-28T17:07:46.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/165 2023-11-28T17:07:46.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/165 2023-11-28T17:07:46.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/165 2023-11-28T17:07:46.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/165 2023-11-28T17:07:46.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/165 2023-11-28T17:07:46.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/165 2023-11-28T17:07:46.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/165 2023-11-28T17:07:46.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/165 2023-11-28T17:07:46.164 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/165 2023-11-28T17:07:46.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/165 2023-11-28T17:07:46.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/165 2023-11-28T17:07:46.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/165 2023-11-28T17:07:46.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/165 2023-11-28T17:07:46.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/165 2023-11-28T17:07:46.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 136/165 2023-11-28T17:07:46.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/165 2023-11-28T17:07:46.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 138/165 2023-11-28T17:07:46.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/165 2023-11-28T17:07:46.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/165 2023-11-28T17:07:46.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/165 2023-11-28T17:07:46.165 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.45.6-1.el9.x86_64 142/165 2023-11-28T17:07:46.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 143/165 2023-11-28T17:07:46.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.45.6-1.el9.x86_64 144/165 2023-11-28T17:07:46.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 145/165 2023-11-28T17:07:46.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.45.6-1.el9.x86_64 146/165 2023-11-28T17:07:46.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/165 2023-11-28T17:07:46.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.45.6-1.el9.x86_64 148/165 2023-11-28T17:07:46.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 149/165 2023-11-28T17:07:46.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.3.1-4.el9.x86_64 150/165 2023-11-28T17:07:46.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.3.1-3.el9.x86_64 151/165 2023-11-28T17:07:46.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.190-1.el9.x86_64 152/165 2023-11-28T17:07:46.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/165 2023-11-28T17:07:46.166 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.190-1.el9.x86_64 154/165 2023-11-28T17:07:46.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 155/165 2023-11-28T17:07:46.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.190-1.el9.x86_64 156/165 2023-11-28T17:07:46.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 157/165 2023-11-28T17:07:46.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 158/165 2023-11-28T17:07:46.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 159/165 2023-11-28T17:07:46.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 160/165 2023-11-28T17:07:46.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 161/165 2023-11-28T17:07:46.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 162/165 2023-11-28T17:07:46.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 163/165 2023-11-28T17:07:46.167 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 164/165 2023-11-28T17:07:48.394 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 165/165 2023-11-28T17:07:48.394 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:48.394 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2023-11-28T17:07:48.394 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-1:1.45.6-1.el9.x86_64 2023-11-28T17:07:48.395 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm-1:1.45.6-1.el9.x86_64 2023-11-28T17:07:48.395 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team-1:1.45.6-1.el9.x86_64 2023-11-28T17:07:48.395 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui-1:1.45.6-1.el9.x86_64 2023-11-28T17:07:48.395 INFO:teuthology.orchestra.run.smithi116.stdout: acl-2.3.1-4.el9.x86_64 2023-11-28T17:07:48.395 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client-0.190-1.el9.x86_64 2023-11-28T17:07:48.395 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-0.190-1.el9.x86_64 2023-11-28T17:07:48.395 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs-0.190-1.el9.x86_64 2023-11-28T17:07:48.395 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-2.3.1-4.el9.x86_64 2023-11-28T17:07:48.395 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-11-28T17:07:48.395 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.395 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.395 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-1:3.0.7-25.el9.x86_64 2023-11-28T17:07:48.396 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2023-11-28T17:07:48.396 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-11-28T17:07:48.396 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-11-28T17:07:48.396 INFO:teuthology.orchestra.run.smithi116.stdout: automake-1.16.2-8.el9.noarch 2023-11-28T17:07:48.396 INFO:teuthology.orchestra.run.smithi116.stdout: bison-3.7.4-5.el9.x86_64 2023-11-28T17:07:48.396 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-28T17:07:48.396 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.396 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.396 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.396 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.396 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:48.396 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.396 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.397 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.397 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:48.397 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:48.397 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:48.397 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:48.397 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:48.397 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.397 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.397 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:48.397 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.397 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.397 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.397 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:48.398 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:48.398 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.398 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top-2:18.0.0-7531.g47c1c547.el9.noarch 2023-11-28T17:07:48.398 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel-0.190-1.el9.x86_64 2023-11-28T17:07:48.398 INFO:teuthology.orchestra.run.smithi116.stdout: flex-2.6.4-9.el9.x86_64 2023-11-28T17:07:48.398 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-28T17:07:48.398 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-28T17:07:48.398 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-28T17:07:48.398 INFO:teuthology.orchestra.run.smithi116.stdout: fuse-2.9.9-15.el9.x86_64 2023-11-28T17:07:48.398 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-28T17:07:48.398 INFO:teuthology.orchestra.run.smithi116.stdout: jq-1.6-16.el9.x86_64 2023-11-28T17:07:48.398 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-11-28T17:07:48.398 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-28T17:07:48.399 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-28T17:07:48.399 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.399 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.399 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.399 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-28T17:07:48.399 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-28T17:07:48.399 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-28T17:07:48.399 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-28T17:07:48.399 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-28T17:07:48.399 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.399 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.399 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-28T17:07:48.400 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.400 INFO:teuthology.orchestra.run.smithi116.stdout: libtool-2.4.6-45.el9.x86_64 2023-11-28T17:07:48.400 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-28T17:07:48.400 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-11-28T17:07:48.400 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-11-28T17:07:48.400 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-28T17:07:48.400 INFO:teuthology.orchestra.run.smithi116.stdout: lua-5.4.4-4.el9.x86_64 2023-11-28T17:07:48.400 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-11-28T17:07:48.400 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks-3.9.2-1.el9.noarch 2023-11-28T17:07:48.400 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-28T17:07:48.400 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-28T17:07:48.400 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-28T17:07:48.400 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2023-11-28T17:07:48.401 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-28T17:07:48.401 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-13.11-1.el9.x86_64 2023-11-28T17:07:48.401 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-11-28T17:07:48.401 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-11-28T17:07:48.401 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server-13.11-1.el9.x86_64 2023-11-28T17:07:48.401 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-28T17:07:48.401 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-28T17:07:48.401 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-28T17:07:48.401 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-28T17:07:48.401 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.401 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.401 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.402 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-28T17:07:48.402 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-28T17:07:48.402 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-28T17:07:48.402 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-28T17:07:48.402 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-28T17:07:48.402 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-28T17:07:48.402 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-28T17:07:48.402 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-28T17:07:48.402 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-28T17:07:48.402 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-28T17:07:48.402 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-28T17:07:48.402 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-28T17:07:48.402 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-28T17:07:48.403 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-28T17:07:48.403 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-28T17:07:48.403 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-28T17:07:48.403 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-28T17:07:48.403 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-28T17:07:48.403 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-28T17:07:48.403 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-28T17:07:48.403 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-28T17:07:48.403 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-28T17:07:48.403 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-28T17:07:48.403 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-28T17:07:48.403 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-28T17:07:48.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-28T17:07:48.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-28T17:07:48.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2023-11-28T17:07:48.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-28T17:07:48.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-28T17:07:48.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-28T17:07:48.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-28T17:07:48.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-28T17:07:48.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-28T17:07:48.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-28T17:07:48.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-28T17:07:48.405 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-28T17:07:48.405 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-28T17:07:48.405 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.405 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-28T17:07:48.405 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-28T17:07:48.405 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-28T17:07:48.405 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-28T17:07:48.405 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-28T17:07:48.405 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-28T17:07:48.405 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-28T17:07:48.405 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-28T17:07:48.405 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-28T17:07:48.406 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-28T17:07:48.406 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-28T17:07:48.406 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.406 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.406 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:18.0.0-7531.g47c1c547.el9.x86_64 2023-11-28T17:07:48.406 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-3.el9.x86_64 2023-11-28T17:07:48.406 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-28T17:07:48.406 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-28T17:07:48.406 INFO:teuthology.orchestra.run.smithi116.stdout: uuid-1.6.2-55.el9.x86_64 2023-11-28T17:07:48.406 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-11-28T17:07:48.406 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-28T17:07:48.407 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-11-28T17:07:48.407 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:48.407 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-11-28T17:07:49.031 DEBUG:teuthology.parallel:result is None 2023-11-28T17:07:49.032 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T17:07:49.168 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-28T17:07:49.226 INFO:teuthology.orchestra.run.smithi116.stdout:18.0.0-7531.g47c1c547.el9 2023-11-28T17:07:49.226 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7531.g47c1c547.el9 2023-11-28T17:07:49.226 INFO:teuthology.task.install:The correct ceph version 18.0.0-7531.g47c1c547 is installed. 2023-11-28T17:07:49.227 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=47c1c547372ed5bff7984647dc200bac0e019caf 2023-11-28T17:07:49.363 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-28T17:07:49.439 INFO:teuthology.orchestra.run.smithi156.stdout:18.0.0-7531.g47c1c547.el9 2023-11-28T17:07:49.439 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7531.g47c1c547.el9 2023-11-28T17:07:49.440 INFO:teuthology.task.install:The correct ceph version 18.0.0-7531.g47c1c547 is installed. 2023-11-28T17:07:49.440 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-28T17:07:49.440 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:07:49.440 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T17:07:49.528 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:07:49.528 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T17:07:49.636 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-28T17:07:49.636 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:07:49.637 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T17:07:49.716 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T17:07:49.839 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:07:49.839 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T17:07:49.940 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T17:07:50.071 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-28T17:07:50.072 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:07:50.072 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T17:07:50.155 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T17:07:50.289 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:07:50.289 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T17:07:50.392 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T17:07:50.531 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-28T17:07:50.532 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:07:50.532 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T17:07:50.611 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T17:07:50.722 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:07:50.722 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T17:07:50.825 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T17:07:50.920 INFO:teuthology.run_tasks:Running task ceph... 2023-11-28T17:07:50.994 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-28T17:07:50.994 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /var/log/ceph 2023-11-28T17:07:51.000 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /var/log/ceph 2023-11-28T17:07:51.103 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-28T17:07:51.104 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-28T17:07:51.109 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-28T17:07:51.205 INFO:tasks.ceph:Creating extra log directories... 2023-11-28T17:07:51.205 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-28T17:07:51.210 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-28T17:07:51.317 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-11-28T17:07:51.318 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-11-28T17:07:51.318 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-11-28T17:07:51.318 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:07:51.318 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-28T17:07:51.358 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-28T17:07:51.568 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-28T17:07:51.654 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:07:51.654 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-28T17:07:51.714 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-28T17:07:51.989 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-28T17:07:52.095 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-28T17:07:52.095 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'client session with non-allowable root'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-28T17:07:52.095 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360', 'branch': 'wip-vshankar-testing-20231127.102654', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7469360', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20231127.102654'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'client session with non-allowable root'], 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20231127.102654', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1098, 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'sleep_before_teardown': 0, 'subset': '854/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20231127.102654', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi116.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAd/T2uVnzOHp9OpssfoR7Mv6f0LDVHPMou5adf1YT1rkfXwdtt2VxAB+RNPeCVBdlQuqDkrd4w7MPavNdXVc+w=', 'smithi156.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHTMAA4YyLkkHqT3TQSZosSRhTfgBHoAWQmyxP5/lAyfF2GZ1OX97dufJBP9jzaMDtkuTglviynajI16ajnPc+k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'client session with non-allowable root'], 'sha1': '47c1c547372ed5bff7984647dc200bac0e019caf', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_sessionmap']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '5ae09a5cc7064ca2bf994df19f3006b79794789a', 'timestamp': '2023-11-28_06:23:42', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-28T17:07:52.096 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-28T17:07:52.102 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-28T17:07:52.156 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/run/ceph 2023-11-28T17:07:52.191 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/run/ceph 2023-11-28T17:07:52.292 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:07:52.292 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T17:07:52.336 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T17:07:52.337 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2023-11-28T17:07:52.421 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T17:07:52.421 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:07:52.421 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 632 Links: 1 2023-11-28T17:07:52.421 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:07:52.421 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:07:52.421 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-11-28 17:07:42.047635608 +0000 2023-11-28T17:07:52.421 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-11-28 17:04:59.567522053 +0000 2023-11-28T17:07:52.422 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-11-28 17:04:59.567522053 +0000 2023-11-28T17:07:52.422 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2023-11-28 17:04:59.567522053 +0000 2023-11-28T17:07:52.422 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T17:07:52.538 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-11-28T17:07:52.539 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-11-28T17:07:52.539 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000372339 s, 1.4 MB/s 2023-11-28T17:07:52.542 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T17:07:52.588 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2023-11-28T17:07:52.670 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T17:07:52.670 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:07:52.670 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 652 Links: 1 2023-11-28T17:07:52.671 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:07:52.671 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:07:52.671 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-11-28 17:07:42.047635608 +0000 2023-11-28T17:07:52.671 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-11-28 17:05:00.603504956 +0000 2023-11-28T17:07:52.671 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-11-28 17:05:00.603504956 +0000 2023-11-28T17:07:52.671 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2023-11-28 17:05:00.603504956 +0000 2023-11-28T17:07:52.671 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T17:07:52.785 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-11-28T17:07:52.785 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-11-28T17:07:52.786 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000338424 s, 1.5 MB/s 2023-11-28T17:07:52.789 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T17:07:52.843 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2023-11-28T17:07:52.892 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T17:07:52.892 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:07:52.892 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 669 Links: 1 2023-11-28T17:07:52.892 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:07:52.892 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:07:52.892 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-11-28 17:07:42.048635592 +0000 2023-11-28T17:07:52.893 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-11-28 17:05:01.643487794 +0000 2023-11-28T17:07:52.893 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-11-28 17:05:01.643487794 +0000 2023-11-28T17:07:52.893 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2023-11-28 17:05:01.643487794 +0000 2023-11-28T17:07:52.893 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T17:07:52.978 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-11-28T17:07:52.978 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-11-28T17:07:52.978 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000397022 s, 1.3 MB/s 2023-11-28T17:07:52.981 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T17:07:53.032 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2023-11-28T17:07:53.072 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T17:07:53.072 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:07:53.072 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 691 Links: 1 2023-11-28T17:07:53.073 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:07:53.073 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:07:53.073 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-11-28 17:07:42.048635592 +0000 2023-11-28T17:07:53.073 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-11-28 17:05:02.660471010 +0000 2023-11-28T17:07:53.073 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-11-28 17:05:02.660471010 +0000 2023-11-28T17:07:53.073 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2023-11-28 17:05:02.660471010 +0000 2023-11-28T17:07:53.073 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T17:07:53.189 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-11-28T17:07:53.190 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-11-28T17:07:53.190 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000264266 s, 1.9 MB/s 2023-11-28T17:07:53.193 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T17:07:53.240 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T17:07:53.240 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:07:53.240 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T17:07:53.301 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T17:07:53.301 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2023-11-28T17:07:53.356 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T17:07:53.356 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:07:53.356 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 631 Links: 1 2023-11-28T17:07:53.356 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:07:53.357 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:07:53.357 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-28 17:07:30.719041470 +0000 2023-11-28T17:07:53.357 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-28 17:04:59.540910891 +0000 2023-11-28T17:07:53.357 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-28 17:04:59.540910891 +0000 2023-11-28T17:07:53.357 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2023-11-28 17:04:59.540910891 +0000 2023-11-28T17:07:53.357 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T17:07:53.448 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-28T17:07:53.448 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-28T17:07:53.449 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000252225 s, 2.0 MB/s 2023-11-28T17:07:53.452 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T17:07:53.509 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2023-11-28T17:07:53.560 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T17:07:53.560 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:07:53.560 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 653 Links: 1 2023-11-28T17:07:53.560 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:07:53.561 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:07:53.561 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-28 17:07:30.719041470 +0000 2023-11-28T17:07:53.561 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-28 17:05:00.530896569 +0000 2023-11-28T17:07:53.561 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-28 17:05:00.530896569 +0000 2023-11-28T17:07:53.561 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2023-11-28 17:05:00.530896569 +0000 2023-11-28T17:07:53.561 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T17:07:53.646 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-28T17:07:53.646 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-28T17:07:53.646 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000223875 s, 2.3 MB/s 2023-11-28T17:07:53.650 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T17:07:53.707 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2023-11-28T17:07:53.757 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T17:07:53.757 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:07:53.757 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 671 Links: 1 2023-11-28T17:07:53.757 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:07:53.758 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:07:53.758 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-28 17:07:30.720041456 +0000 2023-11-28T17:07:53.758 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-28 17:05:01.538881987 +0000 2023-11-28T17:07:53.758 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-28 17:05:01.538881987 +0000 2023-11-28T17:07:53.758 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2023-11-28 17:05:01.538881987 +0000 2023-11-28T17:07:53.758 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T17:07:53.843 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-28T17:07:53.843 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-28T17:07:53.843 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.00022206 s, 2.3 MB/s 2023-11-28T17:07:53.846 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T17:07:53.903 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2023-11-28T17:07:53.955 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T17:07:53.955 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T17:07:53.956 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 690 Links: 1 2023-11-28T17:07:53.956 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T17:07:53.956 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T17:07:53.956 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-11-28 17:07:30.721041441 +0000 2023-11-28T17:07:53.956 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-11-28 17:05:02.541867477 +0000 2023-11-28T17:07:53.956 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-11-28 17:05:02.541867477 +0000 2023-11-28T17:07:53.956 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2023-11-28 17:05:02.541867477 +0000 2023-11-28T17:07:53.956 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T17:07:54.044 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-11-28T17:07:54.045 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-11-28T17:07:54.045 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000233471 s, 2.2 MB/s 2023-11-28T17:07:54.048 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T17:07:54.106 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T17:07:54.106 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-28T17:07:54.106 INFO:tasks.ceph:Generating config... 2023-11-28T17:07:54.108 INFO:tasks.ceph:[client] client mount timeout = 600 2023-11-28T17:07:54.108 INFO:tasks.ceph:[client] debug client = 20 2023-11-28T17:07:54.108 INFO:tasks.ceph:[client] debug ms = 1 2023-11-28T17:07:54.108 INFO:tasks.ceph:[client] pid file = 2023-11-28T17:07:54.108 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-11-28T17:07:54.108 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-11-28T17:07:54.108 INFO:tasks.ceph:[mds] debug mds = 20 2023-11-28T17:07:54.108 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-11-28T17:07:54.108 INFO:tasks.ceph:[mds] debug ms = 1 2023-11-28T17:07:54.109 INFO:tasks.ceph:[mds] mds debug frag = True 2023-11-28T17:07:54.109 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-11-28T17:07:54.109 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-11-28T17:07:54.109 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-11-28T17:07:54.109 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-11-28T17:07:54.109 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-11-28T17:07:54.109 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-11-28T17:07:54.109 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-28T17:07:54.109 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-28T17:07:54.109 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-28T17:07:54.109 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-28T17:07:54.109 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-28T17:07:54.109 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-11-28T17:07:54.110 INFO:tasks.ceph:[osd] bdev async discard = True 2023-11-28T17:07:54.110 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-11-28T17:07:54.110 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-11-28T17:07:54.110 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-28T17:07:54.110 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-28T17:07:54.110 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-11-28T17:07:54.110 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-11-28T17:07:54.110 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-28T17:07:54.110 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-28T17:07:54.111 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-11-28T17:07:54.111 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-28T17:07:54.111 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-28T17:07:54.111 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-28T17:07:54.111 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-28T17:07:54.111 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-28T17:07:54.111 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-11-28T17:07:54.111 INFO:tasks.ceph:Setting up mon.a... 2023-11-28T17:07:54.111 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-28T17:07:54.238 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.keyring 2023-11-28T17:07:54.244 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-28T17:07:54.366 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T17:07:54.444 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.116'), ('mon.b', '172.21.15.156'), ('mon.c', '[v2:172.21.15.156:3301,v1:172.21.15.156:6790]')] 2023-11-28T17:07:54.445 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.116,172.21.15.156,[v2:172.21.15.156:3301,v1:172.21.15.156:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-11-28T17:07:54.446 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:07:54.446 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-28T17:07:54.488 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.116 --add b 172.21.15.156 --addv c '[v2:172.21.15.156:3301,v1:172.21.15.156:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:07:54.628 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:07:54.628 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: generated fsid a5f2fbcf-fde4-4c38-94da-08178784d9fc 2023-11-28T17:07:54.628 INFO:teuthology.orchestra.run.smithi116.stdout:setting min_mon_release = pacific 2023-11-28T17:07:54.628 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 0 2023-11-28T17:07:54.628 INFO:teuthology.orchestra.run.smithi116.stdout:fsid a5f2fbcf-fde4-4c38-94da-08178784d9fc 2023-11-28T17:07:54.628 INFO:teuthology.orchestra.run.smithi116.stdout:last_changed 2023-11-28T17:07:54.627632+0000 2023-11-28T17:07:54.628 INFO:teuthology.orchestra.run.smithi116.stdout:created 2023-11-28T17:07:54.627632+0000 2023-11-28T17:07:54.628 INFO:teuthology.orchestra.run.smithi116.stdout:min_mon_release 16 (pacific) 2023-11-28T17:07:54.628 INFO:teuthology.orchestra.run.smithi116.stdout:election_strategy: 1 2023-11-28T17:07:54.628 INFO:teuthology.orchestra.run.smithi116.stdout:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2023-11-28T17:07:54.629 INFO:teuthology.orchestra.run.smithi116.stdout:1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2023-11-28T17:07:54.629 INFO:teuthology.orchestra.run.smithi116.stdout:2: [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] mon.c 2023-11-28T17:07:54.629 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-28T17:07:54.631 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-28T17:07:54.672 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a5f2fbcf-fde4-4c38-94da-08178784d9fc... 2023-11-28T17:07:54.673 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-28T17:07:54.720 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-28T17:07:54.874 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2023-11-28T17:07:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2023-11-28T17:07:54.874 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-28T17:07:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2023-11-28T17:07:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2023-11-28T17:07:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2023-11-28T17:07:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2023-11-28T17:07:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2023-11-28T17:07:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2023-11-28T17:07:54.875 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2023-11-28T17:07:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-28T17:07:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2023-11-28T17:07:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2023-11-28T17:07:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-28T17:07:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2023-11-28T17:07:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2023-11-28T17:07:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2023-11-28T17:07:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2023-11-28T17:07:54.876 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2023-11-28T17:07:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2023-11-28T17:07:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2023-11-28T17:07:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2023-11-28T17:07:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-28T17:07:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-28T17:07:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2023-11-28T17:07:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-28T17:07:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2023-11-28T17:07:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2023-11-28T17:07:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2023-11-28T17:07:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2023-11-28T17:07:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2023-11-28T17:07:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "172.21.15.116,172.21.15.156,[v2:172.21.15.156:3301,v1:172.21.15.156:6790]" 2023-11-28T17:07:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = a5f2fbcf-fde4-4c38-94da-08178784d9fc 2023-11-28T17:07:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.878 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2023-11-28T17:07:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2023-11-28T17:07:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2023-11-28T17:07:54.878 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2023-11-28T17:07:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2023-11-28T17:07:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2023-11-28T17:07:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2023-11-28T17:07:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.879 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2023-11-28T17:07:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2023-11-28T17:07:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2023-11-28T17:07:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2023-11-28T17:07:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.880 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2023-11-28T17:07:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2023-11-28T17:07:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2023-11-28T17:07:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2023-11-28T17:07:54.881 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2023-11-28T17:07:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2023-11-28T17:07:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2023-11-28T17:07:54.882 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2023-11-28T17:07:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2023-11-28T17:07:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: bdev async discard = True 2023-11-28T17:07:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: bdev enable discard = True 2023-11-28T17:07:54.883 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore allocator = bitmap 2023-11-28T17:07:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2023-11-28T17:07:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2023-11-28T17:07:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 1/20 2023-11-28T17:07:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 1/20 2023-11-28T17:07:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-11-28T17:07:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2023-11-28T17:07:54.884 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 4/10 2023-11-28T17:07:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2023-11-28T17:07:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2023-11-28T17:07:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2023-11-28T17:07:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2023-11-28T17:07:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2023-11-28T17:07:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2023-11-28T17:07:54.885 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.886 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2023-11-28T17:07:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-11-28T17:07:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2023-11-28T17:07:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2023-11-28T17:07:54.886 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2023-11-28T17:07:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2023-11-28T17:07:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2023-11-28T17:07:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2023-11-28T17:07:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.887 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2023-11-28T17:07:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-11-28T17:07:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2023-11-28T17:07:54.887 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2023-11-28T17:07:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2023-11-28T17:07:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2023-11-28T17:07:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2023-11-28T17:07:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2023-11-28T17:07:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2023-11-28T17:07:54.888 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2023-11-28T17:07:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2023-11-28T17:07:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-28T17:07:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2023-11-28T17:07:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2023-11-28T17:07:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2023-11-28T17:07:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-28T17:07:54.889 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-28T17:07:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2023-11-28T17:07:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2023-11-28T17:07:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2023-11-28T17:07:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: mon op complaint time = 120 2023-11-28T17:07:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:07:54.890 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2023-11-28T17:07:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2023-11-28T17:07:54.890 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2023-11-28T17:07:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2023-11-28T17:07:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-28T17:07:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-28T17:07:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: client mount timeout = 600 2023-11-28T17:07:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: debug client = 20 2023-11-28T17:07:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-11-28T17:07:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = "" 2023-11-28T17:07:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2023-11-28T17:07:54.891 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2023-11-28T17:07:54.891 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2023-11-28T17:07:54.891 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2023-11-28T17:07:54.892 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2023-11-28T17:07:54.892 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2023-11-28T17:07:54.892 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2023-11-28T17:07:54.892 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2023-11-28T17:07:54.892 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2023-11-28T17:07:54.892 INFO:teuthology.orchestra.run.smithi116.stdout:[mds] 2023-11-28T17:07:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds = 20 2023-11-28T17:07:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds balancer = 20 2023-11-28T17:07:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-11-28T17:07:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug frag = True 2023-11-28T17:07:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug scatterstat = True 2023-11-28T17:07:54.892 INFO:teuthology.orchestra.run.smithi116.stdout: mds op complaint time = 180 2023-11-28T17:07:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: mds verify scatter = True 2023-11-28T17:07:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2023-11-28T17:07:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2023-11-28T17:07:54.893 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2023-11-28T17:07:54.901 INFO:teuthology.orchestra.run.smithi156.stdout:[global] 2023-11-28T17:07:54.901 INFO:teuthology.orchestra.run.smithi156.stdout: chdir = "" 2023-11-28T17:07:54.901 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-28T17:07:54.901 INFO:teuthology.orchestra.run.smithi156.stdout: auth supported = cephx 2023-11-28T17:07:54.902 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.902 INFO:teuthology.orchestra.run.smithi156.stdout: filestore xattr use omap = true 2023-11-28T17:07:54.902 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.902 INFO:teuthology.orchestra.run.smithi156.stdout: mon clock drift allowed = 1.000 2023-11-28T17:07:54.902 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.902 INFO:teuthology.orchestra.run.smithi156.stdout: osd crush chooseleaf type = 0 2023-11-28T17:07:54.902 INFO:teuthology.orchestra.run.smithi156.stdout: auth debug = true 2023-11-28T17:07:54.902 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.902 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on old message = true 2023-11-28T17:07:54.902 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on bug = true 2023-11-28T17:07:54.902 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.903 INFO:teuthology.orchestra.run.smithi156.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-28T17:07:54.903 INFO:teuthology.orchestra.run.smithi156.stdout: mon pg warn max object skew = 0 2023-11-28T17:07:54.903 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.903 INFO:teuthology.orchestra.run.smithi156.stdout: # disable pg_autoscaler by default for new pools 2023-11-28T17:07:54.903 INFO:teuthology.orchestra.run.smithi156.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-28T17:07:54.903 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.903 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default size = 2 2023-11-28T17:07:54.903 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.903 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow primary affinity = true 2023-11-28T17:07:54.904 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow pg remap = true 2023-11-28T17:07:54.904 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on legacy crush tunables = false 2023-11-28T17:07:54.904 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on crush straw calc version zero = false 2023-11-28T17:07:54.904 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on no sortbitwise = false 2023-11-28T17:07:54.904 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on osd down out interval zero = false 2023-11-28T17:07:54.904 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on too few osds = false 2023-11-28T17:07:54.904 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-28T17:07:54.905 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-28T17:07:54.905 INFO:teuthology.orchestra.run.smithi156.stdout: mon_allow_pool_size_one = true 2023-11-28T17:07:54.905 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.905 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-28T17:07:54.905 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.905 INFO:teuthology.orchestra.run.smithi156.stdout: osd default data pool replay window = 5 2023-11-28T17:07:54.905 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.906 INFO:teuthology.orchestra.run.smithi156.stdout: mon allow pool delete = true 2023-11-28T17:07:54.906 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.906 INFO:teuthology.orchestra.run.smithi156.stdout: mon cluster log file level = debug 2023-11-28T17:07:54.906 INFO:teuthology.orchestra.run.smithi156.stdout: debug asserts on shutdown = true 2023-11-28T17:07:54.906 INFO:teuthology.orchestra.run.smithi156.stdout: mon health detail to clog = false 2023-11-28T17:07:54.906 INFO:teuthology.orchestra.run.smithi156.stdout: mon host = "172.21.15.116,172.21.15.156,[v2:172.21.15.156:3301,v1:172.21.15.156:6790]" 2023-11-28T17:07:54.906 INFO:teuthology.orchestra.run.smithi156.stdout: fsid = a5f2fbcf-fde4-4c38-94da-08178784d9fc 2023-11-28T17:07:54.907 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.907 INFO:teuthology.orchestra.run.smithi156.stdout:[osd] 2023-11-28T17:07:54.907 INFO:teuthology.orchestra.run.smithi156.stdout: osd journal size = 100 2023-11-28T17:07:54.907 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.907 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub load threshold = 5.0 2023-11-28T17:07:54.907 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub max interval = 600 2023-11-28T17:07:54.908 INFO:teuthology.orchestra.run.smithi156.stdout: osd mclock profile = high_recovery_ops 2023-11-28T17:07:54.908 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.908 INFO:teuthology.orchestra.run.smithi156.stdout: osd recover clone overlap = true 2023-11-28T17:07:54.908 INFO:teuthology.orchestra.run.smithi156.stdout: osd recovery max chunk = 1048576 2023-11-28T17:07:54.908 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.908 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug shutdown = true 2023-11-28T17:07:54.909 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug op order = true 2023-11-28T17:07:54.909 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug verify stray on activate = true 2023-11-28T17:07:54.909 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug trim objects = true 2023-11-28T17:07:54.909 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.909 INFO:teuthology.orchestra.run.smithi156.stdout: osd open classes on start = true 2023-11-28T17:07:54.909 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug pg log writeout = true 2023-11-28T17:07:54.910 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.910 INFO:teuthology.orchestra.run.smithi156.stdout: osd deep scrub update digest min age = 30 2023-11-28T17:07:54.910 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.910 INFO:teuthology.orchestra.run.smithi156.stdout: osd map max advance = 10 2023-11-28T17:07:54.910 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.910 INFO:teuthology.orchestra.run.smithi156.stdout: journal zero on create = true 2023-11-28T17:07:54.910 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.911 INFO:teuthology.orchestra.run.smithi156.stdout: filestore ondisk finisher threads = 3 2023-11-28T17:07:54.911 INFO:teuthology.orchestra.run.smithi156.stdout: filestore apply finisher threads = 3 2023-11-28T17:07:54.911 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.911 INFO:teuthology.orchestra.run.smithi156.stdout: bdev debug aio = true 2023-11-28T17:07:54.911 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug misdirected ops = true 2023-11-28T17:07:54.912 INFO:teuthology.orchestra.run.smithi156.stdout: bdev async discard = True 2023-11-28T17:07:54.912 INFO:teuthology.orchestra.run.smithi156.stdout: bdev enable discard = True 2023-11-28T17:07:54.912 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore allocator = bitmap 2023-11-28T17:07:54.912 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore block size = 96636764160 2023-11-28T17:07:54.912 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore fsck on mount = True 2023-11-28T17:07:54.912 INFO:teuthology.orchestra.run.smithi156.stdout: debug bluefs = 1/20 2023-11-28T17:07:54.913 INFO:teuthology.orchestra.run.smithi156.stdout: debug bluestore = 1/20 2023-11-28T17:07:54.913 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2023-11-28T17:07:54.913 INFO:teuthology.orchestra.run.smithi156.stdout: debug osd = 20 2023-11-28T17:07:54.913 INFO:teuthology.orchestra.run.smithi156.stdout: debug rocksdb = 4/10 2023-11-28T17:07:54.913 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd backfillfull_ratio = 0.85 2023-11-28T17:07:54.913 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd full ratio = 0.9 2023-11-28T17:07:54.914 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd nearfull ratio = 0.8 2023-11-28T17:07:54.914 INFO:teuthology.orchestra.run.smithi156.stdout: osd failsafe full ratio = 0.95 2023-11-28T17:07:54.914 INFO:teuthology.orchestra.run.smithi156.stdout: osd objectstore = bluestore 2023-11-28T17:07:54.914 INFO:teuthology.orchestra.run.smithi156.stdout: osd op complaint time = 180 2023-11-28T17:07:54.914 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.914 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr] 2023-11-28T17:07:54.914 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2023-11-28T17:07:54.914 INFO:teuthology.orchestra.run.smithi156.stdout: debug mgr = 20 2023-11-28T17:07:54.914 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2023-11-28T17:07:54.914 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2023-11-28T17:07:54.915 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2023-11-28T17:07:54.915 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2023-11-28T17:07:54.915 INFO:teuthology.orchestra.run.smithi156.stdout: mgr/telemetry/nag = false 2023-11-28T17:07:54.915 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.915 INFO:teuthology.orchestra.run.smithi156.stdout:[mon] 2023-11-28T17:07:54.915 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2023-11-28T17:07:54.915 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2023-11-28T17:07:54.915 INFO:teuthology.orchestra.run.smithi156.stdout: debug paxos = 20 2023-11-28T17:07:54.915 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2023-11-28T17:07:54.915 INFO:teuthology.orchestra.run.smithi156.stdout: mon data avail warn = 5 2023-11-28T17:07:54.915 INFO:teuthology.orchestra.run.smithi156.stdout: mon mgr mkfs grace = 240 2023-11-28T17:07:54.916 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2023-11-28T17:07:54.916 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd reporter subtree level = osd 2023-11-28T17:07:54.916 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd prime pg temp = true 2023-11-28T17:07:54.916 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2023-11-28T17:07:54.916 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.916 INFO:teuthology.orchestra.run.smithi156.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-28T17:07:54.916 INFO:teuthology.orchestra.run.smithi156.stdout: auth mon ticket ttl = 660 # 11m 2023-11-28T17:07:54.916 INFO:teuthology.orchestra.run.smithi156.stdout: auth service ticket ttl = 240 # 4m 2023-11-28T17:07:54.916 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.916 INFO:teuthology.orchestra.run.smithi156.stdout: # don't complain about insecure global_id in the test suite 2023-11-28T17:07:54.916 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-28T17:07:54.917 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-28T17:07:54.917 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.917 INFO:teuthology.orchestra.run.smithi156.stdout: # 1m isn't quite enough 2023-11-28T17:07:54.917 INFO:teuthology.orchestra.run.smithi156.stdout: mon_down_mkfs_grace = 2m 2023-11-28T17:07:54.917 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.917 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_filestore_osds = false 2023-11-28T17:07:54.917 INFO:teuthology.orchestra.run.smithi156.stdout: mon op complaint time = 120 2023-11-28T17:07:54.917 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-11-28T17:07:54.917 INFO:teuthology.orchestra.run.smithi156.stdout:[client] 2023-11-28T17:07:54.917 INFO:teuthology.orchestra.run.smithi156.stdout: rgw cache enabled = true 2023-11-28T17:07:54.917 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable ops log = true 2023-11-28T17:07:54.917 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable usage log = true 2023-11-28T17:07:54.918 INFO:teuthology.orchestra.run.smithi156.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-28T17:07:54.918 INFO:teuthology.orchestra.run.smithi156.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-28T17:07:54.918 INFO:teuthology.orchestra.run.smithi156.stdout: client mount timeout = 600 2023-11-28T17:07:54.918 INFO:teuthology.orchestra.run.smithi156.stdout: debug client = 20 2023-11-28T17:07:54.918 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2023-11-28T17:07:54.918 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = "" 2023-11-28T17:07:54.918 INFO:teuthology.orchestra.run.smithi156.stdout: rados mon op timeout = 900 2023-11-28T17:07:54.918 INFO:teuthology.orchestra.run.smithi156.stdout: rados osd op timeout = 900 2023-11-28T17:07:54.918 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.a] 2023-11-28T17:07:54.919 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.b] 2023-11-28T17:07:54.919 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.c] 2023-11-28T17:07:54.919 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.a] 2023-11-28T17:07:54.919 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.c] 2023-11-28T17:07:54.919 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.b] 2023-11-28T17:07:54.919 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.d] 2023-11-28T17:07:54.919 INFO:teuthology.orchestra.run.smithi156.stdout:[mds] 2023-11-28T17:07:54.919 INFO:teuthology.orchestra.run.smithi156.stdout: debug mds = 20 2023-11-28T17:07:54.919 INFO:teuthology.orchestra.run.smithi156.stdout: debug mds balancer = 20 2023-11-28T17:07:54.919 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2023-11-28T17:07:54.920 INFO:teuthology.orchestra.run.smithi156.stdout: mds debug frag = True 2023-11-28T17:07:54.920 INFO:teuthology.orchestra.run.smithi156.stdout: mds debug scatterstat = True 2023-11-28T17:07:54.920 INFO:teuthology.orchestra.run.smithi156.stdout: mds op complaint time = 180 2023-11-28T17:07:54.920 INFO:teuthology.orchestra.run.smithi156.stdout: mds verify scatter = True 2023-11-28T17:07:54.920 INFO:teuthology.orchestra.run.smithi156.stdout: osd op complaint time = 180 2023-11-28T17:07:54.920 INFO:teuthology.orchestra.run.smithi156.stdout: rados mon op timeout = 900 2023-11-28T17:07:54.920 INFO:teuthology.orchestra.run.smithi156.stdout: rados osd op timeout = 900 2023-11-28T17:07:54.951 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-28T17:07:54.951 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-28T17:07:55.102 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-28T17:07:55.102 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:07:55.102 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-28T17:07:55.156 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:07:55.156 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-28T17:07:55.207 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:07:55.208 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:07:55.208 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-28T17:07:55.208 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T17:07:55.353 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:07:55.353 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:07:55.452 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:07:55.452 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:07:55.452 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-28T17:07:55.453 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-28T17:07:55.608 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:07:55.608 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:07:55.708 INFO:tasks.ceph:Setting up mon nodes... 2023-11-28T17:07:55.708 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-28T17:07:55.708 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-11-28T17:07:55.895 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-11-28T17:07:55.906 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-28T17:07:56.101 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-28T17:07:56.112 INFO:tasks.ceph:Setting up mds nodes... 2023-11-28T17:07:56.112 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-11-28T17:07:56.300 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-11-28T17:07:56.311 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-11-28T17:07:56.408 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-11-28T17:07:56.593 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-11-28T17:07:56.604 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-11-28T17:07:56.696 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-11-28T17:07:56.892 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-11-28T17:07:56.902 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-11-28T17:07:56.994 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-11-28T17:07:57.160 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-11-28T17:07:57.169 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-11-28T17:07:57.261 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-28T17:07:57.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-28T17:07:57.397 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-28T17:07:57.460 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-11-28T17:07:57.589 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-11-28T17:07:57.641 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-11-28T17:07:57.774 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-11-28T17:07:57.836 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-11-28T17:07:57.963 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-11-28T17:07:58.026 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-28T17:07:58.026 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-28T17:07:58.027 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-28T17:07:58.133 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T17:07:58.133 INFO:tasks.ceph:role: osd.0 2023-11-28T17:07:58.133 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:07:58.133 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-28T17:07:58.235 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:07:58.235 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:07:58.235 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:07:58.235 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:07:58.236 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:07:58.236 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:07:58.236 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:07:58.236 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:07:58.236 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:07:58.236 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:07:58.748 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-11-28T17:07:58.754 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-11-28T17:07:58.755 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-28T17:07:58.919 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-28T17:07:59.025 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-28T17:07:59.117 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T17:07:59.117 INFO:tasks.ceph:role: osd.1 2023-11-28T17:07:59.117 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:07:59.117 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-28T17:07:59.215 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:07:59.215 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:07:59.216 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:07:59.216 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:07:59.216 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:07:59.216 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:07:59.216 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:07:59.216 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:07:59.216 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:07:59.216 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:07:59.687 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-11-28T17:07:59.695 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-11-28T17:07:59.695 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-28T17:07:59.841 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-28T17:07:59.936 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-28T17:08:00.025 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T17:08:00.025 INFO:tasks.ceph:role: osd.2 2023-11-28T17:08:00.025 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:08:00.025 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-28T17:08:00.123 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:08:00.123 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:08:00.123 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:08:00.123 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:08:00.123 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:08:00.123 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:08:00.123 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:08:00.124 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:08:00.124 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:08:00.124 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:08:00.611 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-11-28T17:08:00.616 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-11-28T17:08:00.616 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-28T17:08:00.764 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-28T17:08:00.867 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-28T17:08:00.952 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-28T17:08:00.952 INFO:tasks.ceph:role: osd.3 2023-11-28T17:08:00.952 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:08:00.953 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-28T17:08:01.051 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:08:01.052 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:08:01.052 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:08:01.052 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:08:01.052 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:08:01.053 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:08:01.053 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:08:01.053 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:08:01.053 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:08:01.053 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:08:01.500 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-11-28T17:08:01.506 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-11-28T17:08:01.506 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-11-28T17:08:01.646 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-28T17:08:01.744 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:08:01.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:01.915+0000 7fcc27b20500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-28T17:08:01.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:01.919+0000 7fcc27b20500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-28T17:08:01.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:01.919+0000 7fcc27b20500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T17:08:01.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:01.919+0000 7fcc27b20500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T17:08:01.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:01.919+0000 7fcc27b20500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-28T17:08:01.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:01.926+0000 7fcc27b20500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-28T17:08:05.115 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-28T17:08:05.217 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:08:05.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:05.355+0000 7fbe9dc25500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-28T17:08:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:05.355+0000 7fbe9dc25500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-28T17:08:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:05.355+0000 7fbe9dc25500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T17:08:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:05.355+0000 7fbe9dc25500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T17:08:05.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:05.355+0000 7fbe9dc25500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-28T17:08:05.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:05.358+0000 7fbe9dc25500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-28T17:08:08.531 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-28T17:08:08.625 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:08:08.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:08.769+0000 7fc564ec2500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-28T17:08:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:08.769+0000 7fc564ec2500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-28T17:08:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:08.769+0000 7fc564ec2500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T17:08:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:08.769+0000 7fc564ec2500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T17:08:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:08.769+0000 7fc564ec2500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-28T17:08:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:08.772+0000 7fc564ec2500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-28T17:08:11.946 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-28T17:08:12.049 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:08:12.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:12.187+0000 7f3e8cc22500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-11-28T17:08:12.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:12.187+0000 7f3e8cc22500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-11-28T17:08:12.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:12.187+0000 7f3e8cc22500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T17:08:12.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:12.187+0000 7f3e8cc22500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T17:08:12.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:12.187+0000 7f3e8cc22500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-28T17:08:12.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:12.190+0000 7f3e8cc22500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-11-28T17:08:15.115 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-28T17:08:15.210 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-11-28T17:08:15.315 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T17:08:15.315 INFO:tasks.ceph:role: osd.4 2023-11-28T17:08:15.315 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:08:15.315 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-28T17:08:15.406 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:08:15.406 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:08:15.406 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:08:15.406 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:08:15.406 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:08:15.406 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:08:15.406 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:08:15.407 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:08:15.407 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:08:15.407 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:08:15.441 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2023-11-28T17:08:15.445 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2023-11-28T17:08:15.445 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-11-28T17:08:15.554 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-11-28T17:08:15.650 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-11-28T17:08:15.735 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T17:08:15.735 INFO:tasks.ceph:role: osd.5 2023-11-28T17:08:15.735 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:08:15.735 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-28T17:08:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:08:15.822 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:08:15.822 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:08:15.822 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:08:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:08:15.822 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:08:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:08:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:08:15.823 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:08:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:08:15.855 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2023-11-28T17:08:15.861 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2023-11-28T17:08:15.861 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-11-28T17:08:15.968 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-11-28T17:08:16.062 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-11-28T17:08:16.154 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T17:08:16.154 INFO:tasks.ceph:role: osd.6 2023-11-28T17:08:16.154 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:08:16.154 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-28T17:08:16.248 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:08:16.248 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:08:16.248 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:08:16.248 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:08:16.248 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:08:16.249 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:08:16.249 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:08:16.249 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:08:16.249 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:08:16.249 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:08:16.282 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2023-11-28T17:08:16.288 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2023-11-28T17:08:16.288 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-11-28T17:08:16.401 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-11-28T17:08:16.486 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-11-28T17:08:16.570 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-28T17:08:16.571 INFO:tasks.ceph:role: osd.7 2023-11-28T17:08:16.571 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:08:16.571 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-28T17:08:16.661 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-28T17:08:16.661 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-28T17:08:16.661 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-28T17:08:16.662 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-28T17:08:16.662 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-28T17:08:16.662 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2023-11-28T17:08:16.662 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-28T17:08:16.662 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-28T17:08:16.662 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-28T17:08:16.662 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-28T17:08:16.696 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2023-11-28T17:08:16.701 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2023-11-28T17:08:16.702 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-11-28T17:08:16.823 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-11-28T17:08:16.908 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:08:17.053 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:17.052+0000 7fd474f1a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-11-28T17:08:17.054 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:17.052+0000 7fd474f1a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-11-28T17:08:17.054 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:17.052+0000 7fd474f1a500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T17:08:17.054 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:17.052+0000 7fd474f1a500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T17:08:17.054 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:17.052+0000 7fd474f1a500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-28T17:08:17.056 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:17.055+0000 7fd474f1a500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-11-28T17:08:20.234 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-11-28T17:08:20.334 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:08:20.470 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:20.469+0000 7f436abad500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-11-28T17:08:20.471 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:20.470+0000 7f436abad500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-11-28T17:08:20.471 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:20.470+0000 7f436abad500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T17:08:20.471 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:20.470+0000 7f436abad500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T17:08:20.471 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:20.470+0000 7f436abad500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-28T17:08:20.473 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:20.472+0000 7f436abad500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-11-28T17:08:22.097 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:08:22.107 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:08:23.391 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-11-28T17:08:23.489 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:08:23.618 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:23.617+0000 7f9b567be500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-11-28T17:08:23.619 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:23.618+0000 7f9b567be500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-11-28T17:08:23.619 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:23.618+0000 7f9b567be500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T17:08:23.619 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:23.618+0000 7f9b567be500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T17:08:23.619 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:23.618+0000 7f9b567be500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-28T17:08:23.621 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:23.620+0000 7f9b567be500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-11-28T17:08:26.522 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-11-28T17:08:26.625 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:08:26.766 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:26.765+0000 7f70eb025500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-11-28T17:08:26.767 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:26.766+0000 7f70eb025500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-11-28T17:08:26.767 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:26.766+0000 7f70eb025500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T17:08:26.767 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:26.766+0000 7f70eb025500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T17:08:26.768 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:26.766+0000 7f70eb025500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-28T17:08:26.769 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:26.768+0000 7f70eb025500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-11-28T17:08:29.948 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-11-28T17:08:30.050 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-28T17:08:30.050 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:08:30.050 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-11-28T17:08:30.155 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:08:30.155 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-11-28T17:08:30.249 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:08:30.249 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-11-28T17:08:30.345 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:08:30.345 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-28T17:08:30.441 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:08:30.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-28T17:08:30.527 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:08:30.527 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-28T17:08:30.655 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:08:30.655 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-28T17:08:30.739 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:08:30.740 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-28T17:08:30.843 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:08:30.843 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-11-28T17:08:30.936 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:08:30.937 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-11-28T17:08:31.076 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:08:31.077 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-11-28T17:08:31.172 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:08:31.172 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-11-28T17:08:31.265 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:08:31.266 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-11-28T17:08:31.404 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:08:31.405 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-11-28T17:08:31.496 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:08:31.496 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-28T17:08:31.549 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:08:31.549 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-11-28T17:08:31.646 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:08:31.647 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-11-28T17:08:31.707 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:08:31.708 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-11-28T17:08:31.765 INFO:tasks.ceph:Adding keys to all mons... 2023-11-28T17:08:31.765 DEBUG:teuthology.orchestra.run.smithi116:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-28T17:08:31.775 DEBUG:teuthology.orchestra.run.smithi156:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-28T17:08:31.851 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.y] 2023-11-28T17:08:31.852 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBrHmZl159eNRAAqk03SlXduu04+ZNQe8YLUg== 2023-11-28T17:08:31.852 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2023-11-28T17:08:31.852 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBsHmZlDk7pERAAzfB2wxucZ6BlKfc/eBL5fA== 2023-11-28T17:08:31.852 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2023-11-28T17:08:31.852 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBsHmZlvVteIxAAqG8H+6/g2owr7mI87oYHgA== 2023-11-28T17:08:31.852 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2023-11-28T17:08:31.852 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBxHmZlUqmdNhAAc7C+EUtBtgCaLUkr0PPrHg== 2023-11-28T17:08:31.852 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2023-11-28T17:08:31.852 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQB1HmZlKQ07FRAAKncDe3f+D1TfLU8lhQVaGA== 2023-11-28T17:08:31.852 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2023-11-28T17:08:31.852 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQB4HmZlvpDnLRAAG2kEjnvjJmln7TCBM9hUTw== 2023-11-28T17:08:31.852 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.3] 2023-11-28T17:08:31.852 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQB8HmZlGD45CxAA2LESF1EBeNjPX6fjvSyEFA== 2023-11-28T17:08:31.853 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.x] 2023-11-28T17:08:31.853 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBsHmZlRWsIBhAAmDx975LSXMvBGPQf/P3etQ== 2023-11-28T17:08:31.853 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2023-11-28T17:08:31.853 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBsHmZln5I1NRAAK30dyAqRuGLeJ4mGw90Rpg== 2023-11-28T17:08:31.853 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2023-11-28T17:08:31.853 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBtHmZl6ieLCRAAVEyizVZ3mGTtPGRQeee11A== 2023-11-28T17:08:31.853 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.4] 2023-11-28T17:08:31.853 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCBHmZlJOgqAxAAc23riVIAOTXw28t8OrTjQg== 2023-11-28T17:08:31.853 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.5] 2023-11-28T17:08:31.853 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCEHmZlvTMIHBAAuVkPBcYAAuTQUVqdkZ5liw== 2023-11-28T17:08:31.853 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.6] 2023-11-28T17:08:31.853 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCHHmZls5zfJBAAmH97PMKvbQXpeznCSkt6IQ== 2023-11-28T17:08:31.854 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.7] 2023-11-28T17:08:31.854 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCKHmZlahyvLRAADZqskufX275VqtUDlUHU7Q== 2023-11-28T17:08:31.854 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2023-11-28T17:08:31.854 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBtHmZlPRewFxAAfLJPZrTljlPIoghT5ZzoBA== 2023-11-28T17:08:31.854 INFO:teuthology.orchestra.run.smithi116.stdout:[client.2] 2023-11-28T17:08:31.854 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBtHmZleq8ZIxAAOZsBs+OlL6TXLGqwtX11pw== 2023-11-28T17:08:31.854 INFO:teuthology.orchestra.run.smithi116.stdout:[client.1] 2023-11-28T17:08:31.854 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBtHmZlWG0nLhAA4xwi4Ut4dfJWwY275L/mHQ== 2023-11-28T17:08:31.854 INFO:teuthology.orchestra.run.smithi116.stdout:[client.3] 2023-11-28T17:08:31.854 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBtHmZlkoZmORAAPWGkPoFfqsaCfAjnWmfqtw== 2023-11-28T17:08:31.870 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr.y] 2023-11-28T17:08:31.870 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBrHmZl159eNRAAqk03SlXduu04+ZNQe8YLUg== 2023-11-28T17:08:31.870 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.a] 2023-11-28T17:08:31.870 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBsHmZlDk7pERAAzfB2wxucZ6BlKfc/eBL5fA== 2023-11-28T17:08:31.871 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.c] 2023-11-28T17:08:31.871 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBsHmZlvVteIxAAqG8H+6/g2owr7mI87oYHgA== 2023-11-28T17:08:31.871 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.0] 2023-11-28T17:08:31.871 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBxHmZlUqmdNhAAc7C+EUtBtgCaLUkr0PPrHg== 2023-11-28T17:08:31.871 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.1] 2023-11-28T17:08:31.872 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQB1HmZlKQ07FRAAKncDe3f+D1TfLU8lhQVaGA== 2023-11-28T17:08:31.872 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.2] 2023-11-28T17:08:31.872 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQB4HmZlvpDnLRAAG2kEjnvjJmln7TCBM9hUTw== 2023-11-28T17:08:31.872 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.3] 2023-11-28T17:08:31.872 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQB8HmZlGD45CxAA2LESF1EBeNjPX6fjvSyEFA== 2023-11-28T17:08:31.872 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr.x] 2023-11-28T17:08:31.873 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBsHmZlRWsIBhAAmDx975LSXMvBGPQf/P3etQ== 2023-11-28T17:08:31.873 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.b] 2023-11-28T17:08:31.873 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBsHmZln5I1NRAAK30dyAqRuGLeJ4mGw90Rpg== 2023-11-28T17:08:31.873 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.d] 2023-11-28T17:08:31.873 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBtHmZl6ieLCRAAVEyizVZ3mGTtPGRQeee11A== 2023-11-28T17:08:31.873 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.4] 2023-11-28T17:08:31.874 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCBHmZlJOgqAxAAc23riVIAOTXw28t8OrTjQg== 2023-11-28T17:08:31.874 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.5] 2023-11-28T17:08:31.874 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCEHmZlvTMIHBAAuVkPBcYAAuTQUVqdkZ5liw== 2023-11-28T17:08:31.874 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.6] 2023-11-28T17:08:31.874 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCHHmZls5zfJBAAmH97PMKvbQXpeznCSkt6IQ== 2023-11-28T17:08:31.874 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.7] 2023-11-28T17:08:31.875 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCKHmZlahyvLRAADZqskufX275VqtUDlUHU7Q== 2023-11-28T17:08:31.875 INFO:teuthology.orchestra.run.smithi156.stdout:[client.0] 2023-11-28T17:08:31.875 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBtHmZlPRewFxAAfLJPZrTljlPIoghT5ZzoBA== 2023-11-28T17:08:31.875 INFO:teuthology.orchestra.run.smithi156.stdout:[client.2] 2023-11-28T17:08:31.875 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBtHmZleq8ZIxAAOZsBs+OlL6TXLGqwtX11pw== 2023-11-28T17:08:31.875 INFO:teuthology.orchestra.run.smithi156.stdout:[client.1] 2023-11-28T17:08:31.876 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBtHmZlWG0nLhAA4xwi4Ut4dfJWwY275L/mHQ== 2023-11-28T17:08:31.876 INFO:teuthology.orchestra.run.smithi156.stdout:[client.3] 2023-11-28T17:08:31.876 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBtHmZlkoZmORAAPWGkPoFfqsaCfAjnWmfqtw== 2023-11-28T17:08:31.877 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T17:08:31.905 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T17:08:32.054 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:08:32.064 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:08:32.214 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:08:32.223 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:08:32.369 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:32.378 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:32.523 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:32.532 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:32.674 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:32.683 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:32.821 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:32.826 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:32.973 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T17:08:32.981 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-28T17:08:33.125 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:08:33.134 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:08:33.284 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:08:33.292 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-28T17:08:33.437 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:33.445 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:33.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:33.599 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:33.739 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:33.746 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:33.883 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:33.893 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-28T17:08:34.042 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T17:08:34.051 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T17:08:34.205 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T17:08:34.213 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T17:08:34.363 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T17:08:34.370 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T17:08:34.518 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T17:08:34.527 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-28T17:08:34.668 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-28T17:08:34.669 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-28T17:08:34.763 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T17:08:34.922 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-28T17:08:35.009 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-28T17:08:35.114 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T17:08:35.269 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-28T17:08:35.362 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-28T17:08:35.459 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-28T17:08:35.614 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-28T17:08:35.709 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:08:35.718 DEBUG:teuthology.orchestra.run.smithi156:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-28T17:08:35.772 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-28T17:08:35.772 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-28T17:08:35.772 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-28T17:08:35.778 INFO:tasks.ceph.mon.a:Started 2023-11-28T17:08:35.778 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-28T17:08:35.779 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-28T17:08:35.785 INFO:tasks.ceph.mon.b:Started 2023-11-28T17:08:35.785 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-28T17:08:35.785 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-28T17:08:35.790 INFO:tasks.ceph.mon.c:Started 2023-11-28T17:08:35.790 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-28T17:08:35.791 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-11-28T17:08:35.791 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-11-28T17:08:35.796 INFO:tasks.ceph.mgr.y:Started 2023-11-28T17:08:35.796 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-28T17:08:35.796 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-28T17:08:35.802 INFO:tasks.ceph.mgr.x:Started 2023-11-28T17:08:35.802 INFO:tasks.ceph:Setting crush tunables to default 2023-11-28T17:08:35.802 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd crush tunables default 2023-11-28T17:08:35.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.985+0000 7fa15a67c640 1 Processor -- start 2023-11-28T17:08:35.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.985+0000 7fa15a67c640 1 -- start start 2023-11-28T17:08:35.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.986+0000 7fa15a67c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540d1780 0x7fa1540d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:35.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.986+0000 7fa15a67c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa1540b20a0 0x7fa1540d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:35.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.986+0000 7fa15a67c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa1540b2ad0 0x7fa1540ce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:35.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.986+0000 7fa15a67c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1540cdcb0 con 0x7fa1540b2ad0 2023-11-28T17:08:35.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.986+0000 7fa15a67c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1540d20a0 con 0x7fa1540b20a0 2023-11-28T17:08:35.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.986+0000 7fa15a67c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1540d39e0 con 0x7fa1540d1780 2023-11-28T17:08:35.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.986+0000 7fa158c19640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa1540b2ad0 0x7fa1540ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:35.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.986+0000 7fa158c19640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa1540b2ad0 0x7fa1540ce750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35594/0 (socket says 172.21.15.116:35594) 2023-11-28T17:08:35.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.986+0000 7fa158c19640 1 -- 172.21.15.116:0/335835827 learned_addr learned my addr 172.21.15.116:0/335835827 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:35.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.986+0000 7fa1537fe640 1 --2- 172.21.15.116:0/335835827 >> v2:172.21.15.156:3300/0 conn(0x7fa1540b20a0 0x7fa1540d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:35.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.986+0000 7fa158c19640 1 -- 172.21.15.116:0/335835827 >> v2:172.21.15.116:3300/0 conn(0x7fa1540b2ad0 msgr2=0x7fa1540ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:08:35.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.986+0000 7fa158c19640 1 -- 172.21.15.116:0/335835827 >> v2:172.21.15.116:3300/0 conn(0x7fa1540b2ad0 msgr2=0x7fa1540ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:08:35.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.986+0000 7fa158c19640 1 --2- 172.21.15.116:0/335835827 >> v2:172.21.15.116:3300/0 conn(0x7fa1540b2ad0 0x7fa1540ce750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:08:35.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.987+0000 7fa1537fe640 1 -- 172.21.15.116:0/335835827 >> v2:172.21.15.156:3300/0 conn(0x7fa1540b20a0 msgr2=0x7fa1540d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:08:35.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.987+0000 7fa1537fe640 1 -- 172.21.15.116:0/335835827 >> v2:172.21.15.156:3300/0 conn(0x7fa1540b20a0 msgr2=0x7fa1540d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:08:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.987+0000 7fa1537fe640 1 --2- 172.21.15.116:0/335835827 >> v2:172.21.15.156:3300/0 conn(0x7fa1540b20a0 0x7fa1540d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:08:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.987+0000 7fa158c19640 1 --2- 172.21.15.116:0/335835827 >> v2:172.21.15.116:3300/0 conn(0x7fa1540b2ad0 0x7fa1540ce750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:08:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.987+0000 7fa1537fe640 1 --2- 172.21.15.116:0/335835827 >> v2:172.21.15.156:3300/0 conn(0x7fa1540b20a0 0x7fa1540d1b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:08:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.987+0000 7fa153fff640 1 --2- 172.21.15.116:0/335835827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540d1780 0x7fa1540d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.987+0000 7fa153fff640 1 -- 172.21.15.116:0/335835827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540d1780 msgr2=0x7fa1540d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:08:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.987+0000 7fa153fff640 1 -- 172.21.15.116:0/335835827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540d1780 msgr2=0x7fa1540d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:08:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.987+0000 7fa153fff640 1 --2- 172.21.15.116:0/335835827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540d1780 0x7fa1540d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:08:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:35.987+0000 7fa153fff640 1 --2- 172.21.15.116:0/335835827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540d1780 0x7fa1540d01e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:08:36.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.188+0000 7fa158c19640 1 --2- 172.21.15.116:0/335835827 >> v2:172.21.15.116:3300/0 conn(0x7fa1540b2ad0 0x7fa1540ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:36.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.188+0000 7fa158c19640 1 -- 172.21.15.116:0/335835827 >> v2:172.21.15.116:3300/0 conn(0x7fa1540b2ad0 msgr2=0x7fa1540ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-28T17:08:36.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.188+0000 7fa158c19640 1 -- 172.21.15.116:0/335835827 >> v2:172.21.15.116:3300/0 conn(0x7fa1540b2ad0 msgr2=0x7fa1540ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:08:36.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.188+0000 7fa158c19640 1 --2- 172.21.15.116:0/335835827 >> v2:172.21.15.116:3300/0 conn(0x7fa1540b2ad0 0x7fa1540ce750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:08:36.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.188+0000 7fa158c19640 1 --2- 172.21.15.116:0/335835827 >> v2:172.21.15.116:3300/0 conn(0x7fa1540b2ad0 0x7fa1540ce750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:08:36.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.188+0000 7fa153fff640 1 --2- 172.21.15.116:0/335835827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540d1780 0x7fa1540d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:36.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.189+0000 7fa1537fe640 1 --2- 172.21.15.116:0/335835827 >> v2:172.21.15.156:3300/0 conn(0x7fa1540b20a0 0x7fa1540d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:36.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.190+0000 7fa153fff640 1 -- 172.21.15.116:0/335835827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540d1780 msgr2=0x7fa1540d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-11-28T17:08:36.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.190+0000 7fa153fff640 1 -- 172.21.15.116:0/335835827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540d1780 msgr2=0x7fa1540d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:08:36.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.190+0000 7fa153fff640 1 --2- 172.21.15.116:0/335835827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540d1780 0x7fa1540d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:08:36.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.190+0000 7fa153fff640 1 --2- 172.21.15.116:0/335835827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540d1780 0x7fa1540d01e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:08:36.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.190+0000 7fa1537fe640 1 -- 172.21.15.116:0/335835827 >> v2:172.21.15.156:3300/0 conn(0x7fa1540b20a0 msgr2=0x7fa1540d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-28T17:08:36.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.190+0000 7fa1537fe640 1 -- 172.21.15.116:0/335835827 >> v2:172.21.15.156:3300/0 conn(0x7fa1540b20a0 msgr2=0x7fa1540d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-28T17:08:36.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.190+0000 7fa1537fe640 1 --2- 172.21.15.116:0/335835827 >> v2:172.21.15.156:3300/0 conn(0x7fa1540b20a0 0x7fa1540d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:08:36.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.190+0000 7fa1537fe640 1 --2- 172.21.15.116:0/335835827 >> v2:172.21.15.156:3300/0 conn(0x7fa1540b20a0 0x7fa1540d1b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:08:36.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.589+0000 7fa158c19640 1 --2- 172.21.15.116:0/335835827 >> v2:172.21.15.116:3300/0 conn(0x7fa1540b2ad0 0x7fa1540ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:36.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.590+0000 7fa158c19640 1 -- 172.21.15.116:0/335835827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540d1780 msgr2=0x7fa1540d01e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:08:36.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.590+0000 7fa158c19640 1 --2- 172.21.15.116:0/335835827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540d1780 0x7fa1540d01e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:36.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.590+0000 7fa158c19640 1 -- 172.21.15.116:0/335835827 >> v2:172.21.15.156:3300/0 conn(0x7fa1540b20a0 msgr2=0x7fa1540d1b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:08:36.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.590+0000 7fa158c19640 1 --2- 172.21.15.116:0/335835827 >> v2:172.21.15.156:3300/0 conn(0x7fa1540b20a0 0x7fa1540d1b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:36.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.590+0000 7fa158c19640 1 -- 172.21.15.116:0/335835827 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1540d3cc0 con 0x7fa1540b2ad0 2023-11-28T17:08:36.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.593+0000 7fa158c19640 1 --2- 172.21.15.116:0/335835827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540b2ad0 0x7fa1540ce750 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa15408d170 tx=0x7fa14c02c1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff22edd19fcf0c8c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:36.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.593+0000 7fa152ffd640 1 -- 172.21.15.116:0/335835827 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c036400 con 0x7fa1540b2ad0 2023-11-28T17:08:36.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.594+0000 7fa152ffd640 1 -- 172.21.15.116:0/335835827 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14c030070 con 0x7fa1540b2ad0 2023-11-28T17:08:36.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.594+0000 7fa152ffd640 1 -- 172.21.15.116:0/335835827 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c036800 con 0x7fa1540b2ad0 2023-11-28T17:08:36.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.595+0000 7fa15a67c640 1 -- 172.21.15.116:0/335835827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540b2ad0 msgr2=0x7fa1540ce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:36.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.595+0000 7fa15a67c640 1 --2- 172.21.15.116:0/335835827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540b2ad0 0x7fa1540ce750 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa15408d170 tx=0x7fa14c02c1b0 comp rx=0 tx=0).stop 2023-11-28T17:08:36.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.595+0000 7fa15a67c640 1 -- 172.21.15.116:0/335835827 shutdown_connections 2023-11-28T17:08:36.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.595+0000 7fa15a67c640 1 --2- 172.21.15.116:0/335835827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540b2ad0 0x7fa1540ce750 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:36.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.595+0000 7fa15a67c640 1 --2- 172.21.15.116:0/335835827 >> v2:172.21.15.156:3300/0 conn(0x7fa1540b20a0 0x7fa1540d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:36.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.595+0000 7fa15a67c640 1 --2- 172.21.15.116:0/335835827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540d1780 0x7fa1540d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:36.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.595+0000 7fa15a67c640 1 -- 172.21.15.116:0/335835827 >> 172.21.15.116:0/335835827 conn(0x7fa15409c4e0 msgr2=0x7fa1540d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:36.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.596+0000 7fa15a67c640 1 -- 172.21.15.116:0/335835827 shutdown_connections 2023-11-28T17:08:36.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.596+0000 7fa15a67c640 1 -- 172.21.15.116:0/335835827 wait complete. 2023-11-28T17:08:36.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.598+0000 7fa15a67c640 1 Processor -- start 2023-11-28T17:08:36.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.598+0000 7fa15a67c640 1 -- start start 2023-11-28T17:08:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa15a67c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540b20a0 0x7fa15415ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa15a67c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540b2ad0 0x7fa1540ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:36.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa15a67c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1540d1780 0x7fa15414c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa15a67c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1541602f0 con 0x7fa1540b2ad0 2023-11-28T17:08:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa15a67c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1540c9bc0 con 0x7fa1540d1780 2023-11-28T17:08:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa15a67c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa154160710 con 0x7fa1540b20a0 2023-11-28T17:08:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa1537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540b2ad0 0x7fa1540ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa1537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540b2ad0 0x7fa1540ca4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35634/0 (socket says 172.21.15.116:35634) 2023-11-28T17:08:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa1537fe640 1 -- 172.21.15.116:0/3145038211 learned_addr learned my addr 172.21.15.116:0/3145038211 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:36.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa158c19640 1 --2- 172.21.15.116:0/3145038211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1540d1780 0x7fa15414c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:36.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa153fff640 1 --2- 172.21.15.116:0/3145038211 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540b20a0 0x7fa15415ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:36.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa1537fe640 1 -- 172.21.15.116:0/3145038211 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540b20a0 msgr2=0x7fa15415ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:36.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa1537fe640 1 --2- 172.21.15.116:0/3145038211 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540b20a0 0x7fa15415ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa1537fe640 1 -- 172.21.15.116:0/3145038211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1540d1780 msgr2=0x7fa15414c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa1537fe640 1 --2- 172.21.15.116:0/3145038211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1540d1780 0x7fa15414c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.600+0000 7fa1537fe640 1 -- 172.21.15.116:0/3145038211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c0097e0 con 0x7fa1540b2ad0 2023-11-28T17:08:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.601+0000 7fa1537fe640 1 --2- 172.21.15.116:0/3145038211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540b2ad0 0x7fa1540ca4e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa14800f4e0 tx=0x7fa14800f9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.601+0000 7fa150ff9640 1 -- 172.21.15.116:0/3145038211 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148014670 con 0x7fa1540b2ad0 2023-11-28T17:08:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.601+0000 7fa150ff9640 1 -- 172.21.15.116:0/3145038211 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1480147f0 con 0x7fa1540b2ad0 2023-11-28T17:08:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.601+0000 7fa150ff9640 1 -- 172.21.15.116:0/3145038211 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa148014ba0 con 0x7fa1540b2ad0 2023-11-28T17:08:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.601+0000 7fa15a67c640 1 -- 172.21.15.116:0/3145038211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15415f670 con 0x7fa1540b2ad0 2023-11-28T17:08:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.601+0000 7fa15a67c640 1 -- 172.21.15.116:0/3145038211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15415fc30 con 0x7fa1540b2ad0 2023-11-28T17:08:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.602+0000 7fa150ff9640 1 -- 172.21.15.116:0/3145038211 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa14801e410 con 0x7fa1540b2ad0 2023-11-28T17:08:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.602+0000 7fa150ff9640 1 -- 172.21.15.116:0/3145038211 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa1480159b0 con 0x7fa1540b2ad0 2023-11-28T17:08:36.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.602+0000 7fa15a67c640 1 -- 172.21.15.116:0/3145038211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa154160430 con 0x7fa1540b2ad0 2023-11-28T17:08:36.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.606+0000 7fa150ff9640 1 -- 172.21.15.116:0/3145038211 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fa154160430 con 0x7fa1540b2ad0 2023-11-28T17:08:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.677+0000 7fa15a67c640 1 -- 172.21.15.116:0/3145038211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fa154069360 con 0x7fa1540b2ad0 2023-11-28T17:08:36.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.680+0000 7fa150ff9640 1 -- 172.21.15.116:0/3145038211 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa148019030 con 0x7fa1540b2ad0 2023-11-28T17:08:36.682 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2023-11-28T17:08:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.682+0000 7fa15a67c640 1 -- 172.21.15.116:0/3145038211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540b2ad0 msgr2=0x7fa1540ca4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.682+0000 7fa15a67c640 1 --2- 172.21.15.116:0/3145038211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540b2ad0 0x7fa1540ca4e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa14800f4e0 tx=0x7fa14800f9a0 comp rx=0 tx=0).stop 2023-11-28T17:08:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.682+0000 7fa15a67c640 1 -- 172.21.15.116:0/3145038211 shutdown_connections 2023-11-28T17:08:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.682+0000 7fa15a67c640 1 --2- 172.21.15.116:0/3145038211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1540d1780 0x7fa15414c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.682+0000 7fa15a67c640 1 --2- 172.21.15.116:0/3145038211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1540b2ad0 0x7fa1540ca4e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.682+0000 7fa15a67c640 1 --2- 172.21.15.116:0/3145038211 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1540b20a0 0x7fa15415ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.682+0000 7fa15a67c640 1 -- 172.21.15.116:0/3145038211 >> 172.21.15.116:0/3145038211 conn(0x7fa15409c4e0 msgr2=0x7fa1540ce5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.682+0000 7fa15a67c640 1 -- 172.21.15.116:0/3145038211 shutdown_connections 2023-11-28T17:08:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:36.682+0000 7fa15a67c640 1 -- 172.21.15.116:0/3145038211 wait complete. 2023-11-28T17:08:36.700 INFO:tasks.ceph:check_enable_crimson: False 2023-11-28T17:08:36.700 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-28T17:08:36.700 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:08:36.700 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-28T17:08:36.786 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:08:36.786 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-28T17:08:36.840 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:36.839+0000 7fb78c637040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T17:08:36.906 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:36.905+0000 7fb78c637040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T17:08:36.917 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:08:36.917 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-28T17:08:37.038 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:08:37.038 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-28T17:08:37.160 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:37.159+0000 7ff96c02e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T17:08:37.162 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:08:37.162 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-11-28T17:08:37.227 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:37.226+0000 7ff96c02e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T17:08:37.252 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:08:37.253 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-11-28T17:08:37.331 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:08:37.332 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-11-28T17:08:37.451 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:08:37.451 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-11-28T17:08:37.531 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:37.529+0000 7fb78c637040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T17:08:37.581 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 6991e794-5137-4365-be74-be350afa42e3 0 2023-11-28T17:08:37.591 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:37.590+0000 7fb78c637040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T17:08:37.641 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:37.640+0000 7fb78c637040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T17:08:37.711 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:37.710+0000 7fb78c637040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T17:08:37.760 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:37.759+0000 7fb78c637040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T17:08:37.762 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.761+0000 7ff2625cb640 1 Processor -- start 2023-11-28T17:08:37.762 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.762+0000 7ff2625cb640 1 -- start start 2023-11-28T17:08:37.764 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.763+0000 7ff2625cb640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff25c160b80 0x7ff25c0d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:37.764 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.763+0000 7ff2625cb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff25c0b4010 0x7ff25c0d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:37.764 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.763+0000 7ff2625cb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c0d0da0 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.765 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.763+0000 7ff2625cb640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25c160f60 con 0x7ff25c160b80 2023-11-28T17:08:37.765 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.763+0000 7ff2625cb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0d1080 con 0x7ff25c0b4010 2023-11-28T17:08:37.765 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.763+0000 7ff261dca640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7ff25c0cf6f0 0x7ff25c0cfb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:46420/0 (socket says 172.21.15.156:46420) 2023-11-28T17:08:37.765 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.763+0000 7ff261dca640 1 -- 172.21.15.156:0/3727894927 learned_addr learned my addr 172.21.15.156:0/3727894927 (peer_addr_for_me v1:172.21.15.156:0/0) 2023-11-28T17:08:37.765 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.763+0000 7ff260dc8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff25c0b4010 0x7ff25c0d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:37.765 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.763+0000 7ff2615c9640 1 --2- 172.21.15.156:0/3727894927 >> v2:172.21.15.116:3300/0 conn(0x7ff25c160b80 0x7ff25c0d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:37.767 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.763+0000 7ff260dc8640 1 -- 172.21.15.156:0/3727894927 >> v2:172.21.15.116:3300/0 conn(0x7ff25c160b80 msgr2=0x7ff25c0d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:37.767 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.763+0000 7ff260dc8640 1 --2- 172.21.15.156:0/3727894927 >> v2:172.21.15.116:3300/0 conn(0x7ff25c160b80 0x7ff25c0d1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:37.768 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.763+0000 7ff253fff640 1 -- 172.21.15.156:0/3727894927 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570116758 0 0) 0x7ff25c0d0da0 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.768 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.763+0000 7ff260dc8640 1 -- 172.21.15.156:0/3727894927 >> v1:172.21.15.156:6789/0 conn(0x7ff25c0cf6f0 legacy=0x7ff25c0cfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:37.768 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.763+0000 7ff260dc8640 1 -- 172.21.15.156:0/3727894927 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c0ce780 con 0x7ff25c0b4010 2023-11-28T17:08:37.768 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.765+0000 7ff260dc8640 1 --2- 172.21.15.156:0/3727894927 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff25c0b4010 0x7ff25c0d18a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0097a0 tx=0x7ff24c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=922b48c0e6c02c44 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:37.768 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.765+0000 7ff253fff640 1 -- 172.21.15.156:0/3727894927 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c02d040 con 0x7ff25c0b4010 2023-11-28T17:08:37.769 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.765+0000 7ff253fff640 1 -- 172.21.15.156:0/3727894927 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24c031810 con 0x7ff25c0b4010 2023-11-28T17:08:37.769 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.765+0000 7ff253fff640 1 -- 172.21.15.156:0/3727894927 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c031ba0 con 0x7ff25c0b4010 2023-11-28T17:08:37.769 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.765+0000 7ff2625cb640 1 -- 172.21.15.156:0/3727894927 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff25c0b4010 msgr2=0x7ff25c0d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:37.769 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.765+0000 7ff2625cb640 1 --2- 172.21.15.156:0/3727894927 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff25c0b4010 0x7ff25c0d18a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0097a0 tx=0x7ff24c02b8b0 comp rx=0 tx=0).stop 2023-11-28T17:08:37.769 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.766+0000 7ff2625cb640 1 -- 172.21.15.156:0/3727894927 shutdown_connections 2023-11-28T17:08:37.770 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.766+0000 7ff2625cb640 1 --2- 172.21.15.156:0/3727894927 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff25c0b4010 0x7ff25c0d18a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:37.770 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.766+0000 7ff2625cb640 1 --2- 172.21.15.156:0/3727894927 >> v2:172.21.15.116:3300/0 conn(0x7ff25c160b80 0x7ff25c0d1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:37.770 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.766+0000 7ff2625cb640 1 -- 172.21.15.156:0/3727894927 >> 172.21.15.156:0/3727894927 conn(0x7ff25c0d2fc0 msgr2=0x7ff25c0d33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:37.770 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.766+0000 7ff2625cb640 1 -- 172.21.15.156:0/3727894927 shutdown_connections 2023-11-28T17:08:37.770 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.766+0000 7ff2625cb640 1 -- 172.21.15.156:0/3727894927 wait complete. 2023-11-28T17:08:37.773 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.767+0000 7ff2625cb640 1 Processor -- start 2023-11-28T17:08:37.773 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.768+0000 7ff2625cb640 1 -- start start 2023-11-28T17:08:37.773 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.768+0000 7ff2625cb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff25c0b4010 0x7ff25c1f2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:37.773 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.768+0000 7ff2625cb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff25c0cf6f0 0x7ff25c1513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:37.773 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.768+0000 7ff2625cb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff25c160b80 0x7ff25c0cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:37.773 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.768+0000 7ff2625cb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0b7b90 con 0x7ff25c160b80 2023-11-28T17:08:37.773 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.768+0000 7ff2625cb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0b7010 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.773 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.768+0000 7ff2625cb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0b72f0 con 0x7ff25c0b4010 2023-11-28T17:08:37.773 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.769+0000 7ff2615c9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff25c0b4010 0x7ff25c1f2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:37.774 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.769+0000 7ff260dc8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff25c0cf6f0 0x7ff25c1513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:37.774 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.769+0000 7ff260dc8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff25c0cf6f0 0x7ff25c1513b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58986/0 (socket says 172.21.15.156:58986) 2023-11-28T17:08:37.774 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.769+0000 7ff260dc8640 1 -- 172.21.15.156:0/1914337669 learned_addr learned my addr 172.21.15.156:0/1914337669 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-28T17:08:37.774 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.769+0000 7ff261dca640 1 --2- 172.21.15.156:0/1914337669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff25c160b80 0x7ff25c0cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:37.774 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.769+0000 7ff260dc8640 1 -- 172.21.15.156:0/1914337669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff25c0b4010 msgr2=0x7ff25c1f2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:37.774 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.769+0000 7ff260dc8640 1 --2- 172.21.15.156:0/1914337669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff25c0b4010 0x7ff25c1f2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:37.774 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.769+0000 7ff260dc8640 1 -- 172.21.15.156:0/1914337669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff25c160b80 msgr2=0x7ff25c0cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:37.774 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.769+0000 7ff260dc8640 1 --2- 172.21.15.156:0/1914337669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff25c160b80 0x7ff25c0cbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:37.774 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.769+0000 7ff260dc8640 1 -- 172.21.15.156:0/1914337669 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c009400 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.775 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.769+0000 7ff2615c9640 1 --2- 172.21.15.156:0/1914337669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff25c0b4010 0x7ff25c1f2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:08:37.775 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.769+0000 7ff260dc8640 1 --2- 172.21.15.156:0/1914337669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff25c0cf6f0 0x7ff25c1513b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff24c000c00 tx=0x7ff24c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:37.775 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.770+0000 7ff251ffb640 1 -- 172.21.15.156:0/1914337669 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c03ad60 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.775 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.770+0000 7ff251ffb640 1 -- 172.21.15.156:0/1914337669 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24c0443f0 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.776 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.770+0000 7ff251ffb640 1 -- 172.21.15.156:0/1914337669 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c0447a0 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.776 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.770+0000 7ff2625cb640 1 -- 172.21.15.156:0/1914337669 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25c0b75d0 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.776 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.770+0000 7ff2625cb640 1 -- 172.21.15.156:0/1914337669 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25c1518f0 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.777 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.770+0000 7ff251ffb640 1 -- 172.21.15.156:0/1914337669 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff24c044c00 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.777 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.770+0000 7ff251ffb640 1 -- 172.21.15.156:0/1914337669 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff24c04da80 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.777 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.771+0000 7ff2625cb640 1 -- 172.21.15.156:0/1914337669 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25c067980 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.777 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.774+0000 7ff251ffb640 1 -- 172.21.15.156:0/1914337669 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff24c035020 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.852 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.851+0000 7ff2625cb640 1 -- 172.21.15.156:0/1914337669 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6991e794-5137-4365-be74-be350afa42e3", "id": 0} v 0) v1 -- 0x7ff25c0a0eb0 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.854 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:37.853+0000 7ff96c02e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T17:08:37.858 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.857+0000 7ff251ffb640 1 -- 172.21.15.156:0/1914337669 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6991e794-5137-4365-be74-be350afa42e3", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff24c04d450 con 0x7ff25c0cf6f0 2023-11-28T17:08:37.859 INFO:teuthology.orchestra.run.smithi156.stdout:0 2023-11-28T17:08:37.860 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.859+0000 7ff2625cb640 1 -- 172.21.15.156:0/1914337669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff25c0cf6f0 msgr2=0x7ff25c1513b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:37.860 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.859+0000 7ff2625cb640 1 --2- 172.21.15.156:0/1914337669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff25c0cf6f0 0x7ff25c1513b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff24c000c00 tx=0x7ff24c0046c0 comp rx=0 tx=0).stop 2023-11-28T17:08:37.860 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.859+0000 7ff2625cb640 1 -- 172.21.15.156:0/1914337669 shutdown_connections 2023-11-28T17:08:37.860 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.859+0000 7ff2625cb640 1 --2- 172.21.15.156:0/1914337669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff25c160b80 0x7ff25c0cbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:37.861 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.859+0000 7ff2625cb640 1 --2- 172.21.15.156:0/1914337669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff25c0cf6f0 0x7ff25c1513b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:37.861 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.859+0000 7ff2625cb640 1 --2- 172.21.15.156:0/1914337669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff25c0b4010 0x7ff25c1f2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:37.861 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.859+0000 7ff2625cb640 1 -- 172.21.15.156:0/1914337669 >> 172.21.15.156:0/1914337669 conn(0x7ff25c0d2fc0 msgr2=0x7ff25c0cdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:37.861 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.859+0000 7ff2625cb640 1 -- 172.21.15.156:0/1914337669 shutdown_connections 2023-11-28T17:08:37.861 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:37.859+0000 7ff2625cb640 1 -- 172.21.15.156:0/1914337669 wait complete. 2023-11-28T17:08:37.873 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:37.872+0000 7fb78c637040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T17:08:37.877 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new e1e544b0-64e0-472e-a4d3-9ded5287a676 1 2023-11-28T17:08:37.921 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:37.920+0000 7ff96c02e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T17:08:37.952 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:37.951+0000 7fb78c637040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T17:08:37.974 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:37.973+0000 7ff96c02e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T17:08:38.005 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:38.004+0000 7fb78c637040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T17:08:38.039 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:38.038+0000 7ff96c02e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T17:08:38.065 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:38.063+0000 7fb78c637040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T17:08:38.073 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.071+0000 7f994b59e640 1 Processor -- start 2023-11-28T17:08:38.073 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.072+0000 7f994b59e640 1 -- start start 2023-11-28T17:08:38.073 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.073+0000 7f994b59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f994c15d160 0x7f994c0d3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.074 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.073+0000 7f994b59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994c0d15c0 0x7f994c0cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.074 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.073+0000 7f994b59e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c0b5fd0 con 0x7f994c0cff10 2023-11-28T17:08:38.074 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.073+0000 7f994b59e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c0d0380 con 0x7f994c15d160 2023-11-28T17:08:38.074 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.073+0000 7f994b59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0d2060 con 0x7f994c0d15c0 2023-11-28T17:08:38.075 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.073+0000 7f9949d9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994c0d15c0 0x7f994c0cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.075 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.073+0000 7f9949d9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994c0d15c0 0x7f994c0cf9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:59382/0 (socket says 172.21.15.156:59382) 2023-11-28T17:08:38.075 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.073+0000 7f9949d9b640 1 -- 172.21.15.156:0/2020503694 learned_addr learned my addr 172.21.15.156:0/2020503694 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-28T17:08:38.075 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.073+0000 7f994a59c640 1 --2- 172.21.15.156:0/2020503694 >> v2:172.21.15.116:3300/0 conn(0x7f994c15d160 0x7f994c0d3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.075 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.074+0000 7f9949d9b640 1 -- 172.21.15.156:0/2020503694 >> v2:172.21.15.116:3300/0 conn(0x7f994c15d160 msgr2=0x7f994c0d3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:38.075 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.074+0000 7f9949d9b640 1 --2- 172.21.15.156:0/2020503694 >> v2:172.21.15.116:3300/0 conn(0x7f994c15d160 0x7f994c0d3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.075 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.074+0000 7f9949d9b640 1 -- 172.21.15.156:0/2020503694 >> v1:172.21.15.116:6789/0 conn(0x7f994c0cff10 legacy=0x7f994c0cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:38.076 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.074+0000 7f9949d9b640 1 -- 172.21.15.156:0/2020503694 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0d1d80 con 0x7f994c0d15c0 2023-11-28T17:08:38.076 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.074+0000 7f994a59c640 1 --2- 172.21.15.156:0/2020503694 >> v2:172.21.15.116:3300/0 conn(0x7f994c15d160 0x7f994c0d3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:08:38.077 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.075+0000 7f9949d9b640 1 --2- 172.21.15.156:0/2020503694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994c0d15c0 0x7f994c0cf9d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9940009870 tx=0x7f994002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b7fbc04b023fad7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:38.077 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.075+0000 7f994959a640 1 -- 172.21.15.156:0/2020503694 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9940031640 con 0x7f994c0d15c0 2023-11-28T17:08:38.077 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.075+0000 7f994959a640 1 -- 172.21.15.156:0/2020503694 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99400317a0 con 0x7f994c0d15c0 2023-11-28T17:08:38.077 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.075+0000 7f994959a640 1 -- 172.21.15.156:0/2020503694 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9940031b30 con 0x7f994c0d15c0 2023-11-28T17:08:38.077 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.076+0000 7f994b59e640 1 -- 172.21.15.156:0/2020503694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994c0d15c0 msgr2=0x7f994c0cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:38.078 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.076+0000 7f994b59e640 1 --2- 172.21.15.156:0/2020503694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994c0d15c0 0x7f994c0cf9d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9940009870 tx=0x7f994002b8f0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.078 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.076+0000 7f994b59e640 1 -- 172.21.15.156:0/2020503694 shutdown_connections 2023-11-28T17:08:38.078 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.076+0000 7f994b59e640 1 --2- 172.21.15.156:0/2020503694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994c0d15c0 0x7f994c0cf9d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.078 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.076+0000 7f994b59e640 1 --2- 172.21.15.156:0/2020503694 >> v2:172.21.15.116:3300/0 conn(0x7f994c15d160 0x7f994c0d3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.078 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.076+0000 7f994b59e640 1 -- 172.21.15.156:0/2020503694 >> 172.21.15.156:0/2020503694 conn(0x7f994c09c2f0 msgr2=0x7f994c0d2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:38.078 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.076+0000 7f994b59e640 1 -- 172.21.15.156:0/2020503694 shutdown_connections 2023-11-28T17:08:38.078 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.076+0000 7f994b59e640 1 -- 172.21.15.156:0/2020503694 wait complete. 2023-11-28T17:08:38.079 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.078+0000 7f994b59e640 1 Processor -- start 2023-11-28T17:08:38.079 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.078+0000 7f994b59e640 1 -- start start 2023-11-28T17:08:38.080 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994b59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f994c0cff10 0x7f994c0a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f994c0d15c0 0x7f994c147430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994b59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994c15d160 0x7f994c09e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994b59e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c15b6d0 con 0x7f994c0d15c0 2023-11-28T17:08:38.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994b59e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c09f3b0 con 0x7f994c0cff10 2023-11-28T17:08:38.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994b59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c09f690 con 0x7f994c15d160 2023-11-28T17:08:38.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f994c0cff10 0x7f994c0a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994ad9d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994c15d160 0x7f994c09e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f994c0cff10 0x7f994c0a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58992/0 (socket says 172.21.15.156:58992) 2023-11-28T17:08:38.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994a59c640 1 -- 172.21.15.156:0/1945170375 learned_addr learned my addr 172.21.15.156:0/1945170375 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-28T17:08:38.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f9949d9b640 1 --2- 172.21.15.156:0/1945170375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f994c0d15c0 0x7f994c147430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.081 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994ad9d640 1 -- 172.21.15.156:0/1945170375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f994c0cff10 msgr2=0x7f994c0a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:38.082 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994ad9d640 1 --2- 172.21.15.156:0/1945170375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f994c0cff10 0x7f994c0a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.082 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994ad9d640 1 -- 172.21.15.156:0/1945170375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f994c0d15c0 msgr2=0x7f994c147430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:38.082 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994ad9d640 1 --2- 172.21.15.156:0/1945170375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f994c0d15c0 0x7f994c147430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.082 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994ad9d640 1 -- 172.21.15.156:0/1945170375 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99400094d0 con 0x7f994c15d160 2023-11-28T17:08:38.082 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.079+0000 7f994a59c640 1 --2- 172.21.15.156:0/1945170375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f994c0cff10 0x7f994c0a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:08:38.082 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.080+0000 7f994ad9d640 1 --2- 172.21.15.156:0/1945170375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994c15d160 0x7f994c09e840 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f994400c780 tx=0x7f994400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:38.083 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.080+0000 7f993affd640 1 -- 172.21.15.156:0/1945170375 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994400aa50 con 0x7f994c15d160 2023-11-28T17:08:38.083 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.080+0000 7f993affd640 1 -- 172.21.15.156:0/1945170375 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994400abb0 con 0x7f994c15d160 2023-11-28T17:08:38.083 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.080+0000 7f994b59e640 1 -- 172.21.15.156:0/1945170375 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994c09f970 con 0x7f994c15d160 2023-11-28T17:08:38.083 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.080+0000 7f993affd640 1 -- 172.21.15.156:0/1945170375 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994400ce30 con 0x7f994c15d160 2023-11-28T17:08:38.084 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.080+0000 7f994b59e640 1 -- 172.21.15.156:0/1945170375 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c09ff30 con 0x7f994c15d160 2023-11-28T17:08:38.084 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.080+0000 7f993affd640 1 -- 172.21.15.156:0/1945170375 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9944004bf0 con 0x7f994c15d160 2023-11-28T17:08:38.084 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.080+0000 7f993affd640 1 -- 172.21.15.156:0/1945170375 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f9944012420 con 0x7f994c15d160 2023-11-28T17:08:38.084 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.081+0000 7f994b59e640 1 -- 172.21.15.156:0/1945170375 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994c0a0150 con 0x7f994c15d160 2023-11-28T17:08:38.085 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.084+0000 7f993affd640 1 -- 172.21.15.156:0/1945170375 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f9944039ce0 con 0x7f994c15d160 2023-11-28T17:08:38.092 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:38.091+0000 7ff96c02e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T17:08:38.119 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:38.118+0000 7fb78c637040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T17:08:38.163 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.162+0000 7f994b59e640 1 -- 172.21.15.156:0/1945170375 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e1e544b0-64e0-472e-a4d3-9ded5287a676", "id": 1} v 0) v1 -- 0x7f994c0a0ed0 con 0x7f994c15d160 2023-11-28T17:08:38.170 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.169+0000 7f993affd640 1 -- 172.21.15.156:0/1945170375 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e1e544b0-64e0-472e-a4d3-9ded5287a676", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9944028070 con 0x7f994c15d160 2023-11-28T17:08:38.170 INFO:teuthology.orchestra.run.smithi156.stdout:1 2023-11-28T17:08:38.172 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.171+0000 7f994b59e640 1 -- 172.21.15.156:0/1945170375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994c15d160 msgr2=0x7f994c09e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:38.172 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.171+0000 7f994b59e640 1 --2- 172.21.15.156:0/1945170375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994c15d160 0x7f994c09e840 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f994400c780 tx=0x7f994400cc40 comp rx=0 tx=0).stop 2023-11-28T17:08:38.172 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.171+0000 7f994b59e640 1 -- 172.21.15.156:0/1945170375 shutdown_connections 2023-11-28T17:08:38.172 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.171+0000 7f994b59e640 1 --2- 172.21.15.156:0/1945170375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f994c15d160 0x7f994c09e840 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.172 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.171+0000 7f994b59e640 1 --2- 172.21.15.156:0/1945170375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f994c0d15c0 0x7f994c147430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.173 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.171+0000 7f994b59e640 1 --2- 172.21.15.156:0/1945170375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f994c0cff10 0x7f994c0a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.173 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.171+0000 7f994b59e640 1 -- 172.21.15.156:0/1945170375 >> 172.21.15.156:0/1945170375 conn(0x7f994c09c2f0 msgr2=0x7f994c0d2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:38.173 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.171+0000 7f994b59e640 1 -- 172.21.15.156:0/1945170375 shutdown_connections 2023-11-28T17:08:38.173 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.171+0000 7f994b59e640 1 -- 172.21.15.156:0/1945170375 wait complete. 2023-11-28T17:08:38.188 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 1e3bb3de-7bb3-4888-a95b-6a22e4805344 2 2023-11-28T17:08:38.212 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:38.211+0000 7ff96c02e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T17:08:38.297 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:38.296+0000 7ff96c02e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T17:08:38.351 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:38.350+0000 7ff96c02e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T17:08:38.402 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.401+0000 7f7cf494b640 1 Processor -- start 2023-11-28T17:08:38.402 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.401+0000 7f7cf494b640 1 -- start start 2023-11-28T17:08:38.405 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.403+0000 7f7cf494b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7cf01606a0 0x7f7cf00ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.405 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.403+0000 7f7cf494b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7cf00662b0 0x7f7cf00ae000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.405 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.403+0000 7f7cf494b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cf00b25a0 0x7f7cf00ae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.405 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.403+0000 7f7cf494b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf006d830 con 0x7f7cf00662b0 2023-11-28T17:08:38.405 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.403+0000 7f7cf494b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf0160a80 con 0x7f7cf01606a0 2023-11-28T17:08:38.405 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.403+0000 7f7cf494b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf0066a50 con 0x7f7cf00b25a0 2023-11-28T17:08:38.405 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.403+0000 7f7ceed9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7cf01606a0 0x7f7cf00ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.405 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.403+0000 7f7ceed9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7cf01606a0 0x7f7cf00ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:59006/0 (socket says 172.21.15.156:59006) 2023-11-28T17:08:38.406 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.403+0000 7f7ceed9d640 1 -- 172.21.15.156:0/741818414 learned_addr learned my addr 172.21.15.156:0/741818414 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-28T17:08:38.406 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.403+0000 7f7cef59e640 1 --2- 172.21.15.156:0/741818414 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cf00b25a0 0x7f7cf00ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.406 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.404+0000 7f7cee59c640 1 --2- 172.21.15.156:0/741818414 >> v2:172.21.15.116:3300/0 conn(0x7f7cf00662b0 0x7f7cf00ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.406 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.404+0000 7f7ceed9d640 1 -- 172.21.15.156:0/741818414 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cf00b25a0 msgr2=0x7f7cf00ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:38.406 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.404+0000 7f7ceed9d640 1 --2- 172.21.15.156:0/741818414 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cf00b25a0 0x7f7cf00ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.406 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.404+0000 7f7ceed9d640 1 -- 172.21.15.156:0/741818414 >> v2:172.21.15.116:3300/0 conn(0x7f7cf00662b0 msgr2=0x7f7cf00ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:38.406 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.404+0000 7f7ceed9d640 1 --2- 172.21.15.156:0/741818414 >> v2:172.21.15.116:3300/0 conn(0x7f7cf00662b0 0x7f7cf00ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.406 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.404+0000 7f7ceed9d640 1 -- 172.21.15.156:0/741818414 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf0066770 con 0x7f7cf01606a0 2023-11-28T17:08:38.408 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.405+0000 7f7ceed9d640 1 --2- 172.21.15.156:0/741818414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cf01606a0 0x7f7cf00ad310 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0072430 tx=0x7f7ce002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f62de2a4a4417fb4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:38.408 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.405+0000 7f7cedd9b640 1 -- 172.21.15.156:0/741818414 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0031870 con 0x7f7cf01606a0 2023-11-28T17:08:38.408 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.405+0000 7f7cedd9b640 1 -- 172.21.15.156:0/741818414 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce00319d0 con 0x7f7cf01606a0 2023-11-28T17:08:38.408 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.405+0000 7f7cedd9b640 1 -- 172.21.15.156:0/741818414 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0031d60 con 0x7f7cf01606a0 2023-11-28T17:08:38.408 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.406+0000 7f7cf494b640 1 -- 172.21.15.156:0/741818414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cf01606a0 msgr2=0x7f7cf00ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:38.408 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.406+0000 7f7cf494b640 1 --2- 172.21.15.156:0/741818414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cf01606a0 0x7f7cf00ad310 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0072430 tx=0x7f7ce002b910 comp rx=0 tx=0).stop 2023-11-28T17:08:38.408 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.406+0000 7f7cf494b640 1 -- 172.21.15.156:0/741818414 shutdown_connections 2023-11-28T17:08:38.408 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.406+0000 7f7cf494b640 1 --2- 172.21.15.156:0/741818414 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cf00b25a0 0x7f7cf00ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.408 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.406+0000 7f7cf494b640 1 --2- 172.21.15.156:0/741818414 >> v2:172.21.15.116:3300/0 conn(0x7f7cf00662b0 0x7f7cf00ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.408 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.406+0000 7f7cf494b640 1 --2- 172.21.15.156:0/741818414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cf01606a0 0x7f7cf00ad310 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.409 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.406+0000 7f7cf494b640 1 -- 172.21.15.156:0/741818414 >> 172.21.15.156:0/741818414 conn(0x7f7cf0071890 msgr2=0x7f7cf006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:38.409 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.406+0000 7f7cf494b640 1 -- 172.21.15.156:0/741818414 shutdown_connections 2023-11-28T17:08:38.409 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.406+0000 7f7cf494b640 1 -- 172.21.15.156:0/741818414 wait complete. 2023-11-28T17:08:38.410 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.408+0000 7f7cf494b640 1 Processor -- start 2023-11-28T17:08:38.410 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.408+0000 7f7cf494b640 1 -- start start 2023-11-28T17:08:38.411 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7cf494b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cf00662b0 0x7f7cf0148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.411 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7cf494b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cf00b25a0 0x7f7cf01483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.411 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7cf494b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cf01606a0 0x7f7cf01597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.411 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7cf494b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf0159d10 con 0x7f7cf00b25a0 2023-11-28T17:08:38.411 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7cf494b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf0159fb0 con 0x7f7cf01606a0 2023-11-28T17:08:38.411 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7cf494b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf015a290 con 0x7f7cf00662b0 2023-11-28T17:08:38.411 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7ceed9d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cf00662b0 0x7f7cf0148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.411 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7cef59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cf01606a0 0x7f7cf01597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.411 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7ceed9d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cf00662b0 0x7f7cf0148f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:59404/0 (socket says 172.21.15.156:59404) 2023-11-28T17:08:38.411 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7ceed9d640 1 -- 172.21.15.156:0/911436416 learned_addr learned my addr 172.21.15.156:0/911436416 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-28T17:08:38.412 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7cee59c640 1 --2- 172.21.15.156:0/911436416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cf00b25a0 0x7f7cf01483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.412 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7ceed9d640 1 -- 172.21.15.156:0/911436416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cf01606a0 msgr2=0x7f7cf01597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:38.412 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7ceed9d640 1 --2- 172.21.15.156:0/911436416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cf01606a0 0x7f7cf01597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.412 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7ceed9d640 1 -- 172.21.15.156:0/911436416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cf00b25a0 msgr2=0x7f7cf01483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:38.412 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7ceed9d640 1 --2- 172.21.15.156:0/911436416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cf00b25a0 0x7f7cf01483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.412 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7ceed9d640 1 -- 172.21.15.156:0/911436416 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce00094d0 con 0x7f7cf00662b0 2023-11-28T17:08:38.412 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.409+0000 7f7cee59c640 1 --2- 172.21.15.156:0/911436416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cf00b25a0 0x7f7cf01483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:08:38.412 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.410+0000 7f7ceed9d640 1 --2- 172.21.15.156:0/911436416 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cf00662b0 0x7f7cf0148f80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7ce002b950 tx=0x7f7ce00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:38.413 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.410+0000 7f7cd77fe640 1 -- 172.21.15.156:0/911436416 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce003abe0 con 0x7f7cf00662b0 2023-11-28T17:08:38.413 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.410+0000 7f7cd77fe640 1 -- 172.21.15.156:0/911436416 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce003ad60 con 0x7f7cf00662b0 2023-11-28T17:08:38.413 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.410+0000 7f7cf494b640 1 -- 172.21.15.156:0/911436416 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf015a570 con 0x7f7cf00662b0 2023-11-28T17:08:38.414 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.410+0000 7f7cd77fe640 1 -- 172.21.15.156:0/911436416 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0044630 con 0x7f7cf00662b0 2023-11-28T17:08:38.414 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.410+0000 7f7cf494b640 1 -- 172.21.15.156:0/911436416 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf015ab30 con 0x7f7cf00662b0 2023-11-28T17:08:38.414 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.410+0000 7f7cd77fe640 1 -- 172.21.15.156:0/911436416 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7ce0044a90 con 0x7f7cf00662b0 2023-11-28T17:08:38.414 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.410+0000 7f7cd77fe640 1 -- 172.21.15.156:0/911436416 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f7ce002d040 con 0x7f7cf00662b0 2023-11-28T17:08:38.414 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.411+0000 7f7cf494b640 1 -- 172.21.15.156:0/911436416 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf015ad20 con 0x7f7cf00662b0 2023-11-28T17:08:38.417 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:38.414+0000 7ff96c02e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T17:08:38.417 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.414+0000 7f7cd77fe640 1 -- 172.21.15.156:0/911436416 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f7ce0035020 con 0x7f7cf00662b0 2023-11-28T17:08:38.471 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:38.470+0000 7ff96c02e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T17:08:38.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.492+0000 7f7cf494b640 1 -- 172.21.15.156:0/911436416 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1e3bb3de-7bb3-4888-a95b-6a22e4805344", "id": 2} v 0) v1 -- 0x7f7cf0072490 con 0x7f7cf00662b0 2023-11-28T17:08:38.500 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.499+0000 7f7cd77fe640 1 -- 172.21.15.156:0/911436416 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1e3bb3de-7bb3-4888-a95b-6a22e4805344", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7ce004d450 con 0x7f7cf00662b0 2023-11-28T17:08:38.500 INFO:teuthology.orchestra.run.smithi156.stdout:2 2023-11-28T17:08:38.501 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.500+0000 7f7cf494b640 1 -- 172.21.15.156:0/911436416 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cf00662b0 msgr2=0x7f7cf0148f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:38.501 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.500+0000 7f7cf494b640 1 --2- 172.21.15.156:0/911436416 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cf00662b0 0x7f7cf0148f80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7ce002b950 tx=0x7f7ce00046c0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.501 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.500+0000 7f7cf494b640 1 -- 172.21.15.156:0/911436416 shutdown_connections 2023-11-28T17:08:38.502 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.500+0000 7f7cf494b640 1 --2- 172.21.15.156:0/911436416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cf01606a0 0x7f7cf01597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.502 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.500+0000 7f7cf494b640 1 --2- 172.21.15.156:0/911436416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cf00b25a0 0x7f7cf01483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.502 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.500+0000 7f7cf494b640 1 --2- 172.21.15.156:0/911436416 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cf00662b0 0x7f7cf0148f80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.502 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.500+0000 7f7cf494b640 1 -- 172.21.15.156:0/911436416 >> 172.21.15.156:0/911436416 conn(0x7f7cf0071890 msgr2=0x7f7cf00b0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:38.502 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.500+0000 7f7cf494b640 1 -- 172.21.15.156:0/911436416 shutdown_connections 2023-11-28T17:08:38.502 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.500+0000 7f7cf494b640 1 -- 172.21.15.156:0/911436416 wait complete. 2023-11-28T17:08:38.516 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 9c345281-1fc7-42ad-9732-1585e687e9e3 3 2023-11-28T17:08:38.567 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:38.566+0000 7fb78c637040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T17:08:38.697 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.695+0000 7f3dc95ef640 1 Processor -- start 2023-11-28T17:08:38.697 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.696+0000 7f3dc95ef640 1 -- start start 2023-11-28T17:08:38.698 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.697+0000 7f3dc95ef640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3dc415af50 0x7f3dc4164a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.698 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.697+0000 7f3dc95ef640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc40b2f00 con 0x7f3dc406da50 2023-11-28T17:08:38.698 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.697+0000 7f3dc95ef640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc40b31e0 con 0x7f3dc415a460 2023-11-28T17:08:38.699 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.697+0000 7f3dc95ef640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc40b34c0 con 0x7f3dc415af50 2023-11-28T17:08:38.699 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.697+0000 7f3dc3fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f3dc415a460 0x7f3dc406b6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:46432/0 (socket says 172.21.15.156:46432) 2023-11-28T17:08:38.699 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.697+0000 7f3dc3fff640 1 -- 172.21.15.156:0/2170873932 learned_addr learned my addr 172.21.15.156:0/2170873932 (peer_addr_for_me v1:172.21.15.156:0/0) 2023-11-28T17:08:38.699 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.697+0000 7f3dc8dee640 1 --2- 172.21.15.156:0/2170873932 >> v2:172.21.15.156:3300/0 conn(0x7f3dc415af50 0x7f3dc4164a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.700 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.698+0000 7f3dc37fe640 1 -- 172.21.15.156:0/2170873932 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421597878 0 0) 0x7f3dc40b31e0 con 0x7f3dc415a460 2023-11-28T17:08:38.700 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.698+0000 7f3dc37fe640 1 -- 172.21.15.156:0/2170873932 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db0003520 con 0x7f3dc415a460 2023-11-28T17:08:38.700 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.698+0000 7f3dc8dee640 1 -- 172.21.15.156:0/2170873932 >> v1:172.21.15.156:6789/0 conn(0x7f3dc415a460 legacy=0x7f3dc406b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:38.700 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.698+0000 7f3dc8dee640 1 -- 172.21.15.156:0/2170873932 >> v1:172.21.15.116:6789/0 conn(0x7f3dc406da50 legacy=0x7f3dc415a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:38.700 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.698+0000 7f3dc8dee640 1 -- 172.21.15.156:0/2170873932 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc40b37a0 con 0x7f3dc415af50 2023-11-28T17:08:38.702 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.700+0000 7f3dc8dee640 1 --2- 172.21.15.156:0/2170873932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dc415af50 0x7f3dc4164a20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc009d90 tx=0x7f3dbc02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c17221d85252a75a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:38.702 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.700+0000 7f3dc37fe640 1 -- 172.21.15.156:0/2170873932 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc02cc60 con 0x7f3dc415af50 2023-11-28T17:08:38.702 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.700+0000 7f3dc37fe640 1 -- 172.21.15.156:0/2170873932 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc02cdc0 con 0x7f3dc415af50 2023-11-28T17:08:38.702 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.700+0000 7f3dc37fe640 1 -- 172.21.15.156:0/2170873932 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc034660 con 0x7f3dc415af50 2023-11-28T17:08:38.702 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.700+0000 7f3dc95ef640 1 -- 172.21.15.156:0/2170873932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dc415af50 msgr2=0x7f3dc4164a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:38.702 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.700+0000 7f3dc95ef640 1 --2- 172.21.15.156:0/2170873932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dc415af50 0x7f3dc4164a20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc009d90 tx=0x7f3dbc02beb0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.702 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.701+0000 7f3dc95ef640 1 -- 172.21.15.156:0/2170873932 shutdown_connections 2023-11-28T17:08:38.702 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.701+0000 7f3dc95ef640 1 --2- 172.21.15.156:0/2170873932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dc415af50 0x7f3dc4164a20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.702 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.701+0000 7f3dc95ef640 1 -- 172.21.15.156:0/2170873932 >> 172.21.15.156:0/2170873932 conn(0x7f3dc40cf3f0 msgr2=0x7f3dc40cf800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:38.703 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.701+0000 7f3dc95ef640 1 -- 172.21.15.156:0/2170873932 shutdown_connections 2023-11-28T17:08:38.703 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.701+0000 7f3dc95ef640 1 -- 172.21.15.156:0/2170873932 wait complete. 2023-11-28T17:08:38.703 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.702+0000 7f3dc95ef640 1 Processor -- start 2023-11-28T17:08:38.703 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.702+0000 7f3dc95ef640 1 -- start start 2023-11-28T17:08:38.706 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.703+0000 7f3dc95ef640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dc406da50 0x7f3dc415cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.706 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.703+0000 7f3dc95ef640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3dc415a460 0x7f3dc41581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.706 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.703+0000 7f3dc95ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc415af50 0x7f3dc4158700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.706 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.703+0000 7f3dc95ef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc415d470 con 0x7f3dc415af50 2023-11-28T17:08:38.706 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.703+0000 7f3dc95ef640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc415b8c0 con 0x7f3dc406da50 2023-11-28T17:08:38.706 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.703+0000 7f3dc95ef640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc415bba0 con 0x7f3dc415a460 2023-11-28T17:08:38.707 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.704+0000 7f3dc3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dc406da50 0x7f3dc415cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.707 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.704+0000 7f3dc3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dc406da50 0x7f3dc415cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:59032/0 (socket says 172.21.15.156:59032) 2023-11-28T17:08:38.707 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.704+0000 7f3dc3fff640 1 -- 172.21.15.156:0/1500457049 learned_addr learned my addr 172.21.15.156:0/1500457049 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-28T17:08:38.707 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.704+0000 7f3dbbfff640 1 --2- 172.21.15.156:0/1500457049 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3dc415a460 0x7f3dc41581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.707 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.704+0000 7f3dc8dee640 1 --2- 172.21.15.156:0/1500457049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc415af50 0x7f3dc4158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.707 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.704+0000 7f3dc3fff640 1 -- 172.21.15.156:0/1500457049 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3dc415a460 msgr2=0x7f3dc41581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:38.707 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.704+0000 7f3dc3fff640 1 --2- 172.21.15.156:0/1500457049 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3dc415a460 0x7f3dc41581c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.707 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.704+0000 7f3dc3fff640 1 -- 172.21.15.156:0/1500457049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc415af50 msgr2=0x7f3dc4158700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:38.707 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.704+0000 7f3dc3fff640 1 --2- 172.21.15.156:0/1500457049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc415af50 0x7f3dc4158700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.707 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.704+0000 7f3dc3fff640 1 -- 172.21.15.156:0/1500457049 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dbc009a70 con 0x7f3dc406da50 2023-11-28T17:08:38.707 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.704+0000 7f3dc8dee640 1 --2- 172.21.15.156:0/1500457049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc415af50 0x7f3dc4158700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:08:38.707 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.704+0000 7f3dc3fff640 1 --2- 172.21.15.156:0/1500457049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dc406da50 0x7f3dc415cf30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3db400c500 tx=0x7f3db400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:38.708 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.705+0000 7f3dc17fa640 1 -- 172.21.15.156:0/1500457049 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db4005ce0 con 0x7f3dc406da50 2023-11-28T17:08:38.708 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.705+0000 7f3dc17fa640 1 -- 172.21.15.156:0/1500457049 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db4005e40 con 0x7f3dc406da50 2023-11-28T17:08:38.708 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.705+0000 7f3dc95ef640 1 -- 172.21.15.156:0/1500457049 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc415be80 con 0x7f3dc406da50 2023-11-28T17:08:38.708 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.705+0000 7f3dc17fa640 1 -- 172.21.15.156:0/1500457049 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db4007330 con 0x7f3dc406da50 2023-11-28T17:08:38.708 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.705+0000 7f3dc95ef640 1 -- 172.21.15.156:0/1500457049 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc415c440 con 0x7f3dc406da50 2023-11-28T17:08:38.708 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.705+0000 7f3dc17fa640 1 -- 172.21.15.156:0/1500457049 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3db401c440 con 0x7f3dc406da50 2023-11-28T17:08:38.708 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.705+0000 7f3dc17fa640 1 -- 172.21.15.156:0/1500457049 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f3db4012980 con 0x7f3dc406da50 2023-11-28T17:08:38.708 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.706+0000 7f3dc95ef640 1 -- 172.21.15.156:0/1500457049 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc40cb990 con 0x7f3dc406da50 2023-11-28T17:08:38.711 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.710+0000 7f3dc17fa640 1 -- 172.21.15.156:0/1500457049 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f3dc40cb990 con 0x7f3dc406da50 2023-11-28T17:08:38.725 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:38.724+0000 7fb78c637040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T17:08:38.788 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.787+0000 7f3dc95ef640 1 -- 172.21.15.156:0/1500457049 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9c345281-1fc7-42ad-9732-1585e687e9e3", "id": 3} v 0) v1 -- 0x7f3dc41548b0 con 0x7f3dc406da50 2023-11-28T17:08:38.795 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.794+0000 7f3dc17fa640 1 -- 172.21.15.156:0/1500457049 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9c345281-1fc7-42ad-9732-1585e687e9e3", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3db4017070 con 0x7f3dc406da50 2023-11-28T17:08:38.795 INFO:teuthology.orchestra.run.smithi156.stdout:3 2023-11-28T17:08:38.796 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.795+0000 7f3dc95ef640 1 -- 172.21.15.156:0/1500457049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dc406da50 msgr2=0x7f3dc415cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:38.797 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.795+0000 7f3dc95ef640 1 --2- 172.21.15.156:0/1500457049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dc406da50 0x7f3dc415cf30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3db400c500 tx=0x7f3db400c9c0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.797 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.795+0000 7f3dc95ef640 1 -- 172.21.15.156:0/1500457049 shutdown_connections 2023-11-28T17:08:38.797 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.795+0000 7f3dc95ef640 1 --2- 172.21.15.156:0/1500457049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3dc415af50 0x7f3dc4158700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.797 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.795+0000 7f3dc95ef640 1 --2- 172.21.15.156:0/1500457049 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3dc415a460 0x7f3dc41581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.797 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.795+0000 7f3dc95ef640 1 --2- 172.21.15.156:0/1500457049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dc406da50 0x7f3dc415cf30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:38.797 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.795+0000 7f3dc95ef640 1 -- 172.21.15.156:0/1500457049 >> 172.21.15.156:0/1500457049 conn(0x7f3dc40cf3f0 msgr2=0x7f3dc40733f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:38.797 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.795+0000 7f3dc95ef640 1 -- 172.21.15.156:0/1500457049 shutdown_connections 2023-11-28T17:08:38.798 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.795+0000 7f3dc95ef640 1 -- 172.21.15.156:0/1500457049 wait complete. 2023-11-28T17:08:38.812 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new c815b7c4-e49f-4a53-a514-6546fffbe041 4 2023-11-28T17:08:38.868 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:38.867+0000 7fb78c637040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T17:08:38.920 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:38.919+0000 7fb78c637040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T17:08:38.946 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:38.945+0000 7ff96c02e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T17:08:38.996 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.995+0000 7faf6f59e640 1 Processor -- start 2023-11-28T17:08:38.996 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.995+0000 7faf6f59e640 1 -- start start 2023-11-28T17:08:38.998 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6f59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faf70160b80 0x7faf700d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.999 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6f59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faf700b4010 0x7faf700d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.999 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6f59e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7faf700cf6f0 0x7faf700cfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:38.999 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6f59e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf70160f60 con 0x7faf700b4010 2023-11-28T17:08:38.999 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6f59e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf700d0da0 con 0x7faf700cf6f0 2023-11-28T17:08:38.999 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6f59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf700d1080 con 0x7faf70160b80 2023-11-28T17:08:38.999 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6e59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faf70160b80 0x7faf700d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:38.999 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6e59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faf70160b80 0x7faf700d1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:59420/0 (socket says 172.21.15.156:59420) 2023-11-28T17:08:39.000 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6e59c640 1 -- 172.21.15.156:0/2170870121 learned_addr learned my addr 172.21.15.156:0/2170870121 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-28T17:08:39.000 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6ed9d640 1 --2- 172.21.15.156:0/2170870121 >> v2:172.21.15.156:3300/0 conn(0x7faf700cf6f0 0x7faf700cfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.000 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf67fff640 1 --2- 172.21.15.156:0/2170870121 >> v2:172.21.15.116:3300/0 conn(0x7faf700b4010 0x7faf700d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.000 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6e59c640 1 -- 172.21.15.156:0/2170870121 >> v2:172.21.15.156:3300/0 conn(0x7faf700cf6f0 msgr2=0x7faf700cfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:39.000 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6e59c640 1 --2- 172.21.15.156:0/2170870121 >> v2:172.21.15.156:3300/0 conn(0x7faf700cf6f0 0x7faf700cfb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.000 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6e59c640 1 -- 172.21.15.156:0/2170870121 >> v2:172.21.15.116:3300/0 conn(0x7faf700b4010 msgr2=0x7faf700d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:39.000 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6e59c640 1 --2- 172.21.15.156:0/2170870121 >> v2:172.21.15.116:3300/0 conn(0x7faf700b4010 0x7faf700d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.000 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf6e59c640 1 -- 172.21.15.156:0/2170870121 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf700ce780 con 0x7faf70160b80 2023-11-28T17:08:39.000 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.997+0000 7faf67fff640 1 --2- 172.21.15.156:0/2170870121 >> v2:172.21.15.116:3300/0 conn(0x7faf700b4010 0x7faf700d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:08:39.002 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.999+0000 7faf6e59c640 1 --2- 172.21.15.156:0/2170870121 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faf70160b80 0x7faf700d1360 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faf7008c400 tx=0x7faf6002b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a6969449356d80b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:39.002 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.999+0000 7faf6dd9b640 1 -- 172.21.15.156:0/2170870121 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60031870 con 0x7faf70160b80 2023-11-28T17:08:39.002 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.999+0000 7faf6dd9b640 1 -- 172.21.15.156:0/2170870121 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf600319d0 con 0x7faf70160b80 2023-11-28T17:08:39.002 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.999+0000 7faf6dd9b640 1 -- 172.21.15.156:0/2170870121 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60031d60 con 0x7faf70160b80 2023-11-28T17:08:39.002 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.999+0000 7faf6f59e640 1 -- 172.21.15.156:0/2170870121 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faf70160b80 msgr2=0x7faf700d1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.002 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:38.999+0000 7faf6f59e640 1 --2- 172.21.15.156:0/2170870121 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faf70160b80 0x7faf700d1360 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faf7008c400 tx=0x7faf6002b910 comp rx=0 tx=0).stop 2023-11-28T17:08:39.002 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.000+0000 7faf6f59e640 1 -- 172.21.15.156:0/2170870121 shutdown_connections 2023-11-28T17:08:39.002 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.000+0000 7faf6f59e640 1 --2- 172.21.15.156:0/2170870121 >> v2:172.21.15.156:3300/0 conn(0x7faf700cf6f0 0x7faf700cfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.002 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.000+0000 7faf6f59e640 1 --2- 172.21.15.156:0/2170870121 >> v2:172.21.15.116:3300/0 conn(0x7faf700b4010 0x7faf700d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.003 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.000+0000 7faf6f59e640 1 --2- 172.21.15.156:0/2170870121 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faf70160b80 0x7faf700d1360 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.003 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.000+0000 7faf6f59e640 1 -- 172.21.15.156:0/2170870121 >> 172.21.15.156:0/2170870121 conn(0x7faf700d2fc0 msgr2=0x7faf700d33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:39.003 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.000+0000 7faf6f59e640 1 -- 172.21.15.156:0/2170870121 shutdown_connections 2023-11-28T17:08:39.003 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.000+0000 7faf6f59e640 1 -- 172.21.15.156:0/2170870121 wait complete. 2023-11-28T17:08:39.005 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.001+0000 7faf6f59e640 1 Processor -- start 2023-11-28T17:08:39.005 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.001+0000 7faf6f59e640 1 -- start start 2023-11-28T17:08:39.005 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.002+0000 7faf6f59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faf700b4010 0x7faf700b7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.005 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.002+0000 7faf6f59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf700cf6f0 0x7faf700cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.005 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.002+0000 7faf6f59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf70160b80 0x7faf7014a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.005 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.002+0000 7faf6f59e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf700b6fd0 con 0x7faf700cf6f0 2023-11-28T17:08:39.006 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.002+0000 7faf6f59e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf700b63c0 con 0x7faf70160b80 2023-11-28T17:08:39.006 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.002+0000 7faf6f59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf700b66a0 con 0x7faf700b4010 2023-11-28T17:08:39.006 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.003+0000 7faf6e59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faf700b4010 0x7faf700b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.006 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.003+0000 7faf6ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf70160b80 0x7faf7014a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.006 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.003+0000 7faf6e59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faf700b4010 0x7faf700b7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:59426/0 (socket says 172.21.15.156:59426) 2023-11-28T17:08:39.006 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.003+0000 7faf6e59c640 1 -- 172.21.15.156:0/1381478939 learned_addr learned my addr 172.21.15.156:0/1381478939 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-28T17:08:39.006 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.003+0000 7faf67fff640 1 --2- 172.21.15.156:0/1381478939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf700cf6f0 0x7faf700cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.006 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.003+0000 7faf6e59c640 1 -- 172.21.15.156:0/1381478939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf70160b80 msgr2=0x7faf7014a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.006 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.003+0000 7faf6e59c640 1 --2- 172.21.15.156:0/1381478939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf70160b80 0x7faf7014a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.006 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.003+0000 7faf6e59c640 1 -- 172.21.15.156:0/1381478939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf700cf6f0 msgr2=0x7faf700cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.006 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.003+0000 7faf6e59c640 1 --2- 172.21.15.156:0/1381478939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf700cf6f0 0x7faf700cae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.007 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.003+0000 7faf6e59c640 1 -- 172.21.15.156:0/1381478939 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf600094d0 con 0x7faf700b4010 2023-11-28T17:08:39.007 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.003+0000 7faf67fff640 1 --2- 172.21.15.156:0/1381478939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf700cf6f0 0x7faf700cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:08:39.007 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.003+0000 7faf6e59c640 1 --2- 172.21.15.156:0/1381478939 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faf700b4010 0x7faf700b7460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faf60000c00 tx=0x7faf600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:39.007 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.004+0000 7faf66ffd640 1 -- 172.21.15.156:0/1381478939 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6003abe0 con 0x7faf700b4010 2023-11-28T17:08:39.007 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.004+0000 7faf66ffd640 1 -- 172.21.15.156:0/1381478939 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6003ad40 con 0x7faf700b4010 2023-11-28T17:08:39.007 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.004+0000 7faf66ffd640 1 -- 172.21.15.156:0/1381478939 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60044630 con 0x7faf700b4010 2023-11-28T17:08:39.007 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.004+0000 7faf6f59e640 1 -- 172.21.15.156:0/1381478939 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf700b6980 con 0x7faf700b4010 2023-11-28T17:08:39.007 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.004+0000 7faf6f59e640 1 -- 172.21.15.156:0/1381478939 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf700b5be0 con 0x7faf700b4010 2023-11-28T17:08:39.007 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.004+0000 7faf66ffd640 1 -- 172.21.15.156:0/1381478939 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7faf60044a90 con 0x7faf700b4010 2023-11-28T17:08:39.007 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.004+0000 7faf66ffd640 1 -- 172.21.15.156:0/1381478939 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7faf6002d040 con 0x7faf700b4010 2023-11-28T17:08:39.008 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.005+0000 7faf6f59e640 1 -- 172.21.15.156:0/1381478939 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf70067980 con 0x7faf700b4010 2023-11-28T17:08:39.008 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.007+0000 7faf66ffd640 1 -- 172.21.15.156:0/1381478939 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7faf60035020 con 0x7faf700b4010 2023-11-28T17:08:39.029 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:39.028+0000 7fb78c637040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T17:08:39.085 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.084+0000 7faf6f59e640 1 -- 172.21.15.156:0/1381478939 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c815b7c4-e49f-4a53-a514-6546fffbe041", "id": 4} v 0) v1 -- 0x7faf700a0eb0 con 0x7faf700b4010 2023-11-28T17:08:39.092 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.091+0000 7faf66ffd640 1 -- 172.21.15.156:0/1381478939 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c815b7c4-e49f-4a53-a514-6546fffbe041", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7faf60044d40 con 0x7faf700b4010 2023-11-28T17:08:39.092 INFO:teuthology.orchestra.run.smithi156.stdout:4 2023-11-28T17:08:39.094 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.093+0000 7faf6f59e640 1 -- 172.21.15.156:0/1381478939 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faf700b4010 msgr2=0x7faf700b7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.094 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.093+0000 7faf6f59e640 1 --2- 172.21.15.156:0/1381478939 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faf700b4010 0x7faf700b7460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faf60000c00 tx=0x7faf600046c0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.094 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.093+0000 7faf6f59e640 1 -- 172.21.15.156:0/1381478939 shutdown_connections 2023-11-28T17:08:39.094 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.093+0000 7faf6f59e640 1 --2- 172.21.15.156:0/1381478939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf70160b80 0x7faf7014a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.094 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.093+0000 7faf6f59e640 1 --2- 172.21.15.156:0/1381478939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf700cf6f0 0x7faf700cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.095 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.093+0000 7faf6f59e640 1 --2- 172.21.15.156:0/1381478939 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faf700b4010 0x7faf700b7460 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.095 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.093+0000 7faf6f59e640 1 -- 172.21.15.156:0/1381478939 >> 172.21.15.156:0/1381478939 conn(0x7faf700d2fc0 msgr2=0x7faf700b4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:39.095 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.093+0000 7faf6f59e640 1 -- 172.21.15.156:0/1381478939 shutdown_connections 2023-11-28T17:08:39.095 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.093+0000 7faf6f59e640 1 -- 172.21.15.156:0/1381478939 wait complete. 2023-11-28T17:08:39.109 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 5d573d94-b928-49c4-9c7a-792052e375da 5 2023-11-28T17:08:39.115 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:39.113+0000 7ff96c02e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T17:08:39.139 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:39.137+0000 7fb78c637040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T17:08:39.257 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:39.256+0000 7ff96c02e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T17:08:39.297 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:39.295+0000 7fb78c637040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T17:08:39.301 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.300+0000 7fb34d147640 1 Processor -- start 2023-11-28T17:08:39.301 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.300+0000 7fb34d147640 1 -- start start 2023-11-28T17:08:39.304 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.302+0000 7fb34d147640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb34815d160 0x7fb3480d3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.304 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.302+0000 7fb34d147640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb3480d15c0 0x7fb3480cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.304 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.302+0000 7fb34d147640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3480cff10 0x7fb3480cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.304 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.302+0000 7fb34d147640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3480d0380 con 0x7fb34815d160 2023-11-28T17:08:39.304 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.302+0000 7fb34d147640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3480b5fd0 con 0x7fb3480d15c0 2023-11-28T17:08:39.304 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.302+0000 7fb34d147640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480d2060 con 0x7fb3480cff10 2023-11-28T17:08:39.304 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.302+0000 7fb33ffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb3480d15c0 0x7fb3480cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.304 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.302+0000 7fb34c946640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3480cff10 0x7fb3480cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.304 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.302+0000 7fb33ffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb3480d15c0 0x7fb3480cf9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:59058/0 (socket says 172.21.15.156:59058) 2023-11-28T17:08:39.305 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.302+0000 7fb33ffff640 1 -- 172.21.15.156:0/2973260453 learned_addr learned my addr 172.21.15.156:0/2973260453 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-28T17:08:39.305 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.302+0000 7fb347fff640 1 --2- 172.21.15.156:0/2973260453 >> v2:172.21.15.116:3300/0 conn(0x7fb34815d160 0x7fb3480d3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.305 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.303+0000 7fb33ffff640 1 -- 172.21.15.156:0/2973260453 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3480cff10 msgr2=0x7fb3480cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:39.305 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.303+0000 7fb33ffff640 1 --2- 172.21.15.156:0/2973260453 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3480cff10 0x7fb3480cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.305 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.303+0000 7fb33ffff640 1 -- 172.21.15.156:0/2973260453 >> v2:172.21.15.116:3300/0 conn(0x7fb34815d160 msgr2=0x7fb3480d3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:39.305 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.303+0000 7fb33ffff640 1 --2- 172.21.15.156:0/2973260453 >> v2:172.21.15.116:3300/0 conn(0x7fb34815d160 0x7fb3480d3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.305 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.303+0000 7fb33ffff640 1 -- 172.21.15.156:0/2973260453 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3480d1d80 con 0x7fb3480d15c0 2023-11-28T17:08:39.306 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.304+0000 7fb33ffff640 1 --2- 172.21.15.156:0/2973260453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3480d15c0 0x7fb3480cf9d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb3480d2f40 tx=0x7fb33002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3a7a4ba17c380d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:39.307 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.304+0000 7fb3477fe640 1 -- 172.21.15.156:0/2973260453 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330031870 con 0x7fb3480d15c0 2023-11-28T17:08:39.307 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.304+0000 7fb3477fe640 1 -- 172.21.15.156:0/2973260453 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3300319d0 con 0x7fb3480d15c0 2023-11-28T17:08:39.307 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.304+0000 7fb3477fe640 1 -- 172.21.15.156:0/2973260453 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330031d60 con 0x7fb3480d15c0 2023-11-28T17:08:39.307 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.305+0000 7fb34d147640 1 -- 172.21.15.156:0/2973260453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3480d15c0 msgr2=0x7fb3480cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.307 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.305+0000 7fb34d147640 1 --2- 172.21.15.156:0/2973260453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3480d15c0 0x7fb3480cf9d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb3480d2f40 tx=0x7fb33002b910 comp rx=0 tx=0).stop 2023-11-28T17:08:39.307 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.305+0000 7fb34d147640 1 -- 172.21.15.156:0/2973260453 shutdown_connections 2023-11-28T17:08:39.307 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.305+0000 7fb34d147640 1 --2- 172.21.15.156:0/2973260453 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3480cff10 0x7fb3480cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.307 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.305+0000 7fb34d147640 1 --2- 172.21.15.156:0/2973260453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3480d15c0 0x7fb3480cf9d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.308 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.305+0000 7fb34d147640 1 --2- 172.21.15.156:0/2973260453 >> v2:172.21.15.116:3300/0 conn(0x7fb34815d160 0x7fb3480d3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.308 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.305+0000 7fb34d147640 1 -- 172.21.15.156:0/2973260453 >> 172.21.15.156:0/2973260453 conn(0x7fb34809c2f0 msgr2=0x7fb3480d2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:39.308 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.305+0000 7fb34d147640 1 -- 172.21.15.156:0/2973260453 shutdown_connections 2023-11-28T17:08:39.308 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.305+0000 7fb34d147640 1 -- 172.21.15.156:0/2973260453 wait complete. 2023-11-28T17:08:39.310 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.306+0000 7fb34d147640 1 Processor -- start 2023-11-28T17:08:39.310 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.307+0000 7fb34d147640 1 -- start start 2023-11-28T17:08:39.311 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb34d147640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3480cff10 0x7fb34815f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.311 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb34d147640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3480d15c0 0x7fb34814dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.311 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb34d147640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb34815d160 0x7fb34814e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.311 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb34d147640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34814d7d0 con 0x7fb3480cff10 2023-11-28T17:08:39.311 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb34d147640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34814dc60 con 0x7fb3480d15c0 2023-11-28T17:08:39.311 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb34d147640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34814cbc0 con 0x7fb34815d160 2023-11-28T17:08:39.311 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb33ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3480d15c0 0x7fb34814dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.311 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb33ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3480d15c0 0x7fb34814dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:59068/0 (socket says 172.21.15.156:59068) 2023-11-28T17:08:39.311 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb33ffff640 1 -- 172.21.15.156:0/627064124 learned_addr learned my addr 172.21.15.156:0/627064124 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-28T17:08:39.311 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb34c946640 1 --2- 172.21.15.156:0/627064124 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb34815d160 0x7fb34814e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.311 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb347fff640 1 --2- 172.21.15.156:0/627064124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3480cff10 0x7fb34815f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.312 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb33ffff640 1 -- 172.21.15.156:0/627064124 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb34815d160 msgr2=0x7fb34814e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.312 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb33ffff640 1 --2- 172.21.15.156:0/627064124 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb34815d160 0x7fb34814e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.312 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb33ffff640 1 -- 172.21.15.156:0/627064124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3480cff10 msgr2=0x7fb34815f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.312 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb33ffff640 1 --2- 172.21.15.156:0/627064124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3480cff10 0x7fb34815f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.312 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.308+0000 7fb33ffff640 1 -- 172.21.15.156:0/627064124 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3300094d0 con 0x7fb3480d15c0 2023-11-28T17:08:39.312 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.309+0000 7fb33ffff640 1 --2- 172.21.15.156:0/627064124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3480d15c0 0x7fb34814dee0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb33002b950 tx=0x7fb3300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:39.312 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.309+0000 7fb3457fa640 1 -- 172.21.15.156:0/627064124 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33003abe0 con 0x7fb3480d15c0 2023-11-28T17:08:39.312 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.309+0000 7fb3457fa640 1 -- 172.21.15.156:0/627064124 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33003ad60 con 0x7fb3480d15c0 2023-11-28T17:08:39.312 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.309+0000 7fb3457fa640 1 -- 172.21.15.156:0/627064124 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330044630 con 0x7fb3480d15c0 2023-11-28T17:08:39.312 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.309+0000 7fb34d147640 1 -- 172.21.15.156:0/627064124 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34814cea0 con 0x7fb3480d15c0 2023-11-28T17:08:39.312 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.309+0000 7fb34d147640 1 -- 172.21.15.156:0/627064124 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34814d460 con 0x7fb3480d15c0 2023-11-28T17:08:39.312 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.309+0000 7fb3457fa640 1 -- 172.21.15.156:0/627064124 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb330044a90 con 0x7fb3480d15c0 2023-11-28T17:08:39.313 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.309+0000 7fb3457fa640 1 -- 172.21.15.156:0/627064124 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fb33002d040 con 0x7fb3480d15c0 2023-11-28T17:08:39.313 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.310+0000 7fb34d147640 1 -- 172.21.15.156:0/627064124 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3480b6e60 con 0x7fb3480d15c0 2023-11-28T17:08:39.314 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.313+0000 7fb3457fa640 1 -- 172.21.15.156:0/627064124 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fb330035020 con 0x7fb3480d15c0 2023-11-28T17:08:39.314 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:39.313+0000 7ff96c02e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T17:08:39.347 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:39.346+0000 7fb78c637040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T17:08:39.392 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.391+0000 7fb34d147640 1 -- 172.21.15.156:0/627064124 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5d573d94-b928-49c4-9c7a-792052e375da", "id": 5} v 0) v1 -- 0x7fb34814d940 con 0x7fb3480d15c0 2023-11-28T17:08:39.398 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.397+0000 7fb3457fa640 1 -- 172.21.15.156:0/627064124 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5d573d94-b928-49c4-9c7a-792052e375da", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb33004d450 con 0x7fb3480d15c0 2023-11-28T17:08:39.398 INFO:teuthology.orchestra.run.smithi156.stdout:5 2023-11-28T17:08:39.399 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.398+0000 7fb34d147640 1 -- 172.21.15.156:0/627064124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3480d15c0 msgr2=0x7fb34814dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.400 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.398+0000 7fb34d147640 1 --2- 172.21.15.156:0/627064124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3480d15c0 0x7fb34814dee0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb33002b950 tx=0x7fb3300046c0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.400 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.398+0000 7fb34d147640 1 -- 172.21.15.156:0/627064124 shutdown_connections 2023-11-28T17:08:39.400 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.398+0000 7fb34d147640 1 --2- 172.21.15.156:0/627064124 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb34815d160 0x7fb34814e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.400 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.398+0000 7fb34d147640 1 --2- 172.21.15.156:0/627064124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3480d15c0 0x7fb34814dee0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.400 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.398+0000 7fb34d147640 1 --2- 172.21.15.156:0/627064124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3480cff10 0x7fb34815f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.400 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.398+0000 7fb34d147640 1 -- 172.21.15.156:0/627064124 >> 172.21.15.156:0/627064124 conn(0x7fb34809c2f0 msgr2=0x7fb3480d2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:39.400 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.398+0000 7fb34d147640 1 -- 172.21.15.156:0/627064124 shutdown_connections 2023-11-28T17:08:39.400 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.399+0000 7fb34d147640 1 -- 172.21.15.156:0/627064124 wait complete. 2023-11-28T17:08:39.414 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 85dd8756-df1c-41fa-b542-d339b6e59925 6 2023-11-28T17:08:39.427 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:39.426+0000 7ff96c02e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T17:08:39.503 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:39.501+0000 7fb78c637040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T17:08:39.544 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:39.543+0000 7ff96c02e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T17:08:39.599 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.598+0000 7ff26c6cf640 1 Processor -- start 2023-11-28T17:08:39.599 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.598+0000 7ff26c6cf640 1 -- start start 2023-11-28T17:08:39.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.599+0000 7ff26c6cf640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff26415d860 0x7ff2640cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.599+0000 7ff26c6cf640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26415ddc0 con 0x7ff26415d480 2023-11-28T17:08:39.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.599+0000 7ff26c6cf640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26415e0a0 con 0x7ff2640d0010 2023-11-28T17:08:39.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.599+0000 7ff26c6cf640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2640d04a0 con 0x7ff26415d860 2023-11-28T17:08:39.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.600+0000 7ff269c6a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff26415d860 0x7ff2640cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.600+0000 7ff269c6a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff26415d860 0x7ff2640cfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:59084/0 (socket says 172.21.15.156:59084) 2023-11-28T17:08:39.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.600+0000 7ff269c6a640 1 -- 172.21.15.156:0/919841197 learned_addr learned my addr 172.21.15.156:0/919841197 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-28T17:08:39.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.600+0000 7ff26ac6c640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7ff2640d0010 0x7ff264160270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:46446/0 (socket says 172.21.15.156:46446) 2023-11-28T17:08:39.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.600+0000 7ff269c6a640 1 -- 172.21.15.156:0/919841197 >> v1:172.21.15.156:6789/0 conn(0x7ff2640d0010 legacy=0x7ff264160270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.602 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.600+0000 7ff269c6a640 1 -- 172.21.15.156:0/919841197 >> v1:172.21.15.116:6789/0 conn(0x7ff26415d480 legacy=0x7ff2640d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:39.602 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.600+0000 7ff269c6a640 1 -- 172.21.15.156:0/919841197 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26415e380 con 0x7ff26415d860 2023-11-28T17:08:39.602 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.601+0000 7ff269c6a640 1 --2- 172.21.15.156:0/919841197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff26415d860 0x7ff2640cfad0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff2580098a0 tx=0x7ff25802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a07bbfe6b6fb1def server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:39.602 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.601+0000 7ff269469640 1 -- 172.21.15.156:0/919841197 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258030840 con 0x7ff26415d860 2023-11-28T17:08:39.603 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.602+0000 7ff269469640 1 -- 172.21.15.156:0/919841197 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2580309a0 con 0x7ff26415d860 2023-11-28T17:08:39.603 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.602+0000 7ff269469640 1 -- 172.21.15.156:0/919841197 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258030d30 con 0x7ff26415d860 2023-11-28T17:08:39.603 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.602+0000 7ff26c6cf640 1 -- 172.21.15.156:0/919841197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff26415d860 msgr2=0x7ff2640cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.603 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.602+0000 7ff26c6cf640 1 --2- 172.21.15.156:0/919841197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff26415d860 0x7ff2640cfad0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff2580098a0 tx=0x7ff25802b980 comp rx=0 tx=0).stop 2023-11-28T17:08:39.604 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.602+0000 7ff26c6cf640 1 -- 172.21.15.156:0/919841197 shutdown_connections 2023-11-28T17:08:39.604 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.602+0000 7ff26c6cf640 1 --2- 172.21.15.156:0/919841197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff26415d860 0x7ff2640cfad0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.604 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.602+0000 7ff26c6cf640 1 -- 172.21.15.156:0/919841197 >> 172.21.15.156:0/919841197 conn(0x7ff26409c540 msgr2=0x7ff2640d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:39.604 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.602+0000 7ff26c6cf640 1 -- 172.21.15.156:0/919841197 shutdown_connections 2023-11-28T17:08:39.604 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.602+0000 7ff26c6cf640 1 -- 172.21.15.156:0/919841197 wait complete. 2023-11-28T17:08:39.605 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.604+0000 7ff26c6cf640 1 Processor -- start 2023-11-28T17:08:39.605 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.604+0000 7ff26c6cf640 1 -- start start 2023-11-28T17:08:39.608 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.605+0000 7ff26c6cf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2640d0010 0x7ff2640b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.608 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.605+0000 7ff26c6cf640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff26415d480 0x7ff26415bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.608 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.605+0000 7ff26c6cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff26415d860 0x7ff2640c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.608 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.605+0000 7ff26c6cf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640b2480 con 0x7ff26415d860 2023-11-28T17:08:39.608 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.605+0000 7ff26c6cf640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640aff60 con 0x7ff2640d0010 2023-11-28T17:08:39.608 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.605+0000 7ff26c6cf640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640b0240 con 0x7ff26415d480 2023-11-28T17:08:39.608 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.605+0000 7ff26a46b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2640d0010 0x7ff2640b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.608 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.605+0000 7ff269c6a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff26415d480 0x7ff26415bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.608 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.605+0000 7ff26a46b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2640d0010 0x7ff2640b1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:59100/0 (socket says 172.21.15.156:59100) 2023-11-28T17:08:39.608 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.605+0000 7ff269c6a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff26415d480 0x7ff26415bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:59464/0 (socket says 172.21.15.156:59464) 2023-11-28T17:08:39.608 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.605+0000 7ff26a46b640 1 -- 172.21.15.156:0/1537150527 learned_addr learned my addr 172.21.15.156:0/1537150527 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-28T17:08:39.609 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.606+0000 7ff26ac6c640 1 --2- 172.21.15.156:0/1537150527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff26415d860 0x7ff2640c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.609 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.606+0000 7ff26a46b640 1 -- 172.21.15.156:0/1537150527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff26415d480 msgr2=0x7ff26415bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.609 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.606+0000 7ff26a46b640 1 --2- 172.21.15.156:0/1537150527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff26415d480 0x7ff26415bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.609 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.606+0000 7ff26a46b640 1 -- 172.21.15.156:0/1537150527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff26415d860 msgr2=0x7ff2640c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.609 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.606+0000 7ff26a46b640 1 --2- 172.21.15.156:0/1537150527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff26415d860 0x7ff2640c9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.609 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.606+0000 7ff26a46b640 1 -- 172.21.15.156:0/1537150527 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2580094d0 con 0x7ff2640d0010 2023-11-28T17:08:39.609 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.606+0000 7ff269c6a640 1 --2- 172.21.15.156:0/1537150527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff26415d480 0x7ff26415bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:08:39.609 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.606+0000 7ff26a46b640 1 --2- 172.21.15.156:0/1537150527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2640d0010 0x7ff2640b1c50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff25400c7c0 tx=0x7ff25400cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:39.609 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.606+0000 7ff262ffd640 1 -- 172.21.15.156:0/1537150527 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25400aa50 con 0x7ff2640d0010 2023-11-28T17:08:39.609 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.606+0000 7ff262ffd640 1 -- 172.21.15.156:0/1537150527 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25400abb0 con 0x7ff2640d0010 2023-11-28T17:08:39.609 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.606+0000 7ff262ffd640 1 -- 172.21.15.156:0/1537150527 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff254012420 con 0x7ff2640d0010 2023-11-28T17:08:39.609 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.606+0000 7ff26c6cf640 1 -- 172.21.15.156:0/1537150527 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2640b0520 con 0x7ff2640d0010 2023-11-28T17:08:39.610 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.607+0000 7ff26c6cf640 1 -- 172.21.15.156:0/1537150527 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff26415c190 con 0x7ff2640d0010 2023-11-28T17:08:39.610 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.607+0000 7ff262ffd640 1 -- 172.21.15.156:0/1537150527 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff254012580 con 0x7ff2640d0010 2023-11-28T17:08:39.610 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.607+0000 7ff262ffd640 1 -- 172.21.15.156:0/1537150527 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7ff254012e20 con 0x7ff2640d0010 2023-11-28T17:08:39.610 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.608+0000 7ff26c6cf640 1 -- 172.21.15.156:0/1537150527 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff264067980 con 0x7ff2640d0010 2023-11-28T17:08:39.611 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.610+0000 7ff262ffd640 1 -- 172.21.15.156:0/1537150527 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff254016020 con 0x7ff2640d0010 2023-11-28T17:08:39.688 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.687+0000 7ff26c6cf640 1 -- 172.21.15.156:0/1537150527 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd new", "uuid": "85dd8756-df1c-41fa-b542-d339b6e59925", "id": 6} v 0) v1 -- 0x7ff264069360 con 0x7ff2640d0010 2023-11-28T17:08:39.694 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.693+0000 7ff262ffd640 1 -- 172.21.15.156:0/1537150527 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "85dd8756-df1c-41fa-b542-d339b6e59925", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff254012900 con 0x7ff2640d0010 2023-11-28T17:08:39.694 INFO:teuthology.orchestra.run.smithi156.stdout:6 2023-11-28T17:08:39.695 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.694+0000 7ff26c6cf640 1 -- 172.21.15.156:0/1537150527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2640d0010 msgr2=0x7ff2640b1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.696 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.694+0000 7ff26c6cf640 1 --2- 172.21.15.156:0/1537150527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2640d0010 0x7ff2640b1c50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff25400c7c0 tx=0x7ff25400cc80 comp rx=0 tx=0).stop 2023-11-28T17:08:39.696 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.694+0000 7ff26c6cf640 1 -- 172.21.15.156:0/1537150527 shutdown_connections 2023-11-28T17:08:39.696 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.694+0000 7ff26c6cf640 1 --2- 172.21.15.156:0/1537150527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff26415d860 0x7ff2640c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.696 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.694+0000 7ff26c6cf640 1 --2- 172.21.15.156:0/1537150527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff26415d480 0x7ff26415bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.696 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.694+0000 7ff26c6cf640 1 --2- 172.21.15.156:0/1537150527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2640d0010 0x7ff2640b1c50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.696 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.694+0000 7ff26c6cf640 1 -- 172.21.15.156:0/1537150527 >> 172.21.15.156:0/1537150527 conn(0x7ff26409c540 msgr2=0x7ff26406cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:39.696 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.695+0000 7ff26c6cf640 1 -- 172.21.15.156:0/1537150527 shutdown_connections 2023-11-28T17:08:39.696 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.695+0000 7ff26c6cf640 1 -- 172.21.15.156:0/1537150527 wait complete. 2023-11-28T17:08:39.709 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:39.708+0000 7ff96c02e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T17:08:39.712 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new cda8fbf8-de75-4da7-a71b-c6e90e3f568b 7 2023-11-28T17:08:39.761 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:39.760+0000 7ff96c02e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T17:08:39.924 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:39.923+0000 7ff96c02e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T17:08:39.951 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.950+0000 7fe6ed7db640 1 Processor -- start 2023-11-28T17:08:39.951 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.950+0000 7fe6ed7db640 1 -- start start 2023-11-28T17:08:39.953 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.952+0000 7fe6ed7db640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe6e8160b80 0x7fe6e80d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.953 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.952+0000 7fe6ed7db640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e80d0da0 con 0x7fe6e80b4010 2023-11-28T17:08:39.953 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.952+0000 7fe6ed7db640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e80d1080 con 0x7fe6e80d0700 2023-11-28T17:08:39.953 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.952+0000 7fe6ed7db640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e8160f60 con 0x7fe6e8160b80 2023-11-28T17:08:39.953 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.952+0000 7fe6ecfda640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fe6e80d0700 0x7fe6e80c6c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:46448/0 (socket says 172.21.15.156:46448) 2023-11-28T17:08:39.953 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.952+0000 7fe6ecfda640 1 -- 172.21.15.156:0/2141685178 learned_addr learned my addr 172.21.15.156:0/2141685178 (peer_addr_for_me v1:172.21.15.156:0/0) 2023-11-28T17:08:39.956 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e7fff640 1 --2- 172.21.15.156:0/2141685178 >> v2:172.21.15.116:3300/0 conn(0x7fe6e8160b80 0x7fe6e80d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.956 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 -- 172.21.15.156:0/2141685178 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202417059 0 0) 0x7fe6e80d1080 con 0x7fe6e80d0700 2023-11-28T17:08:39.956 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 -- 172.21.15.156:0/2141685178 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6d4003520 con 0x7fe6e80d0700 2023-11-28T17:08:39.956 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 -- 172.21.15.156:0/2141685178 <== mon.1 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2447304214 0 0) 0x7fe6d4003520 con 0x7fe6e80d0700 2023-11-28T17:08:39.956 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 -- 172.21.15.156:0/2141685178 >> v2:172.21.15.116:3300/0 conn(0x7fe6e8160b80 msgr2=0x7fe6e80d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:39.956 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 --2- 172.21.15.156:0/2141685178 >> v2:172.21.15.116:3300/0 conn(0x7fe6e8160b80 0x7fe6e80d1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.956 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 -- 172.21.15.156:0/2141685178 >> v1:172.21.15.116:6789/0 conn(0x7fe6e80b4010 legacy=0x7fe6e80d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.956 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 -- 172.21.15.156:0/2141685178 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e80ce780 con 0x7fe6e80d0700 2023-11-28T17:08:39.956 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 -- 172.21.15.156:0/2141685178 <== mon.1 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fe6e00019f0 con 0x7fe6e80d0700 2023-11-28T17:08:39.956 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 -- 172.21.15.156:0/2141685178 >> v1:172.21.15.156:6789/0 conn(0x7fe6e80d0700 legacy=0x7fe6e80c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.957 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 --2- 172.21.15.156:0/2141685178 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6d40039c0 0x7fe6d4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.957 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 --2- 172.21.15.156:0/2141685178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d40047c0 0x7fe6d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.957 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 --2- 172.21.15.156:0/2141685178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6d4005180 0x7fe6d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.957 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 -- 172.21.15.156:0/2141685178 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d400c050 con 0x7fe6d40047c0 2023-11-28T17:08:39.957 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 -- 172.21.15.156:0/2141685178 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d400c1c0 con 0x7fe6d4005180 2023-11-28T17:08:39.957 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.953+0000 7fe6e77fe640 1 -- 172.21.15.156:0/2141685178 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d400c380 con 0x7fe6d40039c0 2023-11-28T17:08:39.957 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.954+0000 7fe6ecfda640 1 --2- 172.21.15.156:0/2141685178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6d4005180 0x7fe6d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.957 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.954+0000 7fe6e7fff640 1 --2- 172.21.15.156:0/2141685178 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6d40039c0 0x7fe6d4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.957 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.954+0000 7fe6dffff640 1 --2- 172.21.15.156:0/2141685178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d40047c0 0x7fe6d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.957 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.954+0000 7fe6ecfda640 1 -- 172.21.15.156:0/2141685178 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6d40039c0 msgr2=0x7fe6d4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:39.957 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.954+0000 7fe6ecfda640 1 --2- 172.21.15.156:0/2141685178 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6d40039c0 0x7fe6d4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.957 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.954+0000 7fe6ecfda640 1 -- 172.21.15.156:0/2141685178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d40047c0 msgr2=0x7fe6d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:39.958 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.954+0000 7fe6ecfda640 1 --2- 172.21.15.156:0/2141685178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d40047c0 0x7fe6d4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.958 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.954+0000 7fe6ecfda640 1 -- 172.21.15.156:0/2141685178 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d4003520 con 0x7fe6d4005180 2023-11-28T17:08:39.958 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.954+0000 7fe6e7fff640 1 --2- 172.21.15.156:0/2141685178 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6d40039c0 0x7fe6d4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:08:39.958 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.955+0000 7fe6ecfda640 1 -- 172.21.15.156:0/2141685178 reap_dead start 2023-11-28T17:08:39.963 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.955+0000 7fe6ecfda640 1 --2- 172.21.15.156:0/2141685178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6d4005180 0x7fe6d400b980 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00013d0 tx=0x7fe6d40057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24b1831b7167a163 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:39.963 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.955+0000 7fe6e77fe640 1 -- 172.21.15.156:0/2141685178 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0034630 con 0x7fe6d4005180 2023-11-28T17:08:39.963 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.955+0000 7fe6e77fe640 1 -- 172.21.15.156:0/2141685178 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e002f070 con 0x7fe6d4005180 2023-11-28T17:08:39.963 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.955+0000 7fe6e77fe640 1 -- 172.21.15.156:0/2141685178 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0034a10 con 0x7fe6d4005180 2023-11-28T17:08:39.963 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.956+0000 7fe6ed7db640 1 -- 172.21.15.156:0/2141685178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6d4005180 msgr2=0x7fe6d400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.963 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.956+0000 7fe6ed7db640 1 --2- 172.21.15.156:0/2141685178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6d4005180 0x7fe6d400b980 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00013d0 tx=0x7fe6d40057f0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.963 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.956+0000 7fe6ed7db640 1 -- 172.21.15.156:0/2141685178 shutdown_connections 2023-11-28T17:08:39.964 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.956+0000 7fe6ed7db640 1 --2- 172.21.15.156:0/2141685178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6d4005180 0x7fe6d400b980 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.964 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.956+0000 7fe6ed7db640 1 -- 172.21.15.156:0/2141685178 >> 172.21.15.156:0/2141685178 conn(0x7fe6e80d2fc0 msgr2=0x7fe6e80d33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:39.964 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.956+0000 7fe6ed7db640 1 -- 172.21.15.156:0/2141685178 shutdown_connections 2023-11-28T17:08:39.964 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.956+0000 7fe6ed7db640 1 -- 172.21.15.156:0/2141685178 wait complete. 2023-11-28T17:08:39.964 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.958+0000 7fe6ed7db640 1 Processor -- start 2023-11-28T17:08:39.964 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.958+0000 7fe6ed7db640 1 -- start start 2023-11-28T17:08:39.964 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.959+0000 7fe6ed7db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d4005180 0x7fe6e8148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.964 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.959+0000 7fe6ed7db640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6e814c030 0x7fe6e80b83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.965 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.959+0000 7fe6ed7db640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6e80ccb90 0x7fe6e80cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:39.965 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.959+0000 7fe6ed7db640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e80cb850 con 0x7fe6d4005180 2023-11-28T17:08:39.965 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.959+0000 7fe6ed7db640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e80cbaf0 con 0x7fe6e814c030 2023-11-28T17:08:39.965 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.959+0000 7fe6ed7db640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e80cbdd0 con 0x7fe6e80ccb90 2023-11-28T17:08:39.965 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.959+0000 7fe6dffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6e814c030 0x7fe6e80b83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.965 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.959+0000 7fe6ecfda640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6e80ccb90 0x7fe6e80cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.965 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.959+0000 7fe6dffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6e814c030 0x7fe6e80b83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:59132/0 (socket says 172.21.15.156:59132) 2023-11-28T17:08:39.965 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.959+0000 7fe6dffff640 1 -- 172.21.15.156:0/922097613 learned_addr learned my addr 172.21.15.156:0/922097613 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-11-28T17:08:39.966 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.960+0000 7fe6e7fff640 1 --2- 172.21.15.156:0/922097613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d4005180 0x7fe6e8148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:39.966 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.960+0000 7fe6dffff640 1 -- 172.21.15.156:0/922097613 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6e80ccb90 msgr2=0x7fe6e80cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.966 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.960+0000 7fe6dffff640 1 --2- 172.21.15.156:0/922097613 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6e80ccb90 0x7fe6e80cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.966 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.960+0000 7fe6dffff640 1 -- 172.21.15.156:0/922097613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d4005180 msgr2=0x7fe6e8148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:39.966 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.960+0000 7fe6dffff640 1 --2- 172.21.15.156:0/922097613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d4005180 0x7fe6e8148d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:39.966 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.960+0000 7fe6dffff640 1 -- 172.21.15.156:0/922097613 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e80cc0b0 con 0x7fe6e814c030 2023-11-28T17:08:39.966 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.960+0000 7fe6ecfda640 1 --2- 172.21.15.156:0/922097613 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6e80ccb90 0x7fe6e80cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:08:39.967 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.960+0000 7fe6dffff640 1 --2- 172.21.15.156:0/922097613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6e814c030 0x7fe6e80b83c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000e4d0 tx=0x7fe6d000e990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:39.967 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.960+0000 7fe6e57fa640 1 -- 172.21.15.156:0/922097613 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d00052e0 con 0x7fe6e814c030 2023-11-28T17:08:39.967 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.960+0000 7fe6e57fa640 1 -- 172.21.15.156:0/922097613 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d0005440 con 0x7fe6e814c030 2023-11-28T17:08:39.967 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.960+0000 7fe6ed7db640 1 -- 172.21.15.156:0/922097613 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e80cc390 con 0x7fe6e814c030 2023-11-28T17:08:39.967 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.961+0000 7fe6e57fa640 1 -- 172.21.15.156:0/922097613 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0003810 con 0x7fe6e814c030 2023-11-28T17:08:39.967 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.961+0000 7fe6ed7db640 1 -- 172.21.15.156:0/922097613 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e80b8900 con 0x7fe6e814c030 2023-11-28T17:08:39.967 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.961+0000 7fe6e57fa640 1 -- 172.21.15.156:0/922097613 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe6d0003c70 con 0x7fe6e814c030 2023-11-28T17:08:39.967 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.961+0000 7fe6e57fa640 1 -- 172.21.15.156:0/922097613 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fe6d0013a80 con 0x7fe6e814c030 2023-11-28T17:08:39.968 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.962+0000 7fe6ed7db640 1 -- 172.21.15.156:0/922097613 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e8067980 con 0x7fe6e814c030 2023-11-28T17:08:39.968 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:39.964+0000 7fe6e57fa640 1 -- 172.21.15.156:0/922097613 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fe6d0013d60 con 0x7fe6e814c030 2023-11-28T17:08:40.043 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:40.042+0000 7fe6ed7db640 1 -- 172.21.15.156:0/922097613 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cda8fbf8-de75-4da7-a71b-c6e90e3f568b", "id": 7} v 0) v1 -- 0x7fe6e80b8a90 con 0x7fe6e814c030 2023-11-28T17:08:40.051 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:40.050+0000 7fe6e57fa640 1 -- 172.21.15.156:0/922097613 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cda8fbf8-de75-4da7-a71b-c6e90e3f568b", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe6d002c0b0 con 0x7fe6e814c030 2023-11-28T17:08:40.051 INFO:teuthology.orchestra.run.smithi156.stdout:7 2023-11-28T17:08:40.053 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:40.052+0000 7fe6ed7db640 1 -- 172.21.15.156:0/922097613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6e814c030 msgr2=0x7fe6e80b83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:40.053 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:40.052+0000 7fe6ed7db640 1 --2- 172.21.15.156:0/922097613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6e814c030 0x7fe6e80b83c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000e4d0 tx=0x7fe6d000e990 comp rx=0 tx=0).stop 2023-11-28T17:08:40.053 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:40.052+0000 7fe6ed7db640 1 -- 172.21.15.156:0/922097613 shutdown_connections 2023-11-28T17:08:40.053 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:40.052+0000 7fe6ed7db640 1 --2- 172.21.15.156:0/922097613 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6e80ccb90 0x7fe6e80cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:40.053 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:40.052+0000 7fe6ed7db640 1 --2- 172.21.15.156:0/922097613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6e814c030 0x7fe6e80b83c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:40.053 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:40.052+0000 7fe6ed7db640 1 --2- 172.21.15.156:0/922097613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6d4005180 0x7fe6e8148d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:40.053 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:40.052+0000 7fe6ed7db640 1 -- 172.21.15.156:0/922097613 >> 172.21.15.156:0/922097613 conn(0x7fe6e80d2fc0 msgr2=0x7fe6e80d33a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:40.054 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:40.052+0000 7fe6ed7db640 1 -- 172.21.15.156:0/922097613 shutdown_connections 2023-11-28T17:08:40.054 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:08:40.052+0000 7fe6ed7db640 1 -- 172.21.15.156:0/922097613 wait complete. 2023-11-28T17:08:40.070 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-28T17:08:40.070 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-11-28T17:08:40.076 INFO:tasks.ceph.osd.0:Started 2023-11-28T17:08:40.076 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-28T17:08:40.077 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-11-28T17:08:40.085 INFO:tasks.ceph.osd.1:Started 2023-11-28T17:08:40.085 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-28T17:08:40.085 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-11-28T17:08:40.090 INFO:tasks.ceph.osd.2:Started 2023-11-28T17:08:40.090 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-28T17:08:40.090 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-11-28T17:08:40.094 INFO:tasks.ceph.osd.3:Started 2023-11-28T17:08:40.094 INFO:tasks.ceph.osd.4:Restarting daemon 2023-11-28T17:08:40.095 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-11-28T17:08:40.102 INFO:tasks.ceph.osd.4:Started 2023-11-28T17:08:40.102 INFO:tasks.ceph.osd.5:Restarting daemon 2023-11-28T17:08:40.103 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-11-28T17:08:40.111 INFO:tasks.ceph.osd.5:Started 2023-11-28T17:08:40.111 INFO:tasks.ceph.osd.6:Restarting daemon 2023-11-28T17:08:40.111 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-11-28T17:08:40.116 INFO:tasks.ceph.osd.6:Started 2023-11-28T17:08:40.116 INFO:tasks.ceph.osd.7:Restarting daemon 2023-11-28T17:08:40.117 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-11-28T17:08:40.122 INFO:tasks.ceph.osd.7:Started 2023-11-28T17:08:40.123 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:08:40.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.345+0000 7fda25b16640 1 Processor -- start 2023-11-28T17:08:40.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.346+0000 7fda25b16640 1 -- start start 2023-11-28T17:08:40.348 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.347+0000 7fda25b16640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda2015d4d0 0x7fda200ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:40.348 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.347+0000 7fda25b16640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fda200d0e00 0x7fda200d1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.347+0000 7fda25b16640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda200d1c30 con 0x7fda200d17a0 2023-11-28T17:08:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.347+0000 7fda25b16640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda200cf710 con 0x7fda200d0e00 2023-11-28T17:08:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.347+0000 7fda25b16640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200cee70 con 0x7fda2015d4d0 2023-11-28T17:08:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.347+0000 7fda24b14640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda2015d4d0 0x7fda200ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.347+0000 7fda1ffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fda200d0e00 0x7fda200d1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.347+0000 7fda24b14640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda2015d4d0 0x7fda200ce840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:34432/0 (socket says 172.21.15.116:34432) 2023-11-28T17:08:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.347+0000 7fda24b14640 1 -- 172.21.15.116:0/3540453555 learned_addr learned my addr 172.21.15.116:0/3540453555 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.348+0000 7fda24b14640 1 -- 172.21.15.116:0/3540453555 >> v2:172.21.15.156:3300/0 conn(0x7fda200d0e00 msgr2=0x7fda200d1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.348+0000 7fda24b14640 1 --2- 172.21.15.116:0/3540453555 >> v2:172.21.15.156:3300/0 conn(0x7fda200d0e00 0x7fda200d1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.348+0000 7fda24b14640 1 -- 172.21.15.116:0/3540453555 >> v1:172.21.15.156:6789/0 conn(0x7fda200d17a0 legacy=0x7fda200cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.348+0000 7fda24b14640 1 -- 172.21.15.116:0/3540453555 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200cf150 con 0x7fda2015d4d0 2023-11-28T17:08:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.348+0000 7fda1ffff640 1 --2- 172.21.15.116:0/3540453555 >> v2:172.21.15.156:3300/0 conn(0x7fda200d0e00 0x7fda200d1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:08:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.349+0000 7fda24b14640 1 --2- 172.21.15.116:0/3540453555 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda2015d4d0 0x7fda200ce840 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fda140098a0 tx=0x7fda1402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6c7612b9c5ec5df server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.349+0000 7fda1f7fe640 1 -- 172.21.15.116:0/3540453555 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda14030840 con 0x7fda2015d4d0 2023-11-28T17:08:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.349+0000 7fda1f7fe640 1 -- 172.21.15.116:0/3540453555 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda140309a0 con 0x7fda2015d4d0 2023-11-28T17:08:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.349+0000 7fda1f7fe640 1 -- 172.21.15.116:0/3540453555 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda14030d30 con 0x7fda2015d4d0 2023-11-28T17:08:40.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.350+0000 7fda25b16640 1 -- 172.21.15.116:0/3540453555 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda2015d4d0 msgr2=0x7fda200ce840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:40.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.350+0000 7fda25b16640 1 --2- 172.21.15.116:0/3540453555 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda2015d4d0 0x7fda200ce840 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fda140098a0 tx=0x7fda1402b980 comp rx=0 tx=0).stop 2023-11-28T17:08:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.350+0000 7fda25b16640 1 -- 172.21.15.116:0/3540453555 shutdown_connections 2023-11-28T17:08:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.350+0000 7fda25b16640 1 --2- 172.21.15.116:0/3540453555 >> v2:172.21.15.156:3300/0 conn(0x7fda200d0e00 0x7fda200d1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.350+0000 7fda25b16640 1 --2- 172.21.15.116:0/3540453555 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda2015d4d0 0x7fda200ce840 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.350+0000 7fda25b16640 1 -- 172.21.15.116:0/3540453555 >> 172.21.15.116:0/3540453555 conn(0x7fda200d2f90 msgr2=0x7fda200d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.350+0000 7fda25b16640 1 -- 172.21.15.116:0/3540453555 shutdown_connections 2023-11-28T17:08:40.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.350+0000 7fda25b16640 1 -- 172.21.15.116:0/3540453555 wait complete. 2023-11-28T17:08:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.353+0000 7fda25b16640 1 Processor -- start 2023-11-28T17:08:40.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.353+0000 7fda25b16640 1 -- start start 2023-11-28T17:08:40.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.354+0000 7fda25b16640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda200d0e00 0x7fda201f2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:40.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.354+0000 7fda25b16640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda200d17a0 0x7fda20156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:40.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.354+0000 7fda25b16640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda2015d4d0 0x7fda20156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:40.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.354+0000 7fda25b16640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda200ca5e0 con 0x7fda200d17a0 2023-11-28T17:08:40.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.354+0000 7fda25b16640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda200cb120 con 0x7fda200d0e00 2023-11-28T17:08:40.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.354+0000 7fda25b16640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200ca7f0 con 0x7fda2015d4d0 2023-11-28T17:08:40.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.355+0000 7fda1ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda200d17a0 0x7fda20156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:40.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.355+0000 7fda24b14640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda200d0e00 0x7fda201f2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:40.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.355+0000 7fda24b14640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda200d0e00 0x7fda201f2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:36308/0 (socket says 172.21.15.116:36308) 2023-11-28T17:08:40.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.355+0000 7fda24b14640 1 -- 172.21.15.116:0/1783844347 learned_addr learned my addr 172.21.15.116:0/1783844347 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:40.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.355+0000 7fda1ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda200d17a0 0x7fda20156580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36356/0 (socket says 172.21.15.116:36356) 2023-11-28T17:08:40.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.355+0000 7fda25315640 1 --2- 172.21.15.116:0/1783844347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda2015d4d0 0x7fda20156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:40.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.355+0000 7fda1ffff640 1 -- 172.21.15.116:0/1783844347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda2015d4d0 msgr2=0x7fda20156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.355+0000 7fda1ffff640 1 --2- 172.21.15.116:0/1783844347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda2015d4d0 0x7fda20156ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.355+0000 7fda1ffff640 1 -- 172.21.15.116:0/1783844347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda200d0e00 msgr2=0x7fda201f2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.355+0000 7fda1ffff640 1 --2- 172.21.15.116:0/1783844347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda200d0e00 0x7fda201f2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.355+0000 7fda1ffff640 1 -- 172.21.15.116:0/1783844347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c009400 con 0x7fda200d17a0 2023-11-28T17:08:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.355+0000 7fda1ffff640 1 --2- 172.21.15.116:0/1783844347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda200d17a0 0x7fda20156580 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00e930 tx=0x7fda0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.355+0000 7fda1d7fa640 1 -- 172.21.15.116:0/1783844347 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c00c9a0 con 0x7fda200d17a0 2023-11-28T17:08:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.356+0000 7fda1d7fa640 1 -- 172.21.15.116:0/1783844347 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0c00cb00 con 0x7fda200d17a0 2023-11-28T17:08:40.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.356+0000 7fda1d7fa640 1 -- 172.21.15.116:0/1783844347 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c00ce90 con 0x7fda200d17a0 2023-11-28T17:08:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.356+0000 7fda25b16640 1 -- 172.21.15.116:0/1783844347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda140094d0 con 0x7fda200d17a0 2023-11-28T17:08:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.356+0000 7fda25b16640 1 -- 172.21.15.116:0/1783844347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda200cadb0 con 0x7fda200d17a0 2023-11-28T17:08:40.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.356+0000 7fda1d7fa640 1 -- 172.21.15.116:0/1783844347 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fda0c005c90 con 0x7fda200d17a0 2023-11-28T17:08:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.356+0000 7fda1d7fa640 1 -- 172.21.15.116:0/1783844347 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fda0c013470 con 0x7fda200d17a0 2023-11-28T17:08:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.357+0000 7fda25b16640 1 -- 172.21.15.116:0/1783844347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda20067c40 con 0x7fda200d17a0 2023-11-28T17:08:40.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.360+0000 7fda1d7fa640 1 -- 172.21.15.116:0/1783844347 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fda0c03bce0 con 0x7fda200d17a0 2023-11-28T17:08:40.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.423+0000 7fda25b16640 1 -- 172.21.15.116:0/1783844347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda200a0f60 con 0x7fda200d17a0 2023-11-28T17:08:40.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.424+0000 7fda1d7fa640 1 -- 172.21.15.116:0/1783844347 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fda0c02a070 con 0x7fda200d17a0 2023-11-28T17:08:40.426 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:08:40.426 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":10,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:08:40.044485+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:08:40.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.425+0000 7fda25b16640 1 -- 172.21.15.116:0/1783844347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda200d17a0 msgr2=0x7fda20156580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:40.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.425+0000 7fda25b16640 1 --2- 172.21.15.116:0/1783844347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda200d17a0 0x7fda20156580 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00e930 tx=0x7fda0c00edf0 comp rx=0 tx=0).stop 2023-11-28T17:08:40.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.425+0000 7fda25b16640 1 -- 172.21.15.116:0/1783844347 shutdown_connections 2023-11-28T17:08:40.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.425+0000 7fda25b16640 1 --2- 172.21.15.116:0/1783844347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda2015d4d0 0x7fda20156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:40.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.425+0000 7fda25b16640 1 --2- 172.21.15.116:0/1783844347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda200d17a0 0x7fda20156580 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:40.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.425+0000 7fda25b16640 1 --2- 172.21.15.116:0/1783844347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda200d0e00 0x7fda201f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:40.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.425+0000 7fda25b16640 1 -- 172.21.15.116:0/1783844347 >> 172.21.15.116:0/1783844347 conn(0x7fda200d2f90 msgr2=0x7fda200c8250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:40.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.425+0000 7fda25b16640 1 -- 172.21.15.116:0/1783844347 shutdown_connections 2023-11-28T17:08:40.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:40.425+0000 7fda25b16640 1 -- 172.21.15.116:0/1783844347 wait complete. 2023-11-28T17:08:40.442 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-28T17:08:40.442 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-28T17:08:40.531 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:40.529+0000 7fb78c637040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T17:08:40.787 INFO:tasks.ceph.osd.4.smithi156.stderr:2023-11-28T17:08:40.786+0000 7f9b4a79b500 -1 Falling back to public interface 2023-11-28T17:08:40.796 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-11-28T17:08:40.795+0000 7f4488e1c500 -1 Falling back to public interface 2023-11-28T17:08:40.798 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-11-28T17:08:40.797+0000 7f8d6e238500 -1 Falling back to public interface 2023-11-28T17:08:40.802 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-11-28T17:08:40.801+0000 7fa7dbab6500 -1 Falling back to public interface 2023-11-28T17:08:40.805 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-11-28T17:08:40.804+0000 7fa7daa5f640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6813/0: (98) Address already in use 2023-11-28T17:08:40.806 INFO:tasks.ceph.osd.5.smithi156.stderr:2023-11-28T17:08:40.805+0000 7fa6bce10500 -1 Falling back to public interface 2023-11-28T17:08:40.810 INFO:tasks.ceph.osd.7.smithi156.stderr:2023-11-28T17:08:40.809+0000 7fd1d5a0d500 -1 Falling back to public interface 2023-11-28T17:08:40.820 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-11-28T17:08:40.819+0000 7f6ac780f500 -1 Falling back to public interface 2023-11-28T17:08:40.841 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:40.840+0000 7fb78c637040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T17:08:40.847 INFO:tasks.ceph.osd.6.smithi156.stderr:2023-11-28T17:08:40.846+0000 7f94ffe14500 -1 Falling back to public interface 2023-11-28T17:08:40.901 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:40.900+0000 7fb78c637040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T17:08:41.002 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:41.001+0000 7ff96c02e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T17:08:41.015 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:41.014+0000 7fb78c637040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T17:08:41.248 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T17:08:41.314 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:41.312+0000 7ff96c02e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T17:08:41.374 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:41.373+0000 7ff96c02e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T17:08:41.416 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.415+0000 7fc72d189640 1 Processor -- start 2023-11-28T17:08:41.417 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.415+0000 7fc72d189640 1 -- start start 2023-11-28T17:08:41.418 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.416+0000 7fc72d189640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72813def0 0x7fc7280cdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:41.418 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.416+0000 7fc72d189640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc72813e2d0 0x7fc7280b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:41.418 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.416+0000 7fc72d189640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7280b3a30 con 0x7fc7280b35a0 2023-11-28T17:08:41.418 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.416+0000 7fc72d189640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72815b790 con 0x7fc72813e2d0 2023-11-28T17:08:41.418 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.416+0000 7fc72d189640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72815ba50 con 0x7fc72813def0 2023-11-28T17:08:41.418 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.417+0000 7fc726d9d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72813def0 0x7fc7280cdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:41.418 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.417+0000 7fc72659c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc72813e2d0 0x7fc7280b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:41.418 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.417+0000 7fc726d9d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72813def0 0x7fc7280cdcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:34526/0 (socket says 172.21.15.116:34526) 2023-11-28T17:08:41.419 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.417+0000 7fc726d9d640 1 -- 172.21.15.116:0/2640876150 learned_addr learned my addr 172.21.15.116:0/2640876150 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:41.419 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.417+0000 7fc726d9d640 1 -- 172.21.15.116:0/2640876150 >> v2:172.21.15.156:3300/0 conn(0x7fc72813e2d0 msgr2=0x7fc7280b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:41.419 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.417+0000 7fc726d9d640 1 --2- 172.21.15.116:0/2640876150 >> v2:172.21.15.156:3300/0 conn(0x7fc72813e2d0 0x7fc7280b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:41.419 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.417+0000 7fc726d9d640 1 -- 172.21.15.116:0/2640876150 >> v1:172.21.15.156:6789/0 conn(0x7fc7280b35a0 legacy=0x7fc7280d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:41.420 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.417+0000 7fc726d9d640 1 -- 172.21.15.116:0/2640876150 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72815bd30 con 0x7fc72813def0 2023-11-28T17:08:41.420 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.418+0000 7fc726d9d640 1 --2- 172.21.15.116:0/2640876150 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72813def0 0x7fc7280cdcc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc718009e50 tx=0x7fc71802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=237a215bc0601ea3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:41.421 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.418+0000 7fc725d9b640 1 -- 172.21.15.116:0/2640876150 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71802cb00 con 0x7fc72813def0 2023-11-28T17:08:41.421 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.419+0000 7fc725d9b640 1 -- 172.21.15.116:0/2640876150 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71802cc60 con 0x7fc72813def0 2023-11-28T17:08:41.421 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.419+0000 7fc725d9b640 1 -- 172.21.15.116:0/2640876150 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7180343f0 con 0x7fc72813def0 2023-11-28T17:08:41.421 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.419+0000 7fc72d189640 1 -- 172.21.15.116:0/2640876150 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72813def0 msgr2=0x7fc7280cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:41.421 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.419+0000 7fc72d189640 1 --2- 172.21.15.116:0/2640876150 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72813def0 0x7fc7280cdcc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc718009e50 tx=0x7fc71802bf60 comp rx=0 tx=0).stop 2023-11-28T17:08:41.421 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.419+0000 7fc72d189640 1 -- 172.21.15.116:0/2640876150 shutdown_connections 2023-11-28T17:08:41.421 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.419+0000 7fc72d189640 1 --2- 172.21.15.116:0/2640876150 >> v2:172.21.15.156:3300/0 conn(0x7fc72813e2d0 0x7fc7280b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:41.421 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.419+0000 7fc72d189640 1 --2- 172.21.15.116:0/2640876150 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72813def0 0x7fc7280cdcc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:41.421 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.419+0000 7fc72d189640 1 -- 172.21.15.116:0/2640876150 >> 172.21.15.116:0/2640876150 conn(0x7fc72806a6f0 msgr2=0x7fc72806aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:41.421 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.419+0000 7fc72d189640 1 -- 172.21.15.116:0/2640876150 shutdown_connections 2023-11-28T17:08:41.421 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.419+0000 7fc72d189640 1 -- 172.21.15.116:0/2640876150 wait complete. 2023-11-28T17:08:41.423 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.421+0000 7fc72d189640 1 Processor -- start 2023-11-28T17:08:41.423 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.421+0000 7fc72d189640 1 -- start start 2023-11-28T17:08:41.424 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.422+0000 7fc72d189640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7280b35a0 0x7fc72815a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:41.424 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.422+0000 7fc72d189640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72813def0 0x7fc7281f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:41.424 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.422+0000 7fc72d189640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc72813e2d0 0x7fc7281f1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:41.424 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.422+0000 7fc72d189640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7281f2690 con 0x7fc7280b35a0 2023-11-28T17:08:41.424 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.422+0000 7fc72d189640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72813e910 con 0x7fc72813e2d0 2023-11-28T17:08:41.424 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.422+0000 7fc72d189640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7281f1d50 con 0x7fc72813def0 2023-11-28T17:08:41.424 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.422+0000 7fc72659c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72813def0 0x7fc7281f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:41.424 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.422+0000 7fc726d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7280b35a0 0x7fc72815a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:41.424 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.423+0000 7fc726d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7280b35a0 0x7fc72815a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36426/0 (socket says 172.21.15.116:36426) 2023-11-28T17:08:41.424 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.423+0000 7fc72659c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72813def0 0x7fc7281f2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:34538/0 (socket says 172.21.15.116:34538) 2023-11-28T17:08:41.424 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.423+0000 7fc726d9d640 1 -- 172.21.15.116:0/1474716227 learned_addr learned my addr 172.21.15.116:0/1474716227 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:41.425 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.423+0000 7fc72759e640 1 --2- 172.21.15.116:0/1474716227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc72813e2d0 0x7fc7281f1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:41.426 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.423+0000 7fc72659c640 1 -- 172.21.15.116:0/1474716227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc72813e2d0 msgr2=0x7fc7281f1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:41.426 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.423+0000 7fc72659c640 1 --2- 172.21.15.116:0/1474716227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc72813e2d0 0x7fc7281f1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:41.426 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.423+0000 7fc72659c640 1 -- 172.21.15.116:0/1474716227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7280b35a0 msgr2=0x7fc72815a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:41.426 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.423+0000 7fc72659c640 1 --2- 172.21.15.116:0/1474716227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7280b35a0 0x7fc72815a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:41.426 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.423+0000 7fc72659c640 1 -- 172.21.15.116:0/1474716227 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc718009b30 con 0x7fc72813def0 2023-11-28T17:08:41.426 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.423+0000 7fc726d9d640 1 --2- 172.21.15.116:0/1474716227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7280b35a0 0x7fc72815a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:08:41.426 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.423+0000 7fc72659c640 1 --2- 172.21.15.116:0/1474716227 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72813def0 0x7fc7281f2150 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc71000e840 tx=0x7fc71000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:41.426 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.424+0000 7fc71f7fe640 1 -- 172.21.15.116:0/1474716227 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71000c730 con 0x7fc72813def0 2023-11-28T17:08:41.426 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.424+0000 7fc71f7fe640 1 -- 172.21.15.116:0/1474716227 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71000c890 con 0x7fc72813def0 2023-11-28T17:08:41.426 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.424+0000 7fc71f7fe640 1 -- 172.21.15.116:0/1474716227 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71000cc20 con 0x7fc72813def0 2023-11-28T17:08:41.427 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.424+0000 7fc72d189640 1 -- 172.21.15.116:0/1474716227 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72813ebd0 con 0x7fc72813def0 2023-11-28T17:08:41.427 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.424+0000 7fc72d189640 1 -- 172.21.15.116:0/1474716227 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72813f190 con 0x7fc72813def0 2023-11-28T17:08:41.427 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.425+0000 7fc72d189640 1 -- 172.21.15.116:0/1474716227 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72813f3b0 con 0x7fc72813def0 2023-11-28T17:08:41.427 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.425+0000 7fc71f7fe640 1 -- 172.21.15.116:0/1474716227 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 2) v1 ==== 44726+0+0 (secure 0 0 0) 0x7fc710013470 con 0x7fc72813def0 2023-11-28T17:08:41.427 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.426+0000 7fc71f7fe640 1 -- 172.21.15.116:0/1474716227 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc710049910 con 0x7fc72813def0 2023-11-28T17:08:41.429 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.428+0000 7fc71f7fe640 1 -- 172.21.15.116:0/1474716227 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fc710013720 con 0x7fc72813def0 2023-11-28T17:08:41.489 INFO:tasks.ceph.mgr.x.smithi156.stderr:2023-11-28T17:08:41.487+0000 7ff96c02e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T17:08:41.493 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.492+0000 7fc72d189640 1 -- 172.21.15.116:0/1474716227 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7281371e0 con 0x7fc72813def0 2023-11-28T17:08:41.495 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.494+0000 7fc71f7fe640 1 -- 172.21.15.116:0/1474716227 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fc71001b3b0 con 0x7fc72813def0 2023-11-28T17:08:41.495 INFO:teuthology.misc.health.smithi116.stdout: 2023-11-28T17:08:41.496 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":10,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:08:40.044485+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:08:41.497 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.495+0000 7fc72d189640 1 -- 172.21.15.116:0/1474716227 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72813def0 msgr2=0x7fc7281f2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:41.497 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.495+0000 7fc72d189640 1 --2- 172.21.15.116:0/1474716227 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72813def0 0x7fc7281f2150 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc71000e840 tx=0x7fc71000ed00 comp rx=0 tx=0).stop 2023-11-28T17:08:41.497 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.495+0000 7fc72d189640 1 -- 172.21.15.116:0/1474716227 shutdown_connections 2023-11-28T17:08:41.497 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.495+0000 7fc72d189640 1 --2- 172.21.15.116:0/1474716227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc72813e2d0 0x7fc7281f1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:41.497 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.495+0000 7fc72d189640 1 --2- 172.21.15.116:0/1474716227 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72813def0 0x7fc7281f2150 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:41.497 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.495+0000 7fc72d189640 1 --2- 172.21.15.116:0/1474716227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7280b35a0 0x7fc72815a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:41.497 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.495+0000 7fc72d189640 1 -- 172.21.15.116:0/1474716227 >> 172.21.15.116:0/1474716227 conn(0x7fc72806a6f0 msgr2=0x7fc7280b47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:41.497 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.495+0000 7fc72d189640 1 -- 172.21.15.116:0/1474716227 shutdown_connections 2023-11-28T17:08:41.497 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:41.495+0000 7fc72d189640 1 -- 172.21.15.116:0/1474716227 wait complete. 2023-11-28T17:08:41.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-28T17:08:42.058 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:42.057+0000 7fb72d9ff640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-28T17:08:42.059 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:08:42.057+0000 7fb72d9ff640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-28T17:08:44.295 INFO:tasks.ceph.osd.4.smithi156.stderr:2023-11-28T17:08:44.293+0000 7f9b4a79b500 -1 osd.4 0 log_to_monitors true 2023-11-28T17:08:44.324 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-11-28T17:08:44.323+0000 7f4488e1c500 -1 osd.0 0 log_to_monitors true 2023-11-28T17:08:44.361 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-11-28T17:08:44.359+0000 7f8d6e238500 -1 osd.2 0 log_to_monitors true 2023-11-28T17:08:44.372 INFO:tasks.ceph.osd.7.smithi156.stderr:2023-11-28T17:08:44.370+0000 7fd1d5a0d500 -1 osd.7 0 log_to_monitors true 2023-11-28T17:08:44.376 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-11-28T17:08:44.375+0000 7f6ac780f500 -1 osd.1 0 log_to_monitors true 2023-11-28T17:08:44.389 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-11-28T17:08:44.388+0000 7fa7dbab6500 -1 osd.3 0 log_to_monitors true 2023-11-28T17:08:44.399 INFO:tasks.ceph.osd.6.smithi156.stderr:2023-11-28T17:08:44.397+0000 7f94ffe14500 -1 osd.6 0 log_to_monitors true 2023-11-28T17:08:44.608 INFO:tasks.ceph.osd.5.smithi156.stderr:2023-11-28T17:08:44.606+0000 7fa6bce10500 -1 osd.5 0 log_to_monitors true 2023-11-28T17:08:47.967 INFO:tasks.ceph.osd.4.smithi156.stderr:2023-11-28T17:08:47.965+0000 7f9b45f06640 -1 osd.4 0 waiting for initial osdmap 2023-11-28T17:08:47.976 INFO:tasks.ceph.osd.4.smithi156.stderr:2023-11-28T17:08:47.974+0000 7f9b3e53f640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:08:48.005 INFO:tasks.ceph.osd.6.smithi156.stderr:2023-11-28T17:08:48.003+0000 7f94faf97640 -1 osd.6 0 waiting for initial osdmap 2023-11-28T17:08:48.011 INFO:tasks.ceph.osd.6.smithi156.stderr:2023-11-28T17:08:48.009+0000 7f94f35d0640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:08:48.019 INFO:tasks.ceph.osd.7.smithi156.stderr:2023-11-28T17:08:48.018+0000 7fd1d0bb9640 -1 osd.7 0 waiting for initial osdmap 2023-11-28T17:08:48.028 INFO:tasks.ceph.osd.7.smithi156.stderr:2023-11-28T17:08:48.026+0000 7fd1c91f2640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:08:48.040 INFO:tasks.ceph.osd.5.smithi156.stderr:2023-11-28T17:08:48.038+0000 7fa6b7ffe640 -1 osd.5 0 waiting for initial osdmap 2023-11-28T17:08:48.044 INFO:tasks.ceph.osd.5.smithi156.stderr:2023-11-28T17:08:48.043+0000 7fa6b0637640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:08:48.315 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T17:08:48.542 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-11-28T17:08:48.540+0000 7f4483eca640 -1 osd.0 0 waiting for initial osdmap 2023-11-28T17:08:48.550 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.548+0000 7f8a54bea640 1 Processor -- start 2023-11-28T17:08:48.550 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.549+0000 7f8a54bea640 1 -- start start 2023-11-28T17:08:48.552 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.551+0000 7f8a54bea640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8a500cd020 0x7f8a50071c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:48.552 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.551+0000 7f8a54bea640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a5015c3d0 0x7f8a5015ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:48.552 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.551+0000 7f8a54bea640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a500727a0 con 0x7f8a500b4200 2023-11-28T17:08:48.553 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.551+0000 7f8a54bea640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a5015b6e0 con 0x7f8a500cd020 2023-11-28T17:08:48.553 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.551+0000 7f8a54bea640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5015bf40 con 0x7f8a5015c3d0 2023-11-28T17:08:48.553 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.552+0000 7f8a4e59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8a500b4200 0x7f8a500b45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33028/0 (socket says 172.21.15.116:33028) 2023-11-28T17:08:48.553 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.551+0000 7f8a3ffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8a500cd020 0x7f8a50071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:48.554 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.552+0000 7f8a4e59c640 1 -- 172.21.15.116:0/89938949 learned_addr learned my addr 172.21.15.116:0/89938949 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:08:48.554 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.552+0000 7f8a4ed9d640 1 --2- 172.21.15.116:0/89938949 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a5015c3d0 0x7f8a5015ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:48.554 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f447c503640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:08:48.554 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.552+0000 7f8a4dd9b640 1 -- 172.21.15.116:0/89938949 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989822100 0 0) 0x7f8a500727a0 con 0x7f8a500b4200 2023-11-28T17:08:48.555 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.552+0000 7f8a4dd9b640 1 -- 172.21.15.116:0/89938949 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a30003520 con 0x7f8a500b4200 2023-11-28T17:08:48.555 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f8a4dd9b640 1 -- 172.21.15.116:0/89938949 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3767530332 0 0) 0x7f8a30003520 con 0x7f8a500b4200 2023-11-28T17:08:48.555 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f8a4dd9b640 1 -- 172.21.15.116:0/89938949 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a5015c3d0 msgr2=0x7f8a5015ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:48.556 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f8a4dd9b640 1 --2- 172.21.15.116:0/89938949 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a5015c3d0 0x7f8a5015ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:48.556 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f8a4dd9b640 1 -- 172.21.15.116:0/89938949 >> v2:172.21.15.156:3300/0 conn(0x7f8a500cd020 msgr2=0x7f8a50071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:48.556 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f8a4dd9b640 1 --2- 172.21.15.116:0/89938949 >> v2:172.21.15.156:3300/0 conn(0x7f8a500cd020 0x7f8a50071c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:48.556 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f8a4dd9b640 1 -- 172.21.15.116:0/89938949 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a500724c0 con 0x7f8a500b4200 2023-11-28T17:08:48.556 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f8a4dd9b640 1 -- 172.21.15.116:0/89938949 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f8a400019f0 con 0x7f8a500b4200 2023-11-28T17:08:48.559 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f8a4dd9b640 1 -- 172.21.15.116:0/89938949 >> v1:172.21.15.116:6789/0 conn(0x7f8a500b4200 legacy=0x7f8a500b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:48.559 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f8a4dd9b640 1 --2- 172.21.15.116:0/89938949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a30003970 0x7f8a30003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:48.559 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f8a4dd9b640 1 --2- 172.21.15.116:0/89938949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a300047c0 0x7f8a30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:48.559 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f8a4dd9b640 1 --2- 172.21.15.116:0/89938949 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a30005180 0x7f8a3000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:48.559 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f8a4dd9b640 1 -- 172.21.15.116:0/89938949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5015b6e0 con 0x7f8a300047c0 2023-11-28T17:08:48.559 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f8a4dd9b640 1 -- 172.21.15.116:0/89938949 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5015bf40 con 0x7f8a30003970 2023-11-28T17:08:48.559 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.553+0000 7f8a4dd9b640 1 -- 172.21.15.116:0/89938949 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3000c020 con 0x7f8a30005180 2023-11-28T17:08:48.559 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.554+0000 7f8a3ffff640 1 --2- 172.21.15.116:0/89938949 >> v2:172.21.15.156:3300/0 conn(0x7f8a500cd020 0x7f8a50071c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:08:48.559 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.554+0000 7f8a4e59c640 1 --2- 172.21.15.116:0/89938949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a30003970 0x7f8a30003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:48.560 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.554+0000 7f8a3ffff640 1 --2- 172.21.15.116:0/89938949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a300047c0 0x7f8a30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:48.560 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.554+0000 7f8a4e59c640 1 -- 172.21.15.116:0/89938949 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a30005180 msgr2=0x7f8a3000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:48.560 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.554+0000 7f8a4e59c640 1 --2- 172.21.15.116:0/89938949 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a30005180 0x7f8a3000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:48.560 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.554+0000 7f8a4e59c640 1 -- 172.21.15.116:0/89938949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a300047c0 msgr2=0x7f8a30004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:48.560 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.554+0000 7f8a4e59c640 1 --2- 172.21.15.116:0/89938949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a300047c0 0x7f8a30004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:48.560 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.554+0000 7f8a4e59c640 1 -- 172.21.15.116:0/89938949 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a30003520 con 0x7f8a30003970 2023-11-28T17:08:48.560 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.556+0000 7f8a4e59c640 1 --2- 172.21.15.116:0/89938949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a30003970 0x7f8a30003d90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000a370 tx=0x7f8a300057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=444876f6d1d3d364 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:48.560 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.556+0000 7f8a4dd9b640 1 -- 172.21.15.116:0/89938949 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a40035940 con 0x7f8a30003970 2023-11-28T17:08:48.560 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.556+0000 7f8a4dd9b640 1 -- 172.21.15.116:0/89938949 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a40035ac0 con 0x7f8a30003970 2023-11-28T17:08:48.561 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.556+0000 7f8a4dd9b640 1 -- 172.21.15.116:0/89938949 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a40035e70 con 0x7f8a30003970 2023-11-28T17:08:48.561 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.557+0000 7f8a54bea640 1 -- 172.21.15.116:0/89938949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a30003970 msgr2=0x7f8a30003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:48.561 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.557+0000 7f8a54bea640 1 --2- 172.21.15.116:0/89938949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a30003970 0x7f8a30003d90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000a370 tx=0x7f8a300057f0 comp rx=0 tx=0).stop 2023-11-28T17:08:48.561 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.557+0000 7f8a4ed9d640 1 -- 172.21.15.116:0/89938949 reap_dead start 2023-11-28T17:08:48.561 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.557+0000 7f8a54bea640 1 -- 172.21.15.116:0/89938949 shutdown_connections 2023-11-28T17:08:48.561 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.557+0000 7f8a54bea640 1 -- 172.21.15.116:0/89938949 >> 172.21.15.116:0/89938949 conn(0x7f8a5006c570 msgr2=0x7f8a5006c950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:48.561 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.557+0000 7f8a54bea640 1 -- 172.21.15.116:0/89938949 shutdown_connections 2023-11-28T17:08:48.561 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.558+0000 7f8a54bea640 1 -- 172.21.15.116:0/89938949 wait complete. 2023-11-28T17:08:48.562 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.561+0000 7f8a54bea640 1 Processor -- start 2023-11-28T17:08:48.562 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.561+0000 7f8a54bea640 1 -- start start 2023-11-28T17:08:48.564 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.563+0000 7f8a54bea640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a30003970 0x7f8a500b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:48.564 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.563+0000 7f8a54bea640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a5014f170 0x7f8a50150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:48.564 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.563+0000 7f8a54bea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a50147610 0x7f8a50147f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:48.564 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.563+0000 7f8a54bea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5015fe20 con 0x7f8a50147610 2023-11-28T17:08:48.564 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.563+0000 7f8a54bea640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a500af570 con 0x7f8a30003970 2023-11-28T17:08:48.564 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.563+0000 7f8a54bea640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5014f6a0 con 0x7f8a5014f170 2023-11-28T17:08:48.565 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.563+0000 7f8a4ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a50147610 0x7f8a50147f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:48.565 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.563+0000 7f8a4ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a50147610 0x7f8a50147f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41176/0 (socket says 172.21.15.116:41176) 2023-11-28T17:08:48.565 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.563+0000 7f8a4ed9d640 1 -- 172.21.15.116:0/4254110225 learned_addr learned my addr 172.21.15.116:0/4254110225 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:48.565 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.563+0000 7f8a4e59c640 1 --2- 172.21.15.116:0/4254110225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a30003970 0x7f8a500b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:48.565 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.564+0000 7f8a3ffff640 1 --2- 172.21.15.116:0/4254110225 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a5014f170 0x7f8a50150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:48.565 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.564+0000 7f8a4ed9d640 1 -- 172.21.15.116:0/4254110225 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a5014f170 msgr2=0x7f8a50150180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:48.565 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.564+0000 7f8a4ed9d640 1 --2- 172.21.15.116:0/4254110225 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a5014f170 0x7f8a50150180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:48.566 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.564+0000 7f8a4ed9d640 1 -- 172.21.15.116:0/4254110225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a30003970 msgr2=0x7f8a500b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:48.566 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.564+0000 7f8a4ed9d640 1 --2- 172.21.15.116:0/4254110225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a30003970 0x7f8a500b02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:48.566 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.564+0000 7f8a4ed9d640 1 -- 172.21.15.116:0/4254110225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a500af7c0 con 0x7f8a50147610 2023-11-28T17:08:48.566 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-11-28T17:08:48.564+0000 7f8d699a3640 -1 osd.2 0 waiting for initial osdmap 2023-11-28T17:08:48.566 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.565+0000 7f8a4ed9d640 1 --2- 172.21.15.116:0/4254110225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a50147610 0x7f8a50147f40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8a5015c7b0 tx=0x7f8a4800e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:48.567 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.566+0000 7f8a477fe640 1 -- 172.21.15.116:0/4254110225 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a48013680 con 0x7f8a50147610 2023-11-28T17:08:48.567 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.566+0000 7f8a477fe640 1 -- 172.21.15.116:0/4254110225 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a480137e0 con 0x7f8a50147610 2023-11-28T17:08:48.568 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.566+0000 7f8a477fe640 1 -- 172.21.15.116:0/4254110225 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a48013b90 con 0x7f8a50147610 2023-11-28T17:08:48.568 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.567+0000 7f8a54bea640 1 -- 172.21.15.116:0/4254110225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a500afaa0 con 0x7f8a50147610 2023-11-28T17:08:48.569 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.567+0000 7f8a54bea640 1 -- 172.21.15.116:0/4254110225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a501506c0 con 0x7f8a50147610 2023-11-28T17:08:48.571 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.569+0000 7f8a54bea640 1 -- 172.21.15.116:0/4254110225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a500affc0 con 0x7f8a50147610 2023-11-28T17:08:48.571 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.570+0000 7f8a477fe640 1 -- 172.21.15.116:0/4254110225 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) v1 ==== 89493+0+0 (secure 0 0 0) 0x7f8a48013cf0 con 0x7f8a50147610 2023-11-28T17:08:48.572 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.571+0000 7f8a477fe640 1 --2- 172.21.15.116:0/4254110225 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8a3406b290 0x7f8a3406d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:48.572 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.571+0000 7f8a477fe640 1 -- 172.21.15.116:0/4254110225 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f8a480585b0 con 0x7f8a50147610 2023-11-28T17:08:48.573 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.572+0000 7f8a4e59c640 1 --2- 172.21.15.116:0/4254110225 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8a3406b290 0x7f8a3406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:48.574 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-11-28T17:08:48.573+0000 7f8d61fdc640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:08:48.575 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.574+0000 7f8a4e59c640 1 --2- 172.21.15.116:0/4254110225 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8a3406b290 0x7f8a3406d750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8a4003d000 tx=0x7f8a40007000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:48.582 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.581+0000 7f8a477fe640 1 -- 172.21.15.116:0/4254110225 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a48054980 con 0x7f8a50147610 2023-11-28T17:08:48.597 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-11-28T17:08:48.596+0000 7f6ac29bb640 -1 osd.1 0 waiting for initial osdmap 2023-11-28T17:08:48.606 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-11-28T17:08:48.605+0000 7f6abaff4640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:08:48.676 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-11-28T17:08:48.674+0000 7fa7d7223640 -1 osd.3 0 waiting for initial osdmap 2023-11-28T17:08:48.683 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-11-28T17:08:48.682+0000 7fa7cf85c640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T17:08:48.774 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.773+0000 7f8a54bea640 1 -- 172.21.15.116:0/4254110225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a500affc0 con 0x7f8a50147610 2023-11-28T17:08:48.775 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.774+0000 7f8a477fe640 1 -- 172.21.15.116:0/4254110225 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f8a500affc0 con 0x7f8a50147610 2023-11-28T17:08:48.776 INFO:teuthology.misc.health.smithi116.stdout: 2023-11-28T17:08:48.776 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":12,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:08:46.256885+0000","last_up_change":"0.000000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:08:48.778 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.776+0000 7f8a54bea640 1 -- 172.21.15.116:0/4254110225 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8a3406b290 msgr2=0x7f8a3406d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:48.778 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.776+0000 7f8a54bea640 1 --2- 172.21.15.116:0/4254110225 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8a3406b290 0x7f8a3406d750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8a4003d000 tx=0x7f8a40007000 comp rx=0 tx=0).stop 2023-11-28T17:08:48.778 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.776+0000 7f8a54bea640 1 -- 172.21.15.116:0/4254110225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a50147610 msgr2=0x7f8a50147f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:48.778 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.776+0000 7f8a54bea640 1 --2- 172.21.15.116:0/4254110225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a50147610 0x7f8a50147f40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8a5015c7b0 tx=0x7f8a4800e9d0 comp rx=0 tx=0).stop 2023-11-28T17:08:48.778 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.776+0000 7f8a54bea640 1 -- 172.21.15.116:0/4254110225 shutdown_connections 2023-11-28T17:08:48.778 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.776+0000 7f8a54bea640 1 --2- 172.21.15.116:0/4254110225 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8a3406b290 0x7f8a3406d750 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:48.778 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.776+0000 7f8a54bea640 1 --2- 172.21.15.116:0/4254110225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a50147610 0x7f8a50147f40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:48.778 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.776+0000 7f8a54bea640 1 --2- 172.21.15.116:0/4254110225 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a5014f170 0x7f8a50150180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:48.778 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.776+0000 7f8a54bea640 1 --2- 172.21.15.116:0/4254110225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a30003970 0x7f8a500b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:48.779 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.776+0000 7f8a54bea640 1 -- 172.21.15.116:0/4254110225 >> 172.21.15.116:0/4254110225 conn(0x7f8a5006c570 msgr2=0x7f8a50159340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:48.779 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.777+0000 7f8a54bea640 1 -- 172.21.15.116:0/4254110225 shutdown_connections 2023-11-28T17:08:48.779 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:48.777+0000 7f8a54bea640 1 -- 172.21.15.116:0/4254110225 wait complete. 2023-11-28T17:08:48.791 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-28T17:08:52.247 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:08:52.255 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:08:55.598 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-28T17:08:55.780 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.779+0000 7f01d84ad640 1 Processor -- start 2023-11-28T17:08:55.780 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.779+0000 7f01d84ad640 1 -- start start 2023-11-28T17:08:55.782 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.780+0000 7f01d84ad640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f01d00cd020 0x7f01d0071c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:55.782 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.780+0000 7f01d84ad640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01d015c3d0 0x7f01d015ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:55.783 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.780+0000 7f01d84ad640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d00727a0 con 0x7f01d00b4200 2023-11-28T17:08:55.783 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.780+0000 7f01d84ad640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d015b6e0 con 0x7f01d00cd020 2023-11-28T17:08:55.783 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.780+0000 7f01d84ad640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d015bf40 con 0x7f01d015c3d0 2023-11-28T17:08:55.783 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.781+0000 7f01d5a48640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f01d00cd020 0x7f01d0071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:55.783 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.781+0000 7f01d5a48640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f01d00cd020 0x7f01d0071c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41210/0 (socket says 172.21.15.116:41210) 2023-11-28T17:08:55.783 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.781+0000 7f01d5a48640 1 -- 172.21.15.116:0/325480164 learned_addr learned my addr 172.21.15.116:0/325480164 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:55.783 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.781+0000 7f01d5a48640 1 -- 172.21.15.116:0/325480164 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01d015c3d0 msgr2=0x7f01d015ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:55.783 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.781+0000 7f01d5a48640 1 --2- 172.21.15.116:0/325480164 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01d015c3d0 0x7f01d015ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:55.783 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.781+0000 7f01d5a48640 1 -- 172.21.15.116:0/325480164 >> v1:172.21.15.156:6789/0 conn(0x7f01d00b4200 legacy=0x7f01d00b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:55.783 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.781+0000 7f01d5a48640 1 -- 172.21.15.116:0/325480164 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d00724c0 con 0x7f01d00cd020 2023-11-28T17:08:55.784 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.782+0000 7f01d5a48640 1 --2- 172.21.15.116:0/325480164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00cd020 0x7f01d0071c90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f01c0009870 tx=0x7f01c002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373f0984ba188b6a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:55.785 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.782+0000 7f01d5247640 1 -- 172.21.15.116:0/325480164 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c0031640 con 0x7f01d00cd020 2023-11-28T17:08:55.785 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.782+0000 7f01d5247640 1 -- 172.21.15.116:0/325480164 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c00317a0 con 0x7f01d00cd020 2023-11-28T17:08:55.785 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.782+0000 7f01d5247640 1 -- 172.21.15.116:0/325480164 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c0031b30 con 0x7f01d00cd020 2023-11-28T17:08:55.785 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.783+0000 7f01d84ad640 1 -- 172.21.15.116:0/325480164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00cd020 msgr2=0x7f01d0071c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:55.785 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.783+0000 7f01d84ad640 1 --2- 172.21.15.116:0/325480164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00cd020 0x7f01d0071c90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f01c0009870 tx=0x7f01c002b8f0 comp rx=0 tx=0).stop 2023-11-28T17:08:55.785 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.783+0000 7f01d84ad640 1 -- 172.21.15.116:0/325480164 shutdown_connections 2023-11-28T17:08:55.785 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.783+0000 7f01d84ad640 1 --2- 172.21.15.116:0/325480164 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01d015c3d0 0x7f01d015ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:55.785 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.783+0000 7f01d84ad640 1 --2- 172.21.15.116:0/325480164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00cd020 0x7f01d0071c90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:55.785 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.783+0000 7f01d84ad640 1 -- 172.21.15.116:0/325480164 >> 172.21.15.116:0/325480164 conn(0x7f01d006c570 msgr2=0x7f01d006c950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:55.785 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.783+0000 7f01d84ad640 1 -- 172.21.15.116:0/325480164 shutdown_connections 2023-11-28T17:08:55.785 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.783+0000 7f01d84ad640 1 -- 172.21.15.116:0/325480164 wait complete. 2023-11-28T17:08:55.786 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.785+0000 7f01d84ad640 1 Processor -- start 2023-11-28T17:08:55.786 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.785+0000 7f01d84ad640 1 -- start start 2023-11-28T17:08:55.788 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.786+0000 7f01d84ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00b4200 0x7f01d014c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:55.788 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.786+0000 7f01d84ad640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01d00cd020 0x7f01d0150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:55.788 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.786+0000 7f01d84ad640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01d015c3d0 0x7f01d01406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:55.788 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.786+0000 7f01d84ad640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d014d5a0 con 0x7f01d00b4200 2023-11-28T17:08:55.788 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.786+0000 7f01d84ad640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d014c990 con 0x7f01d00cd020 2023-11-28T17:08:55.788 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.786+0000 7f01d84ad640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d014cc70 con 0x7f01d015c3d0 2023-11-28T17:08:55.788 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.786+0000 7f01d6249640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00b4200 0x7f01d014c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:55.788 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.786+0000 7f01d6249640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00b4200 0x7f01d014c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41212/0 (socket says 172.21.15.116:41212) 2023-11-28T17:08:55.788 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.786+0000 7f01d6249640 1 -- 172.21.15.116:0/1355618460 learned_addr learned my addr 172.21.15.116:0/1355618460 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:55.788 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.786+0000 7f01d6249640 1 -- 172.21.15.116:0/1355618460 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01d015c3d0 msgr2=0x7f01d01406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:55.788 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.786+0000 7f01d6249640 1 --2- 172.21.15.116:0/1355618460 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01d015c3d0 0x7f01d01406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:55.789 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.786+0000 7f01d6249640 1 -- 172.21.15.116:0/1355618460 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01d00cd020 msgr2=0x7f01d0150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:55.789 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.786+0000 7f01d6249640 1 --2- 172.21.15.116:0/1355618460 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01d00cd020 0x7f01d0150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:55.789 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.786+0000 7f01d6249640 1 -- 172.21.15.116:0/1355618460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c00094d0 con 0x7f01d00b4200 2023-11-28T17:08:55.789 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.787+0000 7f01d6249640 1 --2- 172.21.15.116:0/1355618460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00b4200 0x7f01d014c3b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00b3e0 tx=0x7f01cc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:55.790 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.787+0000 7f01c6ffd640 1 -- 172.21.15.116:0/1355618460 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc011830 con 0x7f01d00b4200 2023-11-28T17:08:55.790 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.787+0000 7f01c6ffd640 1 -- 172.21.15.116:0/1355618460 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01cc0119b0 con 0x7f01d00b4200 2023-11-28T17:08:55.790 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.787+0000 7f01d84ad640 1 -- 172.21.15.116:0/1355618460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d014cf50 con 0x7f01d00b4200 2023-11-28T17:08:55.790 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.787+0000 7f01c6ffd640 1 -- 172.21.15.116:0/1355618460 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc011d60 con 0x7f01d00b4200 2023-11-28T17:08:55.790 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.787+0000 7f01d84ad640 1 -- 172.21.15.116:0/1355618460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d014f8f0 con 0x7f01d00b4200 2023-11-28T17:08:55.790 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.788+0000 7f01d84ad640 1 -- 172.21.15.116:0/1355618460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d0069440 con 0x7f01d00b4200 2023-11-28T17:08:55.791 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.789+0000 7f01c6ffd640 1 -- 172.21.15.116:0/1355618460 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f01cc01a400 con 0x7f01d00b4200 2023-11-28T17:08:55.792 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.791+0000 7f01c6ffd640 1 --2- 172.21.15.116:0/1355618460 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f01b406b2f0 0x7f01b406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:55.792 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.791+0000 7f01c6ffd640 1 -- 172.21.15.116:0/1355618460 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f01cc089a90 con 0x7f01d00b4200 2023-11-28T17:08:55.792 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.791+0000 7f01d5a48640 1 --2- 172.21.15.116:0/1355618460 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f01b406b2f0 0x7f01b406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:55.793 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.792+0000 7f01d5a48640 1 --2- 172.21.15.116:0/1355618460 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f01b406b2f0 0x7f01b406d7b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f01c0036000 tx=0x7f01c0030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:55.798 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.797+0000 7f01c6ffd640 1 -- 172.21.15.116:0/1355618460 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01cc054c30 con 0x7f01d00b4200 2023-11-28T17:08:55.989 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.988+0000 7f01d84ad640 1 -- 172.21.15.116:0/1355618460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01d00ae840 con 0x7f01d00b4200 2023-11-28T17:08:55.992 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.990+0000 7f01c6ffd640 1 -- 172.21.15.116:0/1355618460 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12862 (secure 0 0 0) 0x7f01cc015020 con 0x7f01d00b4200 2023-11-28T17:08:55.992 INFO:teuthology.misc.health.smithi116.stdout: 2023-11-28T17:08:55.992 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":16,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:08:51.982083+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:08:55.995 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.993+0000 7f01d84ad640 1 -- 172.21.15.116:0/1355618460 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f01b406b2f0 msgr2=0x7f01b406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:55.995 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.993+0000 7f01d84ad640 1 --2- 172.21.15.116:0/1355618460 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f01b406b2f0 0x7f01b406d7b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f01c0036000 tx=0x7f01c0030000 comp rx=0 tx=0).stop 2023-11-28T17:08:55.995 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.993+0000 7f01d84ad640 1 -- 172.21.15.116:0/1355618460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00b4200 msgr2=0x7f01d014c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:55.995 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.993+0000 7f01d84ad640 1 --2- 172.21.15.116:0/1355618460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00b4200 0x7f01d014c3b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00b3e0 tx=0x7f01cc00b8a0 comp rx=0 tx=0).stop 2023-11-28T17:08:55.995 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.993+0000 7f01d84ad640 1 -- 172.21.15.116:0/1355618460 shutdown_connections 2023-11-28T17:08:55.995 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.993+0000 7f01d84ad640 1 --2- 172.21.15.116:0/1355618460 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f01b406b2f0 0x7f01b406d7b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:55.995 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.993+0000 7f01d84ad640 1 --2- 172.21.15.116:0/1355618460 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01d015c3d0 0x7f01d01406b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:55.995 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.993+0000 7f01d84ad640 1 --2- 172.21.15.116:0/1355618460 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01d00cd020 0x7f01d0150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:55.995 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.993+0000 7f01d84ad640 1 --2- 172.21.15.116:0/1355618460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00b4200 0x7f01d014c3b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:55.996 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.993+0000 7f01d84ad640 1 -- 172.21.15.116:0/1355618460 >> 172.21.15.116:0/1355618460 conn(0x7f01d006c570 msgr2=0x7f01d0159d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:55.996 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.994+0000 7f01d84ad640 1 -- 172.21.15.116:0/1355618460 shutdown_connections 2023-11-28T17:08:55.996 INFO:teuthology.misc.health.smithi116.stderr:2023-11-28T17:08:55.994+0000 7f01d84ad640 1 -- 172.21.15.116:0/1355618460 wait complete. 2023-11-28T17:08:56.010 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-11-28T17:08:56.010 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-28T17:08:56.010 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T17:08:56.010 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T17:08:56.019 INFO:tasks.ceph.mds.a:Started 2023-11-28T17:08:56.019 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T17:08:56.019 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T17:08:56.028 INFO:tasks.ceph.mds.c:Started 2023-11-28T17:08:56.028 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T17:08:56.028 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T17:08:56.037 INFO:tasks.ceph.mds.b:Started 2023-11-28T17:08:56.037 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T17:08:56.037 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T17:08:56.046 INFO:tasks.ceph.mds.d:Started 2023-11-28T17:08:56.046 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-11-28T17:08:56.047 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:08:56.174 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2023-11-28T17:08:56.176 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2023-11-28T17:08:56.211 INFO:tasks.ceph.mds.b.smithi156.stdout:starting mds.b at 2023-11-28T17:08:56.211 INFO:tasks.ceph.mds.d.smithi156.stdout:starting mds.d at 2023-11-28T17:08:56.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.233+0000 7ff3d7c90640 1 Processor -- start 2023-11-28T17:08:56.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.233+0000 7ff3d7c90640 1 -- start start 2023-11-28T17:08:56.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.234+0000 7ff3d7c90640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff3d01544c0 0x7ff3d006cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:56.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.234+0000 7ff3d7c90640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3d006d250 0x7ff3d006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:56.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.234+0000 7ff3d7c90640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d006df90 con 0x7ff3d006ba10 2023-11-28T17:08:56.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.234+0000 7ff3d7c90640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d006bec0 con 0x7ff3d01544c0 2023-11-28T17:08:56.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.234+0000 7ff3d7c90640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d00ad5d0 con 0x7ff3d006d250 2023-11-28T17:08:56.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.234+0000 7ff3d622d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff3d006ba10 0x7ff3d0160f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33046/0 (socket says 172.21.15.116:33046) 2023-11-28T17:08:56.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.234+0000 7ff3d622d640 1 -- 172.21.15.116:0/686235375 learned_addr learned my addr 172.21.15.116:0/686235375 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:08:56.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.234+0000 7ff3d5a2c640 1 --2- 172.21.15.116:0/686235375 >> v2:172.21.15.156:3300/0 conn(0x7ff3d01544c0 0x7ff3d006cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:56.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.234+0000 7ff3d522b640 1 --2- 172.21.15.116:0/686235375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3d006d250 0x7ff3d006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:56.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.235+0000 7ff3d4a2a640 1 -- 172.21.15.116:0/686235375 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110453662 0 0) 0x7ff3d006df90 con 0x7ff3d006ba10 2023-11-28T17:08:56.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.235+0000 7ff3d5a2c640 1 -- 172.21.15.116:0/686235375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3d006d250 msgr2=0x7ff3d006b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.235+0000 7ff3d5a2c640 1 --2- 172.21.15.116:0/686235375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3d006d250 0x7ff3d006b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.235+0000 7ff3d5a2c640 1 -- 172.21.15.116:0/686235375 >> v1:172.21.15.116:6789/0 conn(0x7ff3d006ba10 legacy=0x7ff3d0160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.235+0000 7ff3d5a2c640 1 -- 172.21.15.116:0/686235375 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d01548a0 con 0x7ff3d01544c0 2023-11-28T17:08:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.236+0000 7ff3d5a2c640 1 --2- 172.21.15.116:0/686235375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3d01544c0 0x7ff3d006cd10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0097a0 tx=0x7ff3cc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46a97fdca96e1a51 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.236+0000 7ff3d4a2a640 1 -- 172.21.15.116:0/686235375 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc031830 con 0x7ff3d01544c0 2023-11-28T17:08:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.236+0000 7ff3d4a2a640 1 -- 172.21.15.116:0/686235375 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3cc031990 con 0x7ff3d01544c0 2023-11-28T17:08:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.236+0000 7ff3d4a2a640 1 -- 172.21.15.116:0/686235375 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc031d20 con 0x7ff3d01544c0 2023-11-28T17:08:56.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.237+0000 7ff3d7c90640 1 -- 172.21.15.116:0/686235375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3d01544c0 msgr2=0x7ff3d006cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:56.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.237+0000 7ff3d7c90640 1 --2- 172.21.15.116:0/686235375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3d01544c0 0x7ff3d006cd10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0097a0 tx=0x7ff3cc02b8d0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.237+0000 7ff3d7c90640 1 -- 172.21.15.116:0/686235375 shutdown_connections 2023-11-28T17:08:56.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.237+0000 7ff3d7c90640 1 --2- 172.21.15.116:0/686235375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3d006d250 0x7ff3d006b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.237+0000 7ff3d7c90640 1 --2- 172.21.15.116:0/686235375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3d01544c0 0x7ff3d006cd10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.237+0000 7ff3d7c90640 1 -- 172.21.15.116:0/686235375 >> 172.21.15.116:0/686235375 conn(0x7ff3d00b1b50 msgr2=0x7ff3d0070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:56.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.237+0000 7ff3d7c90640 1 -- 172.21.15.116:0/686235375 shutdown_connections 2023-11-28T17:08:56.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.237+0000 7ff3d7c90640 1 -- 172.21.15.116:0/686235375 wait complete. 2023-11-28T17:08:56.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.239+0000 7ff3d7c90640 1 Processor -- start 2023-11-28T17:08:56.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.239+0000 7ff3d7c90640 1 -- start start 2023-11-28T17:08:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.240+0000 7ff3d7c90640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3d006ba10 0x7ff3d014e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.240+0000 7ff3d7c90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3d006d250 0x7ff3d0152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.240+0000 7ff3d7c90640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3d01544c0 0x7ff3d014a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.240+0000 7ff3d7c90640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d01527f0 con 0x7ff3d006d250 2023-11-28T17:08:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.240+0000 7ff3d7c90640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d0155de0 con 0x7ff3d01544c0 2023-11-28T17:08:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.240+0000 7ff3d7c90640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d0151be0 con 0x7ff3d006ba10 2023-11-28T17:08:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.240+0000 7ff3d522b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3d006d250 0x7ff3d0152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.240+0000 7ff3d522b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3d006d250 0x7ff3d0152fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41244/0 (socket says 172.21.15.116:41244) 2023-11-28T17:08:56.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.240+0000 7ff3d522b640 1 -- 172.21.15.116:0/1826387262 learned_addr learned my addr 172.21.15.116:0/1826387262 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:56.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.240+0000 7ff3d5a2c640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3d006ba10 0x7ff3d014e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:56.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.240+0000 7ff3d622d640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3d01544c0 0x7ff3d014a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:56.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.241+0000 7ff3d522b640 1 -- 172.21.15.116:0/1826387262 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3d006ba10 msgr2=0x7ff3d014e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:56.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.241+0000 7ff3d522b640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3d006ba10 0x7ff3d014e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.241+0000 7ff3d522b640 1 -- 172.21.15.116:0/1826387262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3d01544c0 msgr2=0x7ff3d014a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:56.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.241+0000 7ff3d522b640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3d01544c0 0x7ff3d014a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.241+0000 7ff3d522b640 1 -- 172.21.15.116:0/1826387262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc009400 con 0x7ff3d006d250 2023-11-28T17:08:56.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.241+0000 7ff3d5a2c640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3d006ba10 0x7ff3d014e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:08:56.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.241+0000 7ff3d522b640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3d006d250 0x7ff3d0152fe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000e9e0 tx=0x7ff3c000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:56.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.241+0000 7ff3c67fc640 1 -- 172.21.15.116:0/1826387262 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0009670 con 0x7ff3d006d250 2023-11-28T17:08:56.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.241+0000 7ff3c67fc640 1 -- 172.21.15.116:0/1826387262 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c000c9d0 con 0x7ff3d006d250 2023-11-28T17:08:56.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.241+0000 7ff3c67fc640 1 -- 172.21.15.116:0/1826387262 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c000cd80 con 0x7ff3d006d250 2023-11-28T17:08:56.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.241+0000 7ff3d7c90640 1 -- 172.21.15.116:0/1826387262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d0151ec0 con 0x7ff3d006d250 2023-11-28T17:08:56.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.241+0000 7ff3d7c90640 1 -- 172.21.15.116:0/1826387262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d0152480 con 0x7ff3d006d250 2023-11-28T17:08:56.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.242+0000 7ff3d7c90640 1 -- 172.21.15.116:0/1826387262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d00693d0 con 0x7ff3d006d250 2023-11-28T17:08:56.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.243+0000 7ff3c67fc640 1 -- 172.21.15.116:0/1826387262 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff3c0013470 con 0x7ff3d006d250 2023-11-28T17:08:56.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.244+0000 7ff3c67fc640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff3b406b2a0 0x7ff3b406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:56.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.244+0000 7ff3c67fc640 1 -- 172.21.15.116:0/1826387262 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff3c0089b00 con 0x7ff3d006d250 2023-11-28T17:08:56.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.244+0000 7ff3d5a2c640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff3b406b2a0 0x7ff3b406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:56.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.245+0000 7ff3d5a2c640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff3b406b2a0 0x7ff3b406d760 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc036000 tx=0x7ff3cc008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:56.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.249+0000 7ff3c67fc640 1 -- 172.21.15.116:0/1826387262 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3c008be60 con 0x7ff3d006d250 2023-11-28T17:08:56.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.433+0000 7ff3d7c90640 1 -- 172.21.15.116:0/1826387262 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3d01529a0 con 0x7ff3d006d250 2023-11-28T17:08:56.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.435+0000 7ff3c67fc640 1 -- 172.21.15.116:0/1826387262 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12862 (secure 0 0 0) 0x7ff3c0017090 con 0x7ff3d006d250 2023-11-28T17:08:56.437 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:08:56.437 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:08:51.982083+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:08:56.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.437+0000 7ff3d7c90640 1 -- 172.21.15.116:0/1826387262 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff3b406b2a0 msgr2=0x7ff3b406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:56.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.437+0000 7ff3d7c90640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff3b406b2a0 0x7ff3b406d760 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc036000 tx=0x7ff3cc008030 comp rx=0 tx=0).stop 2023-11-28T17:08:56.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.437+0000 7ff3d7c90640 1 -- 172.21.15.116:0/1826387262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3d006d250 msgr2=0x7ff3d0152fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:56.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.437+0000 7ff3d7c90640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3d006d250 0x7ff3d0152fe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000e9e0 tx=0x7ff3c000eea0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.437+0000 7ff3d7c90640 1 -- 172.21.15.116:0/1826387262 shutdown_connections 2023-11-28T17:08:56.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.437+0000 7ff3d7c90640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff3b406b2a0 0x7ff3b406d760 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.437+0000 7ff3d7c90640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3d01544c0 0x7ff3d014a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.437+0000 7ff3d7c90640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3d006d250 0x7ff3d0152fe0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.437+0000 7ff3d7c90640 1 --2- 172.21.15.116:0/1826387262 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3d006ba10 0x7ff3d014e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.437+0000 7ff3d7c90640 1 -- 172.21.15.116:0/1826387262 >> 172.21.15.116:0/1826387262 conn(0x7ff3d00b1b50 msgr2=0x7ff3d015f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.438+0000 7ff3d7c90640 1 -- 172.21.15.116:0/1826387262 shutdown_connections 2023-11-28T17:08:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.438+0000 7ff3d7c90640 1 -- 172.21.15.116:0/1826387262 wait complete. 2023-11-28T17:08:56.455 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:08:56.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:08:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.666+0000 7f339d236640 1 Processor -- start 2023-11-28T17:08:56.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.667+0000 7f339d236640 1 -- start start 2023-11-28T17:08:56.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.668+0000 7f339d236640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f33980b20a0 0x7f33980d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.668+0000 7f339d236640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f33980b2ad0 0x7f33980b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.668+0000 7f339d236640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33980cdcb0 con 0x7f33980d1780 2023-11-28T17:08:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.668+0000 7f339d236640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33980d39e0 con 0x7f33980b2ad0 2023-11-28T17:08:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.668+0000 7f339d236640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33980d3cc0 con 0x7f33980b20a0 2023-11-28T17:08:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.668+0000 7f3396d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f33980d1780 0x7f33980d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33052/0 (socket says 172.21.15.116:33052) 2023-11-28T17:08:56.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.668+0000 7f3396d9d640 1 -- 172.21.15.116:0/3408657207 learned_addr learned my addr 172.21.15.116:0/3408657207 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:08:56.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.668+0000 7f339759e640 1 --2- 172.21.15.116:0/3408657207 >> v2:172.21.15.116:3300/0 conn(0x7f33980b2ad0 0x7f33980b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.669+0000 7f339659c640 1 --2- 172.21.15.116:0/3408657207 >> v2:172.21.15.156:3300/0 conn(0x7f33980b20a0 0x7f33980d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.669+0000 7f3395d9b640 1 -- 172.21.15.116:0/3408657207 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352902404 0 0) 0x7f33980cdcb0 con 0x7f33980d1780 2023-11-28T17:08:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.669+0000 7f339759e640 1 -- 172.21.15.116:0/3408657207 >> v2:172.21.15.156:3300/0 conn(0x7f33980b20a0 msgr2=0x7f33980d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.669+0000 7f339759e640 1 --2- 172.21.15.116:0/3408657207 >> v2:172.21.15.156:3300/0 conn(0x7f33980b20a0 0x7f33980d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.669+0000 7f339759e640 1 -- 172.21.15.116:0/3408657207 >> v1:172.21.15.116:6789/0 conn(0x7f33980d1780 legacy=0x7f33980d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.669+0000 7f339759e640 1 -- 172.21.15.116:0/3408657207 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33980cdf50 con 0x7f33980b2ad0 2023-11-28T17:08:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.670+0000 7f339759e640 1 --2- 172.21.15.116:0/3408657207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33980b2ad0 0x7f33980b2f00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3398162190 tx=0x7f339002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f20aaf356feeda server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:56.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.670+0000 7f3395d9b640 1 -- 172.21.15.116:0/3408657207 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339002e070 con 0x7f33980b2ad0 2023-11-28T17:08:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.670+0000 7f3395d9b640 1 -- 172.21.15.116:0/3408657207 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33900315a0 con 0x7f33980b2ad0 2023-11-28T17:08:56.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.670+0000 7f3395d9b640 1 -- 172.21.15.116:0/3408657207 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3390031930 con 0x7f33980b2ad0 2023-11-28T17:08:56.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.671+0000 7f339d236640 1 -- 172.21.15.116:0/3408657207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33980b2ad0 msgr2=0x7f33980b2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:56.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.671+0000 7f339d236640 1 --2- 172.21.15.116:0/3408657207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33980b2ad0 0x7f33980b2f00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3398162190 tx=0x7f339002b840 comp rx=0 tx=0).stop 2023-11-28T17:08:56.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.671+0000 7f339d236640 1 -- 172.21.15.116:0/3408657207 shutdown_connections 2023-11-28T17:08:56.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.671+0000 7f339d236640 1 --2- 172.21.15.116:0/3408657207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33980b2ad0 0x7f33980b2f00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.671+0000 7f339d236640 1 --2- 172.21.15.116:0/3408657207 >> v2:172.21.15.156:3300/0 conn(0x7f33980b20a0 0x7f33980d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.671+0000 7f339d236640 1 -- 172.21.15.116:0/3408657207 >> 172.21.15.116:0/3408657207 conn(0x7f339809c4e0 msgr2=0x7f33980d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:56.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.671+0000 7f339d236640 1 -- 172.21.15.116:0/3408657207 shutdown_connections 2023-11-28T17:08:56.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.671+0000 7f339d236640 1 -- 172.21.15.116:0/3408657207 wait complete. 2023-11-28T17:08:56.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.673+0000 7f339d236640 1 Processor -- start 2023-11-28T17:08:56.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.673+0000 7f339d236640 1 -- start start 2023-11-28T17:08:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.674+0000 7f339d236640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33980b20a0 0x7f339815e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.674+0000 7f339d236640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33980b2ad0 0x7f3398155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.674+0000 7f339d236640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33980d1780 0x7f33980d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:56.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.674+0000 7f339d236640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339815f8d0 con 0x7f33980b2ad0 2023-11-28T17:08:56.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.674+0000 7f339d236640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339815ecc0 con 0x7f33980b20a0 2023-11-28T17:08:56.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.674+0000 7f339d236640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339815efa0 con 0x7f33980d1780 2023-11-28T17:08:56.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.674+0000 7f339659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33980b2ad0 0x7f3398155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:56.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.674+0000 7f339659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33980b2ad0 0x7f3398155740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41262/0 (socket says 172.21.15.116:41262) 2023-11-28T17:08:56.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.674+0000 7f339659c640 1 -- 172.21.15.116:0/3466092601 learned_addr learned my addr 172.21.15.116:0/3466092601 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:56.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.674+0000 7f3396d9d640 1 --2- 172.21.15.116:0/3466092601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33980b20a0 0x7f339815e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:56.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.674+0000 7f339659c640 1 -- 172.21.15.116:0/3466092601 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33980d1780 msgr2=0x7f33980d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.674+0000 7f339659c640 1 --2- 172.21.15.116:0/3466092601 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33980d1780 0x7f33980d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.675+0000 7f339659c640 1 -- 172.21.15.116:0/3466092601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33980b20a0 msgr2=0x7f339815e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.675+0000 7f339659c640 1 --2- 172.21.15.116:0/3466092601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33980b20a0 0x7f339815e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.675+0000 7f339659c640 1 -- 172.21.15.116:0/3466092601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3390009400 con 0x7f33980b2ad0 2023-11-28T17:08:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.675+0000 7f3396d9d640 1 --2- 172.21.15.116:0/3466092601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33980b20a0 0x7f339815e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:08:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.675+0000 7f339659c640 1 --2- 172.21.15.116:0/3466092601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33980b2ad0 0x7f3398155740 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f338000d7d0 tx=0x7f338000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:56.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.675+0000 7f338f7fe640 1 -- 172.21.15.116:0/3466092601 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3380018070 con 0x7f33980b2ad0 2023-11-28T17:08:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.675+0000 7f338f7fe640 1 -- 172.21.15.116:0/3466092601 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3380004020 con 0x7f33980b2ad0 2023-11-28T17:08:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.675+0000 7f338f7fe640 1 -- 172.21.15.116:0/3466092601 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33800041b0 con 0x7f33980b2ad0 2023-11-28T17:08:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.675+0000 7f339d236640 1 -- 172.21.15.116:0/3466092601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339815f280 con 0x7f33980b2ad0 2023-11-28T17:08:56.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.675+0000 7f339d236640 1 -- 172.21.15.116:0/3466092601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f339815e9d0 con 0x7f33980b2ad0 2023-11-28T17:08:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.676+0000 7f339d236640 1 -- 172.21.15.116:0/3466092601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339815fa40 con 0x7f33980b2ad0 2023-11-28T17:08:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.677+0000 7f338f7fe640 1 -- 172.21.15.116:0/3466092601 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3380004310 con 0x7f33980b2ad0 2023-11-28T17:08:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.678+0000 7f338f7fe640 1 --2- 172.21.15.116:0/3466092601 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f337806b2a0 0x7f337806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.678+0000 7f338f7fe640 1 -- 172.21.15.116:0/3466092601 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f33800897a0 con 0x7f33980b2ad0 2023-11-28T17:08:56.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.679+0000 7f3396d9d640 1 --2- 172.21.15.116:0/3466092601 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f337806b2a0 0x7f337806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:56.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.679+0000 7f3396d9d640 1 --2- 172.21.15.116:0/3466092601 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f337806b2a0 0x7f337806d760 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3388007fd0 tx=0x7f3388009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.683+0000 7f338f7fe640 1 -- 172.21.15.116:0/3466092601 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3380054950 con 0x7f33980b2ad0 2023-11-28T17:08:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.872+0000 7f339d236640 1 -- 172.21.15.116:0/3466092601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3398069360 con 0x7f33980b2ad0 2023-11-28T17:08:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.873+0000 7f338f7fe640 1 -- 172.21.15.116:0/3466092601 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3380057f70 con 0x7f33980b2ad0 2023-11-28T17:08:56.875 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:08:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.876+0000 7f339d236640 1 -- 172.21.15.116:0/3466092601 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f337806b2a0 msgr2=0x7f337806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.876+0000 7f339d236640 1 --2- 172.21.15.116:0/3466092601 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f337806b2a0 0x7f337806d760 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3388007fd0 tx=0x7f3388009040 comp rx=0 tx=0).stop 2023-11-28T17:08:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.876+0000 7f339d236640 1 -- 172.21.15.116:0/3466092601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33980b2ad0 msgr2=0x7f3398155740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.876+0000 7f339d236640 1 --2- 172.21.15.116:0/3466092601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33980b2ad0 0x7f3398155740 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f338000d7d0 tx=0x7f338000dc90 comp rx=0 tx=0).stop 2023-11-28T17:08:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.876+0000 7f339d236640 1 -- 172.21.15.116:0/3466092601 shutdown_connections 2023-11-28T17:08:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.876+0000 7f339d236640 1 --2- 172.21.15.116:0/3466092601 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f337806b2a0 0x7f337806d760 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.876+0000 7f339d236640 1 --2- 172.21.15.116:0/3466092601 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33980d1780 0x7f33980d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.876+0000 7f339d236640 1 --2- 172.21.15.116:0/3466092601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f33980b2ad0 0x7f3398155740 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.876+0000 7f339d236640 1 --2- 172.21.15.116:0/3466092601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33980b20a0 0x7f339815e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.876+0000 7f339d236640 1 -- 172.21.15.116:0/3466092601 >> 172.21.15.116:0/3466092601 conn(0x7f339809c4e0 msgr2=0x7f33980cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.876+0000 7f339d236640 1 -- 172.21.15.116:0/3466092601 shutdown_connections 2023-11-28T17:08:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:56.876+0000 7f339d236640 1 -- 172.21.15.116:0/3466092601 wait complete. 2023-11-28T17:08:56.895 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:08:57.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.105+0000 7f7e48e2d640 1 Processor -- start 2023-11-28T17:08:57.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.105+0000 7f7e48e2d640 1 -- start start 2023-11-28T17:08:57.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.106+0000 7f7e48e2d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7e4406c960 0x7f7e44162490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:57.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.106+0000 7f7e48e2d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e440b29c0 con 0x7f7e440b3d20 2023-11-28T17:08:57.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.106+0000 7f7e48e2d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e44162af0 con 0x7f7e440b3940 2023-11-28T17:08:57.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.106+0000 7f7e48e2d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e440b2090 con 0x7f7e4406c960 2023-11-28T17:08:57.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.107+0000 7f7e42d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7e4406c960 0x7f7e44162490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:57.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.107+0000 7f7e42d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7e4406c960 0x7f7e44162490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41266/0 (socket says 172.21.15.116:41266) 2023-11-28T17:08:57.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.107+0000 7f7e3bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7e440b3d20 0x7f7e44161d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33056/0 (socket says 172.21.15.116:33056) 2023-11-28T17:08:57.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.107+0000 7f7e42d9d640 1 -- 172.21.15.116:0/2691196680 learned_addr learned my addr 172.21.15.116:0/2691196680 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:57.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.107+0000 7f7e42d9d640 1 -- 172.21.15.116:0/2691196680 >> v1:172.21.15.156:6789/0 conn(0x7f7e440b3940 legacy=0x7f7e4406c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:57.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.107+0000 7f7e42d9d640 1 -- 172.21.15.116:0/2691196680 >> v1:172.21.15.116:6789/0 conn(0x7f7e440b3d20 legacy=0x7f7e44161d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:57.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.107+0000 7f7e42d9d640 1 -- 172.21.15.116:0/2691196680 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e440b2370 con 0x7f7e4406c960 2023-11-28T17:08:57.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.107+0000 7f7e41d9b640 1 -- 172.21.15.116:0/2691196680 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630286999 0 0) 0x7f7e440b29c0 con 0x7f7e440b3d20 2023-11-28T17:08:57.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.108+0000 7f7e42d9d640 1 --2- 172.21.15.116:0/2691196680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e4406c960 0x7f7e44162490 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009e50 tx=0x7f7e3c02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a061bbe660d4d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:57.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.109+0000 7f7e41d9b640 1 -- 172.21.15.116:0/2691196680 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c02cbb0 con 0x7f7e4406c960 2023-11-28T17:08:57.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.109+0000 7f7e41d9b640 1 -- 172.21.15.116:0/2691196680 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c02cd10 con 0x7f7e4406c960 2023-11-28T17:08:57.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.109+0000 7f7e41d9b640 1 -- 172.21.15.116:0/2691196680 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c034610 con 0x7f7e4406c960 2023-11-28T17:08:57.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.109+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2691196680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e4406c960 msgr2=0x7f7e44162490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:57.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.109+0000 7f7e48e2d640 1 --2- 172.21.15.116:0/2691196680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e4406c960 0x7f7e44162490 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009e50 tx=0x7f7e3c02c010 comp rx=0 tx=0).stop 2023-11-28T17:08:57.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.109+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2691196680 shutdown_connections 2023-11-28T17:08:57.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.109+0000 7f7e48e2d640 1 --2- 172.21.15.116:0/2691196680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e4406c960 0x7f7e44162490 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.109+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2691196680 >> 172.21.15.116:0/2691196680 conn(0x7f7e4406a4c0 msgr2=0x7f7e4406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:57.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.109+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2691196680 shutdown_connections 2023-11-28T17:08:57.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.109+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2691196680 wait complete. 2023-11-28T17:08:57.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.111+0000 7f7e48e2d640 1 Processor -- start 2023-11-28T17:08:57.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.111+0000 7f7e48e2d640 1 -- start start 2023-11-28T17:08:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.112+0000 7f7e48e2d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4406c960 0x7f7e440b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.112+0000 7f7e48e2d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e440b3940 0x7f7e441641c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.112+0000 7f7e48e2d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e440b3d20 0x7f7e44159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:57.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.112+0000 7f7e48e2d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e44151330 con 0x7f7e440b3940 2023-11-28T17:08:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.112+0000 7f7e48e2d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4415ab20 con 0x7f7e440b3d20 2023-11-28T17:08:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.112+0000 7f7e48e2d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4415a840 con 0x7f7e4406c960 2023-11-28T17:08:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.112+0000 7f7e3bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e440b3940 0x7f7e441641c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.112+0000 7f7e3bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e440b3940 0x7f7e441641c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41274/0 (socket says 172.21.15.116:41274) 2023-11-28T17:08:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.112+0000 7f7e3bfff640 1 -- 172.21.15.116:0/2014554171 learned_addr learned my addr 172.21.15.116:0/2014554171 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.112+0000 7f7e3bfff640 1 -- 172.21.15.116:0/2014554171 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4406c960 msgr2=0x7f7e440b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.112+0000 7f7e3bfff640 1 --2- 172.21.15.116:0/2014554171 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4406c960 0x7f7e440b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.112+0000 7f7e3bfff640 1 -- 172.21.15.116:0/2014554171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e440b3d20 msgr2=0x7f7e44159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.112+0000 7f7e3bfff640 1 --2- 172.21.15.116:0/2014554171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e440b3d20 0x7f7e44159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.113+0000 7f7e3bfff640 1 -- 172.21.15.116:0/2014554171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c009b30 con 0x7f7e440b3940 2023-11-28T17:08:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.113+0000 7f7e3bfff640 1 --2- 172.21.15.116:0/2014554171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e440b3940 0x7f7e441641c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c00c3c0 tx=0x7f7e2c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:57.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.113+0000 7f7e3affd640 1 -- 172.21.15.116:0/2014554171 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c0074e0 con 0x7f7e440b3940 2023-11-28T17:08:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.113+0000 7f7e3affd640 1 -- 172.21.15.116:0/2014554171 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2c007640 con 0x7f7e440b3940 2023-11-28T17:08:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.113+0000 7f7e3affd640 1 -- 172.21.15.116:0/2014554171 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c004a20 con 0x7f7e440b3940 2023-11-28T17:08:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.113+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2014554171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e440b0370 con 0x7f7e440b3940 2023-11-28T17:08:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.113+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2014554171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e440b0930 con 0x7f7e440b3940 2023-11-28T17:08:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.114+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2014554171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e440b0b50 con 0x7f7e440b3940 2023-11-28T17:08:57.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.115+0000 7f7e3affd640 1 -- 172.21.15.116:0/2014554171 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f7e2c004b80 con 0x7f7e440b3940 2023-11-28T17:08:57.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.116+0000 7f7e3affd640 1 --2- 172.21.15.116:0/2014554171 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7e2406b2d0 0x7f7e2406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:57.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.116+0000 7f7e3affd640 1 -- 172.21.15.116:0/2014554171 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7e2c089660 con 0x7f7e440b3940 2023-11-28T17:08:57.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.116+0000 7f7e4259c640 1 --2- 172.21.15.116:0/2014554171 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7e2406b2d0 0x7f7e2406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:57.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.117+0000 7f7e4259c640 1 --2- 172.21.15.116:0/2014554171 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7e2406b2d0 0x7f7e2406d790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7e34000f60 tx=0x7f7e340079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:57.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.120+0000 7f7e3affd640 1 -- 172.21.15.116:0/2014554171 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e2c054810 con 0x7f7e440b3940 2023-11-28T17:08:57.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.350+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2014554171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e440693d0 con 0x7f7e440b3940 2023-11-28T17:08:57.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.351+0000 7f7e3affd640 1 -- 172.21.15.116:0/2014554171 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3703 (secure 0 0 0) 0x7f7e2c057e30 con 0x7f7e440b3940 2023-11-28T17:08:57.352 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:08:57.352 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3138968863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":3138968863},{"type":"v1","addr":"172.21.15.156:6835","nonce":3138968863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/733981412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":733981412},{"type":"v1","addr":"172.21.15.116:6836","nonce":733981412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2751219705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2751219705},{"type":"v1","addr":"172.21.15.116:6837","nonce":2751219705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4254,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/3268354971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3268354971},{"type":"v1","addr":"172.21.15.156:6834","nonce":3268354971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-11-28T17:08:57.353 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 3 2023-11-28T17:08:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.353+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2014554171 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7e2406b2d0 msgr2=0x7f7e2406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.353+0000 7f7e48e2d640 1 --2- 172.21.15.116:0/2014554171 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7e2406b2d0 0x7f7e2406d790 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7e34000f60 tx=0x7f7e340079b0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.353+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2014554171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e440b3940 msgr2=0x7f7e441641c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.353+0000 7f7e48e2d640 1 --2- 172.21.15.116:0/2014554171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e440b3940 0x7f7e441641c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c00c3c0 tx=0x7f7e2c00c880 comp rx=0 tx=0).stop 2023-11-28T17:08:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.353+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2014554171 shutdown_connections 2023-11-28T17:08:57.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.353+0000 7f7e48e2d640 1 --2- 172.21.15.116:0/2014554171 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7e2406b2d0 0x7f7e2406d790 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.353+0000 7f7e48e2d640 1 --2- 172.21.15.116:0/2014554171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e440b3d20 0x7f7e44159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.353+0000 7f7e48e2d640 1 --2- 172.21.15.116:0/2014554171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e440b3940 0x7f7e441641c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.353+0000 7f7e48e2d640 1 --2- 172.21.15.116:0/2014554171 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e4406c960 0x7f7e440b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.354+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2014554171 >> 172.21.15.116:0/2014554171 conn(0x7f7e4406a4c0 msgr2=0x7f7e440cd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.354+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2014554171 shutdown_connections 2023-11-28T17:08:57.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.354+0000 7f7e48e2d640 1 -- 172.21.15.116:0/2014554171 wait complete. 2023-11-28T17:08:57.375 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:08:57.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.580+0000 7f084f37c640 1 Processor -- start 2023-11-28T17:08:57.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.580+0000 7f084f37c640 1 -- start start 2023-11-28T17:08:57.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.582+0000 7f084f37c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f084806c250 0x7f084806c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:57.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.582+0000 7f084f37c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084815fc10 con 0x7f084815f7b0 2023-11-28T17:08:57.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.582+0000 7f084f37c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0848070bd0 con 0x7f0848144430 2023-11-28T17:08:57.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.582+0000 7f084f37c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0848144810 con 0x7f084806c250 2023-11-28T17:08:57.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.582+0000 7f084c917640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f084815f7b0 0x7f0848071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33064/0 (socket says 172.21.15.116:33064) 2023-11-28T17:08:57.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.582+0000 7f084c917640 1 -- 172.21.15.116:0/3593648861 learned_addr learned my addr 172.21.15.116:0/3593648861 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:08:57.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.582+0000 7f083ffff640 1 -- 172.21.15.116:0/3593648861 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553225188 0 0) 0x7f084815fc10 con 0x7f084815f7b0 2023-11-28T17:08:57.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.582+0000 7f083ffff640 1 -- 172.21.15.116:0/3593648861 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0834003520 con 0x7f084815f7b0 2023-11-28T17:08:57.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.582+0000 7f084d919640 1 --2- 172.21.15.116:0/3593648861 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f084806c250 0x7f084806c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:57.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f083ffff640 1 -- 172.21.15.116:0/3593648861 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2316632877 0 0) 0x7f0834003520 con 0x7f084815f7b0 2023-11-28T17:08:57.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f083ffff640 1 -- 172.21.15.116:0/3593648861 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f084806c250 msgr2=0x7f084806c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:57.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f083ffff640 1 --2- 172.21.15.116:0/3593648861 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f084806c250 0x7f084806c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f083ffff640 1 -- 172.21.15.116:0/3593648861 >> v1:172.21.15.156:6789/0 conn(0x7f0848144430 legacy=0x7f08480703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f083ffff640 1 -- 172.21.15.116:0/3593648861 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0848070eb0 con 0x7f084815f7b0 2023-11-28T17:08:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f083ffff640 1 -- 172.21.15.116:0/3593648861 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f0838001670 con 0x7f084815f7b0 2023-11-28T17:08:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f083ffff640 1 -- 172.21.15.116:0/3593648861 >> v1:172.21.15.116:6789/0 conn(0x7f084815f7b0 legacy=0x7f0848071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f083ffff640 1 --2- 172.21.15.116:0/3593648861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08340039a0 0x7f0834003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f083ffff640 1 --2- 172.21.15.116:0/3593648861 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f084806c250 0x7f08340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f083ffff640 1 --2- 172.21.15.116:0/3593648861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0834004d00 0x7f0834005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f083ffff640 1 -- 172.21.15.116:0/3593648861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0848144810 con 0x7f08340039a0 2023-11-28T17:08:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f083ffff640 1 -- 172.21.15.116:0/3593648861 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083400bb10 con 0x7f0834004d00 2023-11-28T17:08:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f083ffff640 1 -- 172.21.15.116:0/3593648861 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083400bcd0 con 0x7f084806c250 2023-11-28T17:08:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f084d118640 1 --2- 172.21.15.116:0/3593648861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08340039a0 0x7f0834003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.583+0000 7f084c917640 1 --2- 172.21.15.116:0/3593648861 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f084806c250 0x7f08340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.584+0000 7f084d118640 1 -- 172.21.15.116:0/3593648861 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f084806c250 msgr2=0x7f08340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.584+0000 7f084d118640 1 --2- 172.21.15.116:0/3593648861 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f084806c250 0x7f08340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.584+0000 7f084d118640 1 -- 172.21.15.116:0/3593648861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0834004d00 msgr2=0x7f0834005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.584+0000 7f084d118640 1 --2- 172.21.15.116:0/3593648861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0834004d00 0x7f0834005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.584+0000 7f084d118640 1 -- 172.21.15.116:0/3593648861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0834003520 con 0x7f08340039a0 2023-11-28T17:08:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.584+0000 7f084c917640 1 --2- 172.21.15.116:0/3593648861 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f084806c250 0x7f08340047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:08:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.585+0000 7f084d118640 1 --2- 172.21.15.116:0/3593648861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08340039a0 0x7f0834003dc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0844001750 tx=0x7f08340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45e5b352bf5e6a5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.585+0000 7f083ffff640 1 -- 172.21.15.116:0/3593648861 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08440052d0 con 0x7f08340039a0 2023-11-28T17:08:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.585+0000 7f083ffff640 1 -- 172.21.15.116:0/3593648861 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084402e050 con 0x7f08340039a0 2023-11-28T17:08:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.585+0000 7f083ffff640 1 -- 172.21.15.116:0/3593648861 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08440035f0 con 0x7f08340039a0 2023-11-28T17:08:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.585+0000 7f084f37c640 1 -- 172.21.15.116:0/3593648861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08340039a0 msgr2=0x7f0834003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.585+0000 7f084f37c640 1 --2- 172.21.15.116:0/3593648861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08340039a0 0x7f0834003dc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0844001750 tx=0x7f08340057f0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.585+0000 7f084d919640 1 -- 172.21.15.116:0/3593648861 reap_dead start 2023-11-28T17:08:57.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.586+0000 7f084f37c640 1 -- 172.21.15.116:0/3593648861 shutdown_connections 2023-11-28T17:08:57.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.586+0000 7f084f37c640 1 -- 172.21.15.116:0/3593648861 >> 172.21.15.116:0/3593648861 conn(0x7f084806a4c0 msgr2=0x7f084806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:57.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.586+0000 7f084f37c640 1 -- 172.21.15.116:0/3593648861 shutdown_connections 2023-11-28T17:08:57.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.586+0000 7f084f37c640 1 -- 172.21.15.116:0/3593648861 wait complete. 2023-11-28T17:08:57.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.587+0000 7f084f37c640 1 Processor -- start 2023-11-28T17:08:57.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.587+0000 7f084f37c640 1 -- start start 2023-11-28T17:08:57.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.588+0000 7f084f37c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08340039a0 0x7f084815aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:57.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.588+0000 7f084f37c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08480aa990 0x7f08480aae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:57.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.588+0000 7f084f37c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08481407b0 0x7f0848140c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:57.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.588+0000 7f084f37c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084815afd0 con 0x7f08480aa990 2023-11-28T17:08:57.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.588+0000 7f084f37c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08480ab350 con 0x7f08481407b0 2023-11-28T17:08:57.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.588+0000 7f084f37c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08480a8460 con 0x7f08340039a0 2023-11-28T17:08:57.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.589+0000 7f084c917640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08480aa990 0x7f08480aae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:57.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.589+0000 7f084c917640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08480aa990 0x7f08480aae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41298/0 (socket says 172.21.15.116:41298) 2023-11-28T17:08:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.589+0000 7f084c917640 1 -- 172.21.15.116:0/343727602 learned_addr learned my addr 172.21.15.116:0/343727602 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.589+0000 7f084d118640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08340039a0 0x7f084815aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.589+0000 7f084c917640 1 -- 172.21.15.116:0/343727602 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08340039a0 msgr2=0x7f084815aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.589+0000 7f084c917640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08340039a0 0x7f084815aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.589+0000 7f084c917640 1 -- 172.21.15.116:0/343727602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08481407b0 msgr2=0x7f0848140c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.589+0000 7f084d919640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08481407b0 0x7f0848140c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.589+0000 7f084c917640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08481407b0 0x7f0848140c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.589+0000 7f084c917640 1 -- 172.21.15.116:0/343727602 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08480a8d00 con 0x7f08480aa990 2023-11-28T17:08:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.589+0000 7f084d919640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08481407b0 0x7f0848140c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:08:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.589+0000 7f084c917640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08480aa990 0x7f08480aae10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f083800d910 tx=0x7f083800ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:57.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.590+0000 7f083dffb640 1 -- 172.21.15.116:0/343727602 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08380074e0 con 0x7f08480aa990 2023-11-28T17:08:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.590+0000 7f083dffb640 1 -- 172.21.15.116:0/343727602 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0838001d70 con 0x7f08480aa990 2023-11-28T17:08:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.590+0000 7f083dffb640 1 -- 172.21.15.116:0/343727602 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0838004a00 con 0x7f08480aa990 2023-11-28T17:08:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.590+0000 7f084f37c640 1 -- 172.21.15.116:0/343727602 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0848141550 con 0x7f08480aa990 2023-11-28T17:08:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.590+0000 7f084f37c640 1 -- 172.21.15.116:0/343727602 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08480a8990 con 0x7f08480aa990 2023-11-28T17:08:57.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.591+0000 7f084f37c640 1 -- 172.21.15.116:0/343727602 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08480a8e90 con 0x7f08480aa990 2023-11-28T17:08:57.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.592+0000 7f083dffb640 1 -- 172.21.15.116:0/343727602 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0838004b60 con 0x7f08480aa990 2023-11-28T17:08:57.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.593+0000 7f083dffb640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f082c06b2a0 0x7f082c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:57.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.593+0000 7f083dffb640 1 -- 172.21.15.116:0/343727602 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0838089100 con 0x7f08480aa990 2023-11-28T17:08:57.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.593+0000 7f084d118640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f082c06b2a0 0x7f082c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:57.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.594+0000 7f084d118640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f082c06b2a0 0x7f082c06d760 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0844001750 tx=0x7f0844009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:57.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.597+0000 7f083dffb640 1 -- 172.21.15.116:0/343727602 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0838057de0 con 0x7f08480aa990 2023-11-28T17:08:57.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.787+0000 7f084f37c640 1 -- 172.21.15.116:0/343727602 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08480693d0 con 0x7f08480aa990 2023-11-28T17:08:57.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.789+0000 7f083dffb640 1 -- 172.21.15.116:0/343727602 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12862 (secure 0 0 0) 0x7f0838057970 con 0x7f08480aa990 2023-11-28T17:08:57.790 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:08:57.791 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":16,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:08:51.982083+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:08:57.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.791+0000 7f084f37c640 1 -- 172.21.15.116:0/343727602 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f082c06b2a0 msgr2=0x7f082c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:57.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.791+0000 7f084f37c640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f082c06b2a0 0x7f082c06d760 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0844001750 tx=0x7f0844009030 comp rx=0 tx=0).stop 2023-11-28T17:08:57.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.791+0000 7f084f37c640 1 -- 172.21.15.116:0/343727602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08480aa990 msgr2=0x7f08480aae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:57.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.791+0000 7f084f37c640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08480aa990 0x7f08480aae10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f083800d910 tx=0x7f083800ddd0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.791+0000 7f084f37c640 1 -- 172.21.15.116:0/343727602 shutdown_connections 2023-11-28T17:08:57.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.791+0000 7f084f37c640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f082c06b2a0 0x7f082c06d760 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.791+0000 7f084f37c640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08481407b0 0x7f0848140c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.791+0000 7f084f37c640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08480aa990 0x7f08480aae10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.791+0000 7f084f37c640 1 --2- 172.21.15.116:0/343727602 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08340039a0 0x7f084815aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:57.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.791+0000 7f084f37c640 1 -- 172.21.15.116:0/343727602 >> 172.21.15.116:0/343727602 conn(0x7f084806a4c0 msgr2=0x7f084815b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:57.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.792+0000 7f084f37c640 1 -- 172.21.15.116:0/343727602 shutdown_connections 2023-11-28T17:08:57.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:57.792+0000 7f084f37c640 1 -- 172.21.15.116:0/343727602 wait complete. 2023-11-28T17:08:57.811 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:08:57.811 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:08:58.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.023+0000 7f2258927640 1 Processor -- start 2023-11-28T17:08:58.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.024+0000 7f2258927640 1 -- start start 2023-11-28T17:08:58.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.025+0000 7f2258927640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2254152050 0x7f225415bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.025+0000 7f2258927640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f22540b2fe0 0x7f22540ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.025+0000 7f2258927640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22540b2b50 con 0x7f22540abd30 2023-11-28T17:08:58.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.025+0000 7f2258927640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f225415c520 con 0x7f2254152050 2023-11-28T17:08:58.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.025+0000 7f2258927640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22540ac2d0 con 0x7f22540b2fe0 2023-11-28T17:08:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.026+0000 7f22537fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2254152050 0x7f225415bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.026+0000 7f22537fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2254152050 0x7f225415bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43500/0 (socket says 172.21.15.116:43500) 2023-11-28T17:08:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.026+0000 7f22537fe640 1 -- 172.21.15.116:0/131428491 learned_addr learned my addr 172.21.15.116:0/131428491 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.026+0000 7f2252ffd640 1 --2- 172.21.15.116:0/131428491 >> v2:172.21.15.156:3300/0 conn(0x7f22540b2fe0 0x7f22540ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.026+0000 7f22537fe640 1 -- 172.21.15.116:0/131428491 >> v2:172.21.15.156:3300/0 conn(0x7f22540b2fe0 msgr2=0x7f22540ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.026+0000 7f22537fe640 1 --2- 172.21.15.116:0/131428491 >> v2:172.21.15.156:3300/0 conn(0x7f22540b2fe0 0x7f22540ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.026+0000 7f22537fe640 1 -- 172.21.15.116:0/131428491 >> v1:172.21.15.156:6789/0 conn(0x7f22540abd30 legacy=0x7f22540cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:58.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.026+0000 7f22537fe640 1 -- 172.21.15.116:0/131428491 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22540b2220 con 0x7f2254152050 2023-11-28T17:08:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.027+0000 7f22537fe640 1 --2- 172.21.15.116:0/131428491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2254152050 0x7f225415bfe0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f225406e200 tx=0x7f224402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67cbf9ade1bbe903 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.028+0000 7f22527fc640 1 -- 172.21.15.116:0/131428491 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2244031870 con 0x7f2254152050 2023-11-28T17:08:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.028+0000 7f22527fc640 1 -- 172.21.15.116:0/131428491 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22440319d0 con 0x7f2254152050 2023-11-28T17:08:58.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.028+0000 7f22527fc640 1 -- 172.21.15.116:0/131428491 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2244031d60 con 0x7f2254152050 2023-11-28T17:08:58.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.028+0000 7f2258927640 1 -- 172.21.15.116:0/131428491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2254152050 msgr2=0x7f225415bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.028+0000 7f2258927640 1 --2- 172.21.15.116:0/131428491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2254152050 0x7f225415bfe0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f225406e200 tx=0x7f224402b910 comp rx=0 tx=0).stop 2023-11-28T17:08:58.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.028+0000 7f2258927640 1 -- 172.21.15.116:0/131428491 shutdown_connections 2023-11-28T17:08:58.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.028+0000 7f2258927640 1 --2- 172.21.15.116:0/131428491 >> v2:172.21.15.156:3300/0 conn(0x7f22540b2fe0 0x7f22540ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.028+0000 7f2258927640 1 --2- 172.21.15.116:0/131428491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2254152050 0x7f225415bfe0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.028+0000 7f2258927640 1 -- 172.21.15.116:0/131428491 >> 172.21.15.116:0/131428491 conn(0x7f22540b0950 msgr2=0x7f22540b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:58.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.029+0000 7f2258927640 1 -- 172.21.15.116:0/131428491 shutdown_connections 2023-11-28T17:08:58.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.029+0000 7f2258927640 1 -- 172.21.15.116:0/131428491 wait complete. 2023-11-28T17:08:58.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.030+0000 7f2258927640 1 Processor -- start 2023-11-28T17:08:58.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.030+0000 7f2258927640 1 -- start start 2023-11-28T17:08:58.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.031+0000 7f2258927640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22540abd30 0x7f2254152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.031+0000 7f2258927640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22540b2fe0 0x7f22541525b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.031+0000 7f2258927640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2254152050 0x7f2254147840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.031+0000 7f2258927640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2254152af0 con 0x7f2254152050 2023-11-28T17:08:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.031+0000 7f2258927640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2254147d80 con 0x7f22540abd30 2023-11-28T17:08:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.031+0000 7f2258927640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22540701f0 con 0x7f22540b2fe0 2023-11-28T17:08:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.032+0000 7f2253fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2254152050 0x7f2254147840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.032+0000 7f2253fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2254152050 0x7f2254147840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43514/0 (socket says 172.21.15.116:43514) 2023-11-28T17:08:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.032+0000 7f2253fff640 1 -- 172.21.15.116:0/3814900304 learned_addr learned my addr 172.21.15.116:0/3814900304 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.032+0000 7f2252ffd640 1 --2- 172.21.15.116:0/3814900304 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22540b2fe0 0x7f22541525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.032+0000 7f22537fe640 1 --2- 172.21.15.116:0/3814900304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22540abd30 0x7f2254152dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.032+0000 7f2253fff640 1 -- 172.21.15.116:0/3814900304 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22540b2fe0 msgr2=0x7f22541525b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.032+0000 7f2253fff640 1 --2- 172.21.15.116:0/3814900304 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22540b2fe0 0x7f22541525b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.032+0000 7f2253fff640 1 -- 172.21.15.116:0/3814900304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22540abd30 msgr2=0x7f2254152dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.032+0000 7f2253fff640 1 --2- 172.21.15.116:0/3814900304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22540abd30 0x7f2254152dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.032+0000 7f2253fff640 1 -- 172.21.15.116:0/3814900304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22440094d0 con 0x7f2254152050 2023-11-28T17:08:58.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.032+0000 7f2253fff640 1 --2- 172.21.15.116:0/3814900304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2254152050 0x7f2254147840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f224c00c780 tx=0x7f224c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:58.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.033+0000 7f223bfff640 1 -- 172.21.15.116:0/3814900304 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c004b60 con 0x7f2254152050 2023-11-28T17:08:58.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.033+0000 7f223bfff640 1 -- 172.21.15.116:0/3814900304 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224c00ce90 con 0x7f2254152050 2023-11-28T17:08:58.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.033+0000 7f223bfff640 1 -- 172.21.15.116:0/3814900304 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c00a7f0 con 0x7f2254152050 2023-11-28T17:08:58.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.033+0000 7f2258927640 1 -- 172.21.15.116:0/3814900304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22541533a0 con 0x7f2254152050 2023-11-28T17:08:58.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.033+0000 7f2258927640 1 -- 172.21.15.116:0/3814900304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22540704b0 con 0x7f2254152050 2023-11-28T17:08:58.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.034+0000 7f2258927640 1 -- 172.21.15.116:0/3814900304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225409d0b0 con 0x7f2254152050 2023-11-28T17:08:58.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.035+0000 7f223bfff640 1 -- 172.21.15.116:0/3814900304 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f224c00a950 con 0x7f2254152050 2023-11-28T17:08:58.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.036+0000 7f223bfff640 1 --2- 172.21.15.116:0/3814900304 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f224006b2a0 0x7f224006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.036+0000 7f223bfff640 1 -- 172.21.15.116:0/3814900304 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f224c089a60 con 0x7f2254152050 2023-11-28T17:08:58.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.036+0000 7f22537fe640 1 --2- 172.21.15.116:0/3814900304 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f224006b2a0 0x7f224006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.037+0000 7f22537fe640 1 --2- 172.21.15.116:0/3814900304 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f224006b2a0 0x7f224006d760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2244036000 tx=0x7f224403c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:58.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.040+0000 7f223bfff640 1 -- 172.21.15.116:0/3814900304 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f224c054b80 con 0x7f2254152050 2023-11-28T17:08:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.231+0000 7f2258927640 1 -- 172.21.15.116:0/3814900304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f225404e960 con 0x7f2254152050 2023-11-28T17:08:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.232+0000 7f223bfff640 1 -- 172.21.15.116:0/3814900304 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f224c0581a0 con 0x7f2254152050 2023-11-28T17:08:58.234 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:08:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.235+0000 7f2258927640 1 -- 172.21.15.116:0/3814900304 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f224006b2a0 msgr2=0x7f224006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.235+0000 7f2258927640 1 --2- 172.21.15.116:0/3814900304 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f224006b2a0 0x7f224006d760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2244036000 tx=0x7f224403c040 comp rx=0 tx=0).stop 2023-11-28T17:08:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.235+0000 7f2258927640 1 -- 172.21.15.116:0/3814900304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2254152050 msgr2=0x7f2254147840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.235+0000 7f2258927640 1 --2- 172.21.15.116:0/3814900304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2254152050 0x7f2254147840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f224c00c780 tx=0x7f224c00cc40 comp rx=0 tx=0).stop 2023-11-28T17:08:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.235+0000 7f2258927640 1 -- 172.21.15.116:0/3814900304 shutdown_connections 2023-11-28T17:08:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.235+0000 7f2258927640 1 --2- 172.21.15.116:0/3814900304 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f224006b2a0 0x7f224006d760 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.235+0000 7f2258927640 1 --2- 172.21.15.116:0/3814900304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2254152050 0x7f2254147840 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.235+0000 7f2258927640 1 --2- 172.21.15.116:0/3814900304 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22540b2fe0 0x7f22541525b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.235+0000 7f2258927640 1 --2- 172.21.15.116:0/3814900304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22540abd30 0x7f2254152dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.235+0000 7f2258927640 1 -- 172.21.15.116:0/3814900304 >> 172.21.15.116:0/3814900304 conn(0x7f22540b0950 msgr2=0x7f225415c700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.235+0000 7f2258927640 1 -- 172.21.15.116:0/3814900304 shutdown_connections 2023-11-28T17:08:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.235+0000 7f2258927640 1 -- 172.21.15.116:0/3814900304 wait complete. 2023-11-28T17:08:58.256 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T17:08:58.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.462+0000 7f55affff640 1 Processor -- start 2023-11-28T17:08:58.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.462+0000 7f55affff640 1 -- start start 2023-11-28T17:08:58.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.464+0000 7f55affff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f55b00cf2a0 0x7f55b00701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.464+0000 7f55affff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f55b015ae90 0x7f55b00ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.464+0000 7f55affff640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b00cd8e0 con 0x7f55b0159f30 2023-11-28T17:08:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.464+0000 7f55affff640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b0070730 con 0x7f55b015ae90 2023-11-28T17:08:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.464+0000 7f55affff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b0070ee0 con 0x7f55b00cf2a0 2023-11-28T17:08:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.464+0000 7f55aeffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f55b0159f30 0x7f55b00ceb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45514/0 (socket says 172.21.15.116:45514) 2023-11-28T17:08:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.464+0000 7f55aeffd640 1 -- 172.21.15.116:0/309741003 learned_addr learned my addr 172.21.15.116:0/309741003 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:08:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.464+0000 7f55af7fe640 1 --2- 172.21.15.116:0/309741003 >> v2:172.21.15.116:3300/0 conn(0x7f55b015ae90 0x7f55b00ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.464+0000 7f55ae7fc640 1 --2- 172.21.15.116:0/309741003 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f55b00cf2a0 0x7f55b00701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.464+0000 7f55af7fe640 1 -- 172.21.15.116:0/309741003 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f55b00cf2a0 msgr2=0x7f55b00701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:58.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.464+0000 7f55af7fe640 1 --2- 172.21.15.116:0/309741003 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f55b00cf2a0 0x7f55b00701f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.464+0000 7f55af7fe640 1 -- 172.21.15.116:0/309741003 >> v1:172.21.15.116:6789/0 conn(0x7f55b0159f30 legacy=0x7f55b00ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.464+0000 7f55af7fe640 1 -- 172.21.15.116:0/309741003 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b015b310 con 0x7f55b015ae90 2023-11-28T17:08:58.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.464+0000 7f55adffb640 1 -- 172.21.15.116:0/309741003 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753818951 0 0) 0x7f55b00cd8e0 con 0x7f55b0159f30 2023-11-28T17:08:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.465+0000 7f55af7fe640 1 --2- 172.21.15.116:0/309741003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55b015ae90 0x7f55b00ce330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f55a8009d90 tx=0x7f55a802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71febf81cfa49ec1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.466+0000 7f55adffb640 1 -- 172.21.15.116:0/309741003 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a802cc50 con 0x7f55b015ae90 2023-11-28T17:08:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.466+0000 7f55adffb640 1 -- 172.21.15.116:0/309741003 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a802cdb0 con 0x7f55b015ae90 2023-11-28T17:08:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.466+0000 7f55adffb640 1 -- 172.21.15.116:0/309741003 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a8034660 con 0x7f55b015ae90 2023-11-28T17:08:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.466+0000 7f55affff640 1 -- 172.21.15.116:0/309741003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55b015ae90 msgr2=0x7f55b00ce330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.466+0000 7f55affff640 1 --2- 172.21.15.116:0/309741003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55b015ae90 0x7f55b00ce330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f55a8009d90 tx=0x7f55a802bea0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.466+0000 7f55affff640 1 -- 172.21.15.116:0/309741003 shutdown_connections 2023-11-28T17:08:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.466+0000 7f55affff640 1 --2- 172.21.15.116:0/309741003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55b015ae90 0x7f55b00ce330 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.466+0000 7f55affff640 1 --2- 172.21.15.116:0/309741003 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f55b00cf2a0 0x7f55b00701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.466+0000 7f55affff640 1 -- 172.21.15.116:0/309741003 >> 172.21.15.116:0/309741003 conn(0x7f55b00b0950 msgr2=0x7f55b00b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.466+0000 7f55affff640 1 -- 172.21.15.116:0/309741003 shutdown_connections 2023-11-28T17:08:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.466+0000 7f55affff640 1 -- 172.21.15.116:0/309741003 wait complete. 2023-11-28T17:08:58.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.468+0000 7f55affff640 1 Processor -- start 2023-11-28T17:08:58.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.468+0000 7f55affff640 1 -- start start 2023-11-28T17:08:58.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.469+0000 7f55affff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f55b00cf2a0 0x7f55b015cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.469+0000 7f55affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55b0159f30 0x7f55b00acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.469+0000 7f55affff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55b015ae90 0x7f55b00ad4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.469+0000 7f55affff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b015b8b0 con 0x7f55b0159f30 2023-11-28T17:08:58.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.469+0000 7f55affff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b015d050 con 0x7f55b015ae90 2023-11-28T17:08:58.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.469+0000 7f55affff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b01555a0 con 0x7f55b00cf2a0 2023-11-28T17:08:58.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.469+0000 7f55ae7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55b0159f30 0x7f55b00acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.469+0000 7f55ae7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55b0159f30 0x7f55b00acfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43534/0 (socket says 172.21.15.116:43534) 2023-11-28T17:08:58.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.469+0000 7f55ae7fc640 1 -- 172.21.15.116:0/1316885158 learned_addr learned my addr 172.21.15.116:0/1316885158 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.470+0000 7f55aeffd640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f55b00cf2a0 0x7f55b015cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.470+0000 7f55af7fe640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55b015ae90 0x7f55b00ad4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.470+0000 7f55ae7fc640 1 -- 172.21.15.116:0/1316885158 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f55b00cf2a0 msgr2=0x7f55b015cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.470+0000 7f55ae7fc640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f55b00cf2a0 0x7f55b015cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.470+0000 7f55ae7fc640 1 -- 172.21.15.116:0/1316885158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55b015ae90 msgr2=0x7f55b00ad4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.470+0000 7f55ae7fc640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55b015ae90 0x7f55b00ad4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.470+0000 7f55ae7fc640 1 -- 172.21.15.116:0/1316885158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a8009a70 con 0x7f55b0159f30 2023-11-28T17:08:58.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.470+0000 7f55af7fe640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55b015ae90 0x7f55b00ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:08:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.470+0000 7f55ae7fc640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55b0159f30 0x7f55b00acfb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55a000c5c0 tx=0x7f55a000ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.470+0000 7f55a7fff640 1 -- 172.21.15.116:0/1316885158 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a0004220 con 0x7f55b0159f30 2023-11-28T17:08:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.470+0000 7f55a7fff640 1 -- 172.21.15.116:0/1316885158 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a0004380 con 0x7f55b0159f30 2023-11-28T17:08:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.470+0000 7f55a7fff640 1 -- 172.21.15.116:0/1316885158 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a0003830 con 0x7f55b0159f30 2023-11-28T17:08:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.470+0000 7f55affff640 1 -- 172.21.15.116:0/1316885158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b0155880 con 0x7f55b0159f30 2023-11-28T17:08:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.471+0000 7f55affff640 1 -- 172.21.15.116:0/1316885158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b0155e40 con 0x7f55b0159f30 2023-11-28T17:08:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.472+0000 7f55affff640 1 -- 172.21.15.116:0/1316885158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b009d0b0 con 0x7f55b0159f30 2023-11-28T17:08:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.472+0000 7f55a7fff640 1 -- 172.21.15.116:0/1316885158 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f55a0003990 con 0x7f55b0159f30 2023-11-28T17:08:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.473+0000 7f55a7fff640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f559006b2a0 0x7f559006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.474+0000 7f55a7fff640 1 -- 172.21.15.116:0/1316885158 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f55a0089560 con 0x7f55b0159f30 2023-11-28T17:08:58.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.474+0000 7f55aeffd640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f559006b2a0 0x7f559006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.474+0000 7f55aeffd640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f559006b2a0 0x7f559006d760 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5598007fd0 tx=0x7f5598009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:58.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.478+0000 7f55a7fff640 1 -- 172.21.15.116:0/1316885158 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55a0054680 con 0x7f55b0159f30 2023-11-28T17:08:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.669+0000 7f55affff640 1 -- 172.21.15.116:0/1316885158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f55b015e530 con 0x7f55b0159f30 2023-11-28T17:08:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.670+0000 7f55a7fff640 1 -- 172.21.15.116:0/1316885158 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f55a001e090 con 0x7f55b0159f30 2023-11-28T17:08:58.671 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:08:58.672 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2023-11-28T17:08:58.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:08:58.672 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 1, 2023-11-28T17:08:58.672 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": ".mgr" 2023-11-28T17:08:58.672 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:08:58.673 INFO:teuthology.orchestra.run.smithi116.stdout:] 2023-11-28T17:08:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.672+0000 7f55affff640 1 -- 172.21.15.116:0/1316885158 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f559006b2a0 msgr2=0x7f559006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.672+0000 7f55affff640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f559006b2a0 0x7f559006d760 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5598007fd0 tx=0x7f5598009040 comp rx=0 tx=0).stop 2023-11-28T17:08:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.672+0000 7f55affff640 1 -- 172.21.15.116:0/1316885158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55b0159f30 msgr2=0x7f55b00acfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.672+0000 7f55affff640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55b0159f30 0x7f55b00acfb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55a000c5c0 tx=0x7f55a000ca80 comp rx=0 tx=0).stop 2023-11-28T17:08:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.673+0000 7f55affff640 1 -- 172.21.15.116:0/1316885158 shutdown_connections 2023-11-28T17:08:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.673+0000 7f55affff640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f559006b2a0 0x7f559006d760 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.673+0000 7f55affff640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55b015ae90 0x7f55b00ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.673+0000 7f55affff640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55b0159f30 0x7f55b00acfb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.673+0000 7f55affff640 1 --2- 172.21.15.116:0/1316885158 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f55b00cf2a0 0x7f55b015cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.673+0000 7f55affff640 1 -- 172.21.15.116:0/1316885158 >> 172.21.15.116:0/1316885158 conn(0x7f55b00b0950 msgr2=0x7f55b00cb450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.673+0000 7f55affff640 1 -- 172.21.15.116:0/1316885158 shutdown_connections 2023-11-28T17:08:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.673+0000 7f55affff640 1 -- 172.21.15.116:0/1316885158 wait complete. 2023-11-28T17:08:58.691 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T17:08:58.692 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T17:08:58.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.902+0000 7f907c072640 1 Processor -- start 2023-11-28T17:08:58.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.902+0000 7f907c072640 1 -- start start 2023-11-28T17:08:58.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.904+0000 7f907c072640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740d1620 0x7f9074161a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.904+0000 7f907c072640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907406eaf0 con 0x7f9074146b10 2023-11-28T17:08:58.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.904+0000 7f907c072640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907415e4e0 con 0x7f907415d8a0 2023-11-28T17:08:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.904+0000 7f907c072640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907415e7c0 con 0x7f90740d1620 2023-11-28T17:08:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.904+0000 7f9079e0e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9074146b10 0x7f90740d0f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45530/0 (socket says 172.21.15.116:45530) 2023-11-28T17:08:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.904+0000 7f9079e0e640 1 -- 172.21.15.116:0/4197132533 learned_addr learned my addr 172.21.15.116:0/4197132533 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:08:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.904+0000 7f907960d640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740d1620 0x7f9074161a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 -- 172.21.15.116:0/4197132533 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168276698 0 0) 0x7f907406eaf0 con 0x7f9074146b10 2023-11-28T17:08:58.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 -- 172.21.15.116:0/4197132533 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9060003520 con 0x7f9074146b10 2023-11-28T17:08:58.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 -- 172.21.15.116:0/4197132533 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4089714861 0 0) 0x7f9060003520 con 0x7f9074146b10 2023-11-28T17:08:58.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 -- 172.21.15.116:0/4197132533 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740d1620 msgr2=0x7f9074161a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:58.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740d1620 0x7f9074161a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 -- 172.21.15.116:0/4197132533 >> v1:172.21.15.156:6789/0 conn(0x7f907415d8a0 legacy=0x7f907415dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 -- 172.21.15.116:0/4197132533 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9074160f50 con 0x7f9074146b10 2023-11-28T17:08:58.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 -- 172.21.15.116:0/4197132533 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f90700019f0 con 0x7f9074146b10 2023-11-28T17:08:58.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f907960d640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740d1620 0x7f9074161a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:08:58.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 -- 172.21.15.116:0/4197132533 >> v1:172.21.15.116:6789/0 conn(0x7f9074146b10 legacy=0x7f90740d0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90600039a0 0x7f9060003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9060004350 0x7f9060004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740d1620 0x7f9060005070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 -- 172.21.15.116:0/4197132533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907415e7c0 con 0x7f9060004350 2023-11-28T17:08:58.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 -- 172.21.15.116:0/4197132533 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906000b980 con 0x7f90600039a0 2023-11-28T17:08:58.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.905+0000 7f9078e0c640 1 -- 172.21.15.116:0/4197132533 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906000bb40 con 0x7f90740d1620 2023-11-28T17:08:58.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.906+0000 7f907960d640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9060004350 0x7f9060004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.906+0000 7f9079e0e640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90600039a0 0x7f9060003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.906+0000 7f907a60f640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740d1620 0x7f9060005070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.906+0000 7f907960d640 1 -- 172.21.15.116:0/4197132533 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740d1620 msgr2=0x7f9060005070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:58.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.906+0000 7f907960d640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740d1620 0x7f9060005070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.906+0000 7f907960d640 1 -- 172.21.15.116:0/4197132533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90600039a0 msgr2=0x7f9060003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:58.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.906+0000 7f907960d640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90600039a0 0x7f9060003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.906+0000 7f907960d640 1 -- 172.21.15.116:0/4197132533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9064009400 con 0x7f9060004350 2023-11-28T17:08:58.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.906+0000 7f907a60f640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740d1620 0x7f9060005070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:08:58.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.906+0000 7f9079e0e640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90600039a0 0x7f9060003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:08:58.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.907+0000 7f907960d640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9060004350 0x7f9060004820 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9064009c90 tx=0x7f90600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d49b4f7b6e9496c5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:58.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.907+0000 7f9078e0c640 1 -- 172.21.15.116:0/4197132533 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906402c8f0 con 0x7f9060004350 2023-11-28T17:08:58.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.907+0000 7f9078e0c640 1 -- 172.21.15.116:0/4197132533 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906402ca50 con 0x7f9060004350 2023-11-28T17:08:58.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.907+0000 7f9078e0c640 1 -- 172.21.15.116:0/4197132533 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906402ce00 con 0x7f9060004350 2023-11-28T17:08:58.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.908+0000 7f907c072640 1 -- 172.21.15.116:0/4197132533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9060004350 msgr2=0x7f9060004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.908+0000 7f907c072640 1 --2- 172.21.15.116:0/4197132533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9060004350 0x7f9060004820 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9064009c90 tx=0x7f90600057f0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.908+0000 7f907a60f640 1 -- 172.21.15.116:0/4197132533 reap_dead start 2023-11-28T17:08:58.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.908+0000 7f907c072640 1 -- 172.21.15.116:0/4197132533 shutdown_connections 2023-11-28T17:08:58.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.908+0000 7f907c072640 1 -- 172.21.15.116:0/4197132533 >> 172.21.15.116:0/4197132533 conn(0x7f9074070af0 msgr2=0x7f90740662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:58.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.908+0000 7f907c072640 1 -- 172.21.15.116:0/4197132533 shutdown_connections 2023-11-28T17:08:58.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.908+0000 7f907c072640 1 -- 172.21.15.116:0/4197132533 wait complete. 2023-11-28T17:08:58.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.910+0000 7f907c072640 1 Processor -- start 2023-11-28T17:08:58.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.910+0000 7f907c072640 1 -- start start 2023-11-28T17:08:58.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.911+0000 7f907c072640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9060004350 0x7f90741f23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.911+0000 7f907c072640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740b1e50 0x7f90740b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.911+0000 7f907c072640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9074132db0 0x7f9074133260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.911+0000 7f907c072640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90740b2810 con 0x7f9074132db0 2023-11-28T17:08:58.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.911+0000 7f907c072640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9074133ae0 con 0x7f9060004350 2023-11-28T17:08:58.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.911+0000 7f907c072640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9074130c00 con 0x7f90740b1e50 2023-11-28T17:08:58.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.911+0000 7f907a60f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9074132db0 0x7f9074133260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.911+0000 7f907a60f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9074132db0 0x7f9074133260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43544/0 (socket says 172.21.15.116:43544) 2023-11-28T17:08:58.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.911+0000 7f907a60f640 1 -- 172.21.15.116:0/3443454977 learned_addr learned my addr 172.21.15.116:0/3443454977 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:58.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.911+0000 7f907960d640 1 --2- 172.21.15.116:0/3443454977 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740b1e50 0x7f90740b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.911+0000 7f9079e0e640 1 --2- 172.21.15.116:0/3443454977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9060004350 0x7f90741f23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.912+0000 7f907a60f640 1 -- 172.21.15.116:0/3443454977 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740b1e50 msgr2=0x7f90740b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.912+0000 7f907a60f640 1 --2- 172.21.15.116:0/3443454977 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740b1e50 0x7f90740b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.912+0000 7f907a60f640 1 -- 172.21.15.116:0/3443454977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9060004350 msgr2=0x7f90741f23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:58.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.912+0000 7f907a60f640 1 --2- 172.21.15.116:0/3443454977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9060004350 0x7f90741f23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:58.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.912+0000 7f907a60f640 1 -- 172.21.15.116:0/3443454977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9074130ee0 con 0x7f9074132db0 2023-11-28T17:08:58.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.912+0000 7f907a60f640 1 --2- 172.21.15.116:0/3443454977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9074132db0 0x7f9074133260 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9074161f70 tx=0x7f906c00d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:58.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.912+0000 7f906a7fc640 1 -- 172.21.15.116:0/3443454977 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c00b070 con 0x7f9074132db0 2023-11-28T17:08:58.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.912+0000 7f906a7fc640 1 -- 172.21.15.116:0/3443454977 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906c0035f0 con 0x7f9074132db0 2023-11-28T17:08:58.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.912+0000 7f906a7fc640 1 -- 172.21.15.116:0/3443454977 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906c003980 con 0x7f9074132db0 2023-11-28T17:08:58.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.912+0000 7f907c072640 1 -- 172.21.15.116:0/3443454977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90741311c0 con 0x7f9074132db0 2023-11-28T17:08:58.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.912+0000 7f907c072640 1 -- 172.21.15.116:0/3443454977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90741f28f0 con 0x7f9074132db0 2023-11-28T17:08:58.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.913+0000 7f907c072640 1 -- 172.21.15.116:0/3443454977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90741316e0 con 0x7f9074132db0 2023-11-28T17:08:58.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.914+0000 7f906a7fc640 1 -- 172.21.15.116:0/3443454977 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f906c003ae0 con 0x7f9074132db0 2023-11-28T17:08:58.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.915+0000 7f906a7fc640 1 --2- 172.21.15.116:0/3443454977 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f905806b2d0 0x7f905806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:58.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.915+0000 7f906a7fc640 1 -- 172.21.15.116:0/3443454977 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f906c088fa0 con 0x7f9074132db0 2023-11-28T17:08:58.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.915+0000 7f9079e0e640 1 --2- 172.21.15.116:0/3443454977 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f905806b2d0 0x7f905806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:58.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.916+0000 7f9079e0e640 1 --2- 172.21.15.116:0/3443454977 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f905806b2d0 0x7f905806d790 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9070000f80 tx=0x7f9070009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:58.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:58.920+0000 7f906a7fc640 1 -- 172.21.15.116:0/3443454977 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90741316e0 con 0x7f9074132db0 2023-11-28T17:08:59.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.108+0000 7f907c072640 1 -- 172.21.15.116:0/3443454977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f90741316e0 con 0x7f9074132db0 2023-11-28T17:08:59.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.218+0000 7f906a7fc640 1 -- 172.21.15.116:0/3443454977 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f90741316e0 con 0x7f9074132db0 2023-11-28T17:08:59.220 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' created 2023-11-28T17:08:59.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.222+0000 7f907c072640 1 -- 172.21.15.116:0/3443454977 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f905806b2d0 msgr2=0x7f905806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:59.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.222+0000 7f907c072640 1 --2- 172.21.15.116:0/3443454977 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f905806b2d0 0x7f905806d790 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9070000f80 tx=0x7f9070009040 comp rx=0 tx=0).stop 2023-11-28T17:08:59.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.222+0000 7f907c072640 1 -- 172.21.15.116:0/3443454977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9074132db0 msgr2=0x7f9074133260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:59.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.222+0000 7f907c072640 1 --2- 172.21.15.116:0/3443454977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9074132db0 0x7f9074133260 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9074161f70 tx=0x7f906c00d890 comp rx=0 tx=0).stop 2023-11-28T17:08:59.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.222+0000 7f907c072640 1 -- 172.21.15.116:0/3443454977 shutdown_connections 2023-11-28T17:08:59.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.223+0000 7f907c072640 1 --2- 172.21.15.116:0/3443454977 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f905806b2d0 0x7f905806d790 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:59.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.223+0000 7f907c072640 1 --2- 172.21.15.116:0/3443454977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9074132db0 0x7f9074133260 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:59.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.223+0000 7f907c072640 1 --2- 172.21.15.116:0/3443454977 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90740b1e50 0x7f90740b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:59.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.223+0000 7f907c072640 1 --2- 172.21.15.116:0/3443454977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9060004350 0x7f90741f23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:59.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.223+0000 7f907c072640 1 -- 172.21.15.116:0/3443454977 >> 172.21.15.116:0/3443454977 conn(0x7f9074070af0 msgr2=0x7f90740662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:59.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.223+0000 7f907c072640 1 -- 172.21.15.116:0/3443454977 shutdown_connections 2023-11-28T17:08:59.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.223+0000 7f907c072640 1 -- 172.21.15.116:0/3443454977 wait complete. 2023-11-28T17:08:59.250 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T17:08:59.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.443+0000 7fece66b4640 1 Processor -- start 2023-11-28T17:08:59.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.443+0000 7fece66b4640 1 -- start start 2023-11-28T17:08:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.445+0000 7fece66b4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fece015b2d0 0x7fece006dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.445+0000 7fece66b4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fece0070c90 0x7fece00710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.445+0000 7fece66b4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece0071600 con 0x7fece00b4700 2023-11-28T17:08:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.445+0000 7fece66b4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece006b5e0 con 0x7fece0070c90 2023-11-28T17:08:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.445+0000 7fece66b4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece006b7d0 con 0x7fece015b2d0 2023-11-28T17:08:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.445+0000 7fecd77fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fece00b4700 0x7fece006aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45534/0 (socket says 172.21.15.116:45534) 2023-11-28T17:08:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.445+0000 7fecd77fe640 1 -- 172.21.15.116:0/834426692 learned_addr learned my addr 172.21.15.116:0/834426692 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:08:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.445+0000 7fecdf7fe640 1 -- 172.21.15.116:0/834426692 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264951725 0 0) 0x7fece0071600 con 0x7fece00b4700 2023-11-28T17:08:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.445+0000 7fecdffff640 1 --2- 172.21.15.116:0/834426692 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fece015b2d0 0x7fece006dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:59.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.445+0000 7fecdf7fe640 1 -- 172.21.15.116:0/834426692 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feccc003520 con 0x7fece00b4700 2023-11-28T17:08:59.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.445+0000 7fece4c51640 1 --2- 172.21.15.116:0/834426692 >> v2:172.21.15.156:3300/0 conn(0x7fece0070c90 0x7fece00710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:59.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fecdf7fe640 1 -- 172.21.15.116:0/834426692 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3966206077 0 0) 0x7feccc003520 con 0x7fece00b4700 2023-11-28T17:08:59.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fecdf7fe640 1 -- 172.21.15.116:0/834426692 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fece015b2d0 msgr2=0x7fece006dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:59.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fecdf7fe640 1 --2- 172.21.15.116:0/834426692 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fece015b2d0 0x7fece006dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:59.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fecdf7fe640 1 -- 172.21.15.116:0/834426692 >> v2:172.21.15.156:3300/0 conn(0x7fece0070c90 msgr2=0x7fece00710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:59.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fecdf7fe640 1 --2- 172.21.15.116:0/834426692 >> v2:172.21.15.156:3300/0 conn(0x7fece0070c90 0x7fece00710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fecdf7fe640 1 -- 172.21.15.116:0/834426692 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece006d120 con 0x7fece00b4700 2023-11-28T17:08:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fecdf7fe640 1 -- 172.21.15.116:0/834426692 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fecc8001670 con 0x7fece00b4700 2023-11-28T17:08:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fece4c51640 1 --2- 172.21.15.116:0/834426692 >> v2:172.21.15.156:3300/0 conn(0x7fece0070c90 0x7fece00710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:08:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fecdf7fe640 1 -- 172.21.15.116:0/834426692 >> v1:172.21.15.116:6789/0 conn(0x7fece00b4700 legacy=0x7fece006aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fecdf7fe640 1 --2- 172.21.15.116:0/834426692 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fece015b2d0 0x7feccc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fecdf7fe640 1 --2- 172.21.15.116:0/834426692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feccc003ef0 0x7feccc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:59.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fecdf7fe640 1 --2- 172.21.15.116:0/834426692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feccc004d00 0x7feccc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:59.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fecdf7fe640 1 -- 172.21.15.116:0/834426692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece006b7d0 con 0x7feccc004d00 2023-11-28T17:08:59.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fecdf7fe640 1 -- 172.21.15.116:0/834426692 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc00b980 con 0x7feccc003ef0 2023-11-28T17:08:59.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fecdf7fe640 1 -- 172.21.15.116:0/834426692 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feccc00bb10 con 0x7fece015b2d0 2023-11-28T17:08:59.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.446+0000 7fece4c51640 1 --2- 172.21.15.116:0/834426692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feccc004d00 0x7feccc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:59.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.447+0000 7fecdffff640 1 --2- 172.21.15.116:0/834426692 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fece015b2d0 0x7feccc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:59.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.447+0000 7fecd77fe640 1 --2- 172.21.15.116:0/834426692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feccc003ef0 0x7feccc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:59.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.447+0000 7fece4c51640 1 -- 172.21.15.116:0/834426692 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fece015b2d0 msgr2=0x7feccc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:59.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.447+0000 7fece4c51640 1 --2- 172.21.15.116:0/834426692 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fece015b2d0 0x7feccc003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:59.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.447+0000 7fece4c51640 1 -- 172.21.15.116:0/834426692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feccc003ef0 msgr2=0x7feccc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:08:59.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.447+0000 7fece4c51640 1 --2- 172.21.15.116:0/834426692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feccc003ef0 0x7feccc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:59.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.447+0000 7fece4c51640 1 -- 172.21.15.116:0/834426692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc003520 con 0x7feccc004d00 2023-11-28T17:08:59.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.447+0000 7fecd77fe640 1 --2- 172.21.15.116:0/834426692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feccc003ef0 0x7feccc0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:08:59.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.448+0000 7fece4c51640 1 --2- 172.21.15.116:0/834426692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feccc004d00 0x7feccc005180 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fecd80098a0 tx=0x7feccc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f871637dd6b40d66 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:59.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.448+0000 7fecdf7fe640 1 -- 172.21.15.116:0/834426692 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8004230 con 0x7feccc004d00 2023-11-28T17:08:59.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.448+0000 7fecdf7fe640 1 -- 172.21.15.116:0/834426692 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd8004390 con 0x7feccc004d00 2023-11-28T17:08:59.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.448+0000 7fecdf7fe640 1 -- 172.21.15.116:0/834426692 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8003830 con 0x7feccc004d00 2023-11-28T17:08:59.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.449+0000 7fece66b4640 1 -- 172.21.15.116:0/834426692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feccc004d00 msgr2=0x7feccc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:59.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.449+0000 7fece66b4640 1 --2- 172.21.15.116:0/834426692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feccc004d00 0x7feccc005180 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fecd80098a0 tx=0x7feccc0057f0 comp rx=0 tx=0).stop 2023-11-28T17:08:59.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.449+0000 7fece4c51640 1 -- 172.21.15.116:0/834426692 reap_dead start 2023-11-28T17:08:59.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.449+0000 7fece66b4640 1 -- 172.21.15.116:0/834426692 shutdown_connections 2023-11-28T17:08:59.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.449+0000 7fece66b4640 1 -- 172.21.15.116:0/834426692 >> 172.21.15.116:0/834426692 conn(0x7fece00cc100 msgr2=0x7fece00cc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:08:59.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.449+0000 7fece66b4640 1 -- 172.21.15.116:0/834426692 shutdown_connections 2023-11-28T17:08:59.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.449+0000 7fece66b4640 1 -- 172.21.15.116:0/834426692 wait complete. 2023-11-28T17:08:59.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.450+0000 7fece66b4640 1 Processor -- start 2023-11-28T17:08:59.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.451+0000 7fece66b4640 1 -- start start 2023-11-28T17:08:59.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.451+0000 7fece66b4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feccc004d00 0x7fece0156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:59.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.452+0000 7fece66b4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fece0159a70 0x7fece01e3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:59.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.452+0000 7fece66b4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece01f26e0 0x7fece01f2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:59.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.452+0000 7fece66b4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece0157160 con 0x7fece01f26e0 2023-11-28T17:08:59.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.452+0000 7fece66b4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece01e3d80 con 0x7feccc004d00 2023-11-28T17:08:59.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.452+0000 7fece66b4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece01e4060 con 0x7fece0159a70 2023-11-28T17:08:59.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.452+0000 7fece4c51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece01f26e0 0x7fece01f2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:59.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.452+0000 7fece4c51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece01f26e0 0x7fece01f2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43556/0 (socket says 172.21.15.116:43556) 2023-11-28T17:08:59.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.452+0000 7fece4c51640 1 -- 172.21.15.116:0/3653278645 learned_addr learned my addr 172.21.15.116:0/3653278645 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:08:59.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.452+0000 7fece4c51640 1 -- 172.21.15.116:0/3653278645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fece0159a70 msgr2=0x7fece01e3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:59.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.452+0000 7fece4c51640 1 --2- 172.21.15.116:0/3653278645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fece0159a70 0x7fece01e3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:59.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.452+0000 7fece4c51640 1 -- 172.21.15.116:0/3653278645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feccc004d00 msgr2=0x7fece0156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:08:59.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.452+0000 7fece4c51640 1 --2- 172.21.15.116:0/3653278645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feccc004d00 0x7fece0156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:08:59.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.452+0000 7fece4c51640 1 -- 172.21.15.116:0/3653278645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece01e4340 con 0x7fece01f26e0 2023-11-28T17:08:59.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.452+0000 7fece4c51640 1 --2- 172.21.15.116:0/3653278645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece01f26e0 0x7fece01f2b90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fecd802bff0 tx=0x7fecd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:59.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.453+0000 7fecdd7fa640 1 -- 172.21.15.116:0/3653278645 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8042600 con 0x7fece01f26e0 2023-11-28T17:08:59.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.453+0000 7fecdd7fa640 1 -- 172.21.15.116:0/3653278645 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd8042760 con 0x7fece01f26e0 2023-11-28T17:08:59.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.453+0000 7fecdd7fa640 1 -- 172.21.15.116:0/3653278645 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8042ad0 con 0x7fece01f26e0 2023-11-28T17:08:59.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.453+0000 7fece66b4640 1 -- 172.21.15.116:0/3653278645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece01e4620 con 0x7fece01f26e0 2023-11-28T17:08:59.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.453+0000 7fece66b4640 1 -- 172.21.15.116:0/3653278645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece01e39c0 con 0x7fece01f26e0 2023-11-28T17:08:59.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.454+0000 7fece66b4640 1 -- 172.21.15.116:0/3653278645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece01e4b40 con 0x7fece01f26e0 2023-11-28T17:08:59.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.455+0000 7fecdd7fa640 1 -- 172.21.15.116:0/3653278645 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fecd8042c30 con 0x7fece01f26e0 2023-11-28T17:08:59.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.456+0000 7fecdd7fa640 1 --2- 172.21.15.116:0/3653278645 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fecc006b2d0 0x7fecc006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:08:59.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.456+0000 7fecdd7fa640 1 -- 172.21.15.116:0/3653278645 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fecd80ab1a0 con 0x7fece01f26e0 2023-11-28T17:08:59.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.456+0000 7fecdffff640 1 --2- 172.21.15.116:0/3653278645 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fecc006b2d0 0x7fecc006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:08:59.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.457+0000 7fecdffff640 1 --2- 172.21.15.116:0/3653278645 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fecc006b2d0 0x7fecc006d790 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fece006eb60 tx=0x7fecd0009750 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:08:59.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.461+0000 7fecdd7fa640 1 -- 172.21.15.116:0/3653278645 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fece01e4b40 con 0x7fece01f26e0 2023-11-28T17:08:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:08:59.640+0000 7fece66b4640 1 -- 172.21.15.116:0/3653278645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fece0157160 con 0x7fece01f26e0 2023-11-28T17:09:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.224+0000 7fecdd7fa640 1 -- 172.21.15.116:0/3653278645 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7fecd8076200 con 0x7fece01f26e0 2023-11-28T17:09:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' created 2023-11-28T17:09:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.228+0000 7fece66b4640 1 -- 172.21.15.116:0/3653278645 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fecc006b2d0 msgr2=0x7fecc006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.228+0000 7fece66b4640 1 --2- 172.21.15.116:0/3653278645 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fecc006b2d0 0x7fecc006d790 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fece006eb60 tx=0x7fecd0009750 comp rx=0 tx=0).stop 2023-11-28T17:09:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.228+0000 7fece66b4640 1 -- 172.21.15.116:0/3653278645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece01f26e0 msgr2=0x7fece01f2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.228+0000 7fece66b4640 1 --2- 172.21.15.116:0/3653278645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece01f26e0 0x7fece01f2b90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fecd802bff0 tx=0x7fecd8004730 comp rx=0 tx=0).stop 2023-11-28T17:09:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.228+0000 7fece66b4640 1 -- 172.21.15.116:0/3653278645 shutdown_connections 2023-11-28T17:09:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.228+0000 7fece66b4640 1 --2- 172.21.15.116:0/3653278645 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fecc006b2d0 0x7fecc006d790 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.228+0000 7fece66b4640 1 --2- 172.21.15.116:0/3653278645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece01f26e0 0x7fece01f2b90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.228+0000 7fece66b4640 1 --2- 172.21.15.116:0/3653278645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fece0159a70 0x7fece01e3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.228+0000 7fece66b4640 1 --2- 172.21.15.116:0/3653278645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feccc004d00 0x7fece0156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.228+0000 7fece66b4640 1 -- 172.21.15.116:0/3653278645 >> 172.21.15.116:0/3653278645 conn(0x7fece00cc100 msgr2=0x7fece00b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.229+0000 7fece66b4640 1 -- 172.21.15.116:0/3653278645 shutdown_connections 2023-11-28T17:09:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.229+0000 7fece66b4640 1 -- 172.21.15.116:0/3653278645 wait complete. 2023-11-28T17:09:00.260 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T17:09:00.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.441+0000 7f340c724640 1 Processor -- start 2023-11-28T17:09:00.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.441+0000 7f340c724640 1 -- start start 2023-11-28T17:09:00.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.443+0000 7f340c724640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f340415b0c0 0x7f340415ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:00.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.443+0000 7f340c724640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3404072250 0x7f34040cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:00.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.443+0000 7f340c724640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34040cd320 con 0x7f34040ce940 2023-11-28T17:09:00.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.443+0000 7f340c724640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34040ced80 con 0x7f3404072250 2023-11-28T17:09:00.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.443+0000 7f340c724640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34040cbee0 con 0x7f340415b0c0 2023-11-28T17:09:00.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.443+0000 7f340a4c0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f340415b0c0 0x7f340415ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:00.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.444+0000 7f3409cbf640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f34040ce940 0x7f3404071b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.116:35442/0 (socket says 172.21.15.116:35442) 2023-11-28T17:09:00.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.444+0000 7f3409cbf640 1 -- 172.21.15.116:0/1771456332 learned_addr learned my addr 172.21.15.116:0/1771456332 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:00.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.444+0000 7f340acc1640 1 --2- 172.21.15.116:0/1771456332 >> v2:172.21.15.156:3300/0 conn(0x7f3404072250 0x7f34040cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:00.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.444+0000 7f340a4c0640 1 -- 172.21.15.116:0/1771456332 >> v2:172.21.15.156:3300/0 conn(0x7f3404072250 msgr2=0x7f34040cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:00.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.444+0000 7f340a4c0640 1 --2- 172.21.15.116:0/1771456332 >> v2:172.21.15.156:3300/0 conn(0x7f3404072250 0x7f34040cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:00.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.444+0000 7f34094be640 1 -- 172.21.15.116:0/1771456332 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117956519 0 0) 0x7f34040cd320 con 0x7f34040ce940 2023-11-28T17:09:00.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.444+0000 7f340a4c0640 1 -- 172.21.15.116:0/1771456332 >> v1:172.21.15.156:6789/0 conn(0x7f34040ce940 legacy=0x7f3404071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:00.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.444+0000 7f340a4c0640 1 -- 172.21.15.116:0/1771456332 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34040cc710 con 0x7f340415b0c0 2023-11-28T17:09:00.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.446+0000 7f340a4c0640 1 --2- 172.21.15.116:0/1771456332 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f340415b0c0 0x7f340415ed10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f34000097a0 tx=0x7f340002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93fe54e2ecf715f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:00.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.446+0000 7f34094be640 1 -- 172.21.15.116:0/1771456332 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340002e070 con 0x7f340415b0c0 2023-11-28T17:09:00.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.446+0000 7f34094be640 1 -- 172.21.15.116:0/1771456332 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34000315a0 con 0x7f340415b0c0 2023-11-28T17:09:00.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.446+0000 7f34094be640 1 -- 172.21.15.116:0/1771456332 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3400031930 con 0x7f340415b0c0 2023-11-28T17:09:00.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.447+0000 7f340c724640 1 -- 172.21.15.116:0/1771456332 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f340415b0c0 msgr2=0x7f340415ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:00.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.447+0000 7f340c724640 1 --2- 172.21.15.116:0/1771456332 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f340415b0c0 0x7f340415ed10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f34000097a0 tx=0x7f340002b840 comp rx=0 tx=0).stop 2023-11-28T17:09:00.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.447+0000 7f340c724640 1 -- 172.21.15.116:0/1771456332 shutdown_connections 2023-11-28T17:09:00.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.447+0000 7f340c724640 1 --2- 172.21.15.116:0/1771456332 >> v2:172.21.15.156:3300/0 conn(0x7f3404072250 0x7f34040cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:00.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.447+0000 7f340c724640 1 --2- 172.21.15.116:0/1771456332 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f340415b0c0 0x7f340415ed10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:00.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.447+0000 7f340c724640 1 -- 172.21.15.116:0/1771456332 >> 172.21.15.116:0/1771456332 conn(0x7f34040b0cf0 msgr2=0x7f34040b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:00.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.447+0000 7f340c724640 1 -- 172.21.15.116:0/1771456332 shutdown_connections 2023-11-28T17:09:00.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.447+0000 7f340c724640 1 -- 172.21.15.116:0/1771456332 wait complete. 2023-11-28T17:09:00.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.449+0000 7f340c724640 1 Processor -- start 2023-11-28T17:09:00.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.449+0000 7f340c724640 1 -- start start 2023-11-28T17:09:00.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.449+0000 7f340c724640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3404072250 0x7f3404148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:00.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.450+0000 7f340c724640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34040ce940 0x7f34041591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:00.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.450+0000 7f340c724640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f340415b0c0 0x7f34040ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:00.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.450+0000 7f340c724640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34041490e0 con 0x7f3404072250 2023-11-28T17:09:00.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.450+0000 7f340c724640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34040b3270 con 0x7f340415b0c0 2023-11-28T17:09:00.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.450+0000 7f340c724640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34040b3550 con 0x7f34040ce940 2023-11-28T17:09:00.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.450+0000 7f340a4c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3404072250 0x7f3404148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:00.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.450+0000 7f340a4c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3404072250 0x7f3404148830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43558/0 (socket says 172.21.15.116:43558) 2023-11-28T17:09:00.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.450+0000 7f340a4c0640 1 -- 172.21.15.116:0/3149421780 learned_addr learned my addr 172.21.15.116:0/3149421780 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:00.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.450+0000 7f340acc1640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f340415b0c0 0x7f34040ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:00.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.450+0000 7f3409cbf640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34040ce940 0x7f34041591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:00.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.450+0000 7f340a4c0640 1 -- 172.21.15.116:0/3149421780 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34040ce940 msgr2=0x7f34041591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:00.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.450+0000 7f340a4c0640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34040ce940 0x7f34041591f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:00.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.450+0000 7f340a4c0640 1 -- 172.21.15.116:0/3149421780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f340415b0c0 msgr2=0x7f34040ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:00.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.450+0000 7f340a4c0640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f340415b0c0 0x7f34040ad8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:00.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.451+0000 7f340a4c0640 1 -- 172.21.15.116:0/3149421780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3400009400 con 0x7f3404072250 2023-11-28T17:09:00.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.451+0000 7f3409cbf640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34040ce940 0x7f34041591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:09:00.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.451+0000 7f340a4c0640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3404072250 0x7f3404148830 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3400034040 tx=0x7f34000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.451+0000 7f33faffd640 1 -- 172.21.15.116:0/3149421780 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3400032640 con 0x7f3404072250 2023-11-28T17:09:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.451+0000 7f33faffd640 1 -- 172.21.15.116:0/3149421780 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34000327c0 con 0x7f3404072250 2023-11-28T17:09:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.451+0000 7f33faffd640 1 -- 172.21.15.116:0/3149421780 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3400032b70 con 0x7f3404072250 2023-11-28T17:09:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.451+0000 7f340c724640 1 -- 172.21.15.116:0/3149421780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34040b3830 con 0x7f3404072250 2023-11-28T17:09:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.451+0000 7f340c724640 1 -- 172.21.15.116:0/3149421780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34040b3df0 con 0x7f3404072250 2023-11-28T17:09:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.453+0000 7f340c724640 1 -- 172.21.15.116:0/3149421780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34040b3fd0 con 0x7f3404072250 2023-11-28T17:09:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.454+0000 7f33faffd640 1 -- 172.21.15.116:0/3149421780 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3400032cd0 con 0x7f3404072250 2023-11-28T17:09:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.455+0000 7f33faffd640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f33e806b2a0 0x7f33e806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.455+0000 7f33faffd640 1 -- 172.21.15.116:0/3149421780 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f34000ab2f0 con 0x7f3404072250 2023-11-28T17:09:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.455+0000 7f3409cbf640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f33e806b2a0 0x7f33e806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.456+0000 7f3409cbf640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f33e806b2a0 0x7f33e806d760 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f33f4000f60 tx=0x7f33f4007c10 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:00.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.459+0000 7f33faffd640 1 -- 172.21.15.116:0/3149421780 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3400076400 con 0x7f3404072250 2023-11-28T17:09:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:00.671+0000 7f340c724640 1 -- 172.21.15.116:0/3149421780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3404069360 con 0x7f3404072250 2023-11-28T17:09:01.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.247+0000 7f33faffd640 1 -- 172.21.15.116:0/3149421780 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f3400079a20 con 0x7f3404072250 2023-11-28T17:09:01.248 INFO:teuthology.orchestra.run.smithi116.stderr:new fs with metadata pool 2 and data pool 3 2023-11-28T17:09:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.250+0000 7f340c724640 1 -- 172.21.15.116:0/3149421780 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f33e806b2a0 msgr2=0x7f33e806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.250+0000 7f340c724640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f33e806b2a0 0x7f33e806d760 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f33f4000f60 tx=0x7f33f4007c10 comp rx=0 tx=0).stop 2023-11-28T17:09:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.250+0000 7f340c724640 1 -- 172.21.15.116:0/3149421780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3404072250 msgr2=0x7f3404148830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.250+0000 7f340c724640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3404072250 0x7f3404148830 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3400034040 tx=0x7f34000046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.251+0000 7f340c724640 1 -- 172.21.15.116:0/3149421780 shutdown_connections 2023-11-28T17:09:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.251+0000 7f340c724640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f33e806b2a0 0x7f33e806d760 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.251+0000 7f340c724640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f340415b0c0 0x7f34040ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.251+0000 7f340c724640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34040ce940 0x7f34041591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.251+0000 7f340c724640 1 --2- 172.21.15.116:0/3149421780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3404072250 0x7f3404148830 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.251+0000 7f340c724640 1 -- 172.21.15.116:0/3149421780 >> 172.21.15.116:0/3149421780 conn(0x7f34040b0cf0 msgr2=0x7f34040cb420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:01.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.251+0000 7f340c724640 1 -- 172.21.15.116:0/3149421780 shutdown_connections 2023-11-28T17:09:01.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.251+0000 7f340c724640 1 -- 172.21.15.116:0/3149421780 wait complete. 2023-11-28T17:09:01.277 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:01.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.466+0000 7f6196456640 1 Processor -- start 2023-11-28T17:09:01.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.467+0000 7f6196456640 1 -- start start 2023-11-28T17:09:01.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.468+0000 7f6196456640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6190071710 0x7f619006c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.468+0000 7f6196456640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f619006c790 0x7f61900b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.468+0000 7f6196456640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619006cbe0 con 0x7f61901401c0 2023-11-28T17:09:01.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.468+0000 7f6196456640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61901405a0 con 0x7f6190071710 2023-11-28T17:09:01.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.468+0000 7f6196456640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6190140880 con 0x7f619006c790 2023-11-28T17:09:01.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.468+0000 7f618ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f61901401c0 0x7f6190071eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45548/0 (socket says 172.21.15.116:45548) 2023-11-28T17:09:01.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.468+0000 7f618ffff640 1 -- 172.21.15.116:0/1390381768 learned_addr learned my addr 172.21.15.116:0/1390381768 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:01.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.468+0000 7f61877fe640 1 --2- 172.21.15.116:0/1390381768 >> v2:172.21.15.116:3300/0 conn(0x7f6190071710 0x7f619006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:01.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.469+0000 7f61949f3640 1 --2- 172.21.15.116:0/1390381768 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f619006c790 0x7f61900b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:01.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.469+0000 7f61877fe640 1 -- 172.21.15.116:0/1390381768 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f619006c790 msgr2=0x7f61900b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:01.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.469+0000 7f61877fe640 1 --2- 172.21.15.116:0/1390381768 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f619006c790 0x7f61900b3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.469+0000 7f61877fe640 1 -- 172.21.15.116:0/1390381768 >> v1:172.21.15.116:6789/0 conn(0x7f61901401c0 legacy=0x7f6190071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:01.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.469+0000 7f61877fe640 1 -- 172.21.15.116:0/1390381768 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619015db00 con 0x7f6190071710 2023-11-28T17:09:01.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.469+0000 7f618f7fe640 1 -- 172.21.15.116:0/1390381768 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787402516 0 0) 0x7f6180001110 con 0x7f61901401c0 2023-11-28T17:09:01.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.470+0000 7f61877fe640 1 --2- 172.21.15.116:0/1390381768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6190071710 0x7f619006c250 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f61780097a0 tx=0x7f617802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb9f6e025163e1de server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:01.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.470+0000 7f618f7fe640 1 -- 172.21.15.116:0/1390381768 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6178031830 con 0x7f6190071710 2023-11-28T17:09:01.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.470+0000 7f618f7fe640 1 -- 172.21.15.116:0/1390381768 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6178031990 con 0x7f6190071710 2023-11-28T17:09:01.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.470+0000 7f618f7fe640 1 -- 172.21.15.116:0/1390381768 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6178031d20 con 0x7f6190071710 2023-11-28T17:09:01.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.471+0000 7f6196456640 1 -- 172.21.15.116:0/1390381768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6190071710 msgr2=0x7f619006c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:01.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.471+0000 7f6196456640 1 --2- 172.21.15.116:0/1390381768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6190071710 0x7f619006c250 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f61780097a0 tx=0x7f617802b8d0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.471+0000 7f6196456640 1 -- 172.21.15.116:0/1390381768 shutdown_connections 2023-11-28T17:09:01.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.471+0000 7f6196456640 1 --2- 172.21.15.116:0/1390381768 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f619006c790 0x7f61900b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.471+0000 7f6196456640 1 --2- 172.21.15.116:0/1390381768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6190071710 0x7f619006c250 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.471+0000 7f6196456640 1 -- 172.21.15.116:0/1390381768 >> 172.21.15.116:0/1390381768 conn(0x7f619006a4c0 msgr2=0x7f619006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:01.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.471+0000 7f6196456640 1 -- 172.21.15.116:0/1390381768 shutdown_connections 2023-11-28T17:09:01.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.471+0000 7f6196456640 1 -- 172.21.15.116:0/1390381768 wait complete. 2023-11-28T17:09:01.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.473+0000 7f6196456640 1 Processor -- start 2023-11-28T17:09:01.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.473+0000 7f6196456640 1 -- start start 2023-11-28T17:09:01.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.474+0000 7f6196456640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f619006c790 0x7f619013c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.474+0000 7f6196456640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6190071710 0x7f6190159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.474+0000 7f6196456640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61901401c0 0x7f619013e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.474+0000 7f6196456640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619015b7f0 con 0x7f61901401c0 2023-11-28T17:09:01.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.474+0000 7f6196456640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619013cdb0 con 0x7f6190071710 2023-11-28T17:09:01.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.474+0000 7f6196456640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619015a370 con 0x7f619006c790 2023-11-28T17:09:01.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.474+0000 7f61949f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61901401c0 0x7f619013e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.474+0000 7f61949f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61901401c0 0x7f619013e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43584/0 (socket says 172.21.15.116:43584) 2023-11-28T17:09:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.474+0000 7f61949f3640 1 -- 172.21.15.116:0/4210937883 learned_addr learned my addr 172.21.15.116:0/4210937883 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.474+0000 7f61949f3640 1 -- 172.21.15.116:0/4210937883 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f619006c790 msgr2=0x7f619013c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.474+0000 7f61949f3640 1 --2- 172.21.15.116:0/4210937883 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f619006c790 0x7f619013c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.474+0000 7f61949f3640 1 -- 172.21.15.116:0/4210937883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6190071710 msgr2=0x7f6190159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.474+0000 7f61877fe640 1 --2- 172.21.15.116:0/4210937883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6190071710 0x7f6190159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.475+0000 7f61949f3640 1 --2- 172.21.15.116:0/4210937883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6190071710 0x7f6190159a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.475+0000 7f61949f3640 1 -- 172.21.15.116:0/4210937883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6178009400 con 0x7f61901401c0 2023-11-28T17:09:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.475+0000 7f61949f3640 1 --2- 172.21.15.116:0/4210937883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61901401c0 0x7f619013e000 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f618800cda0 tx=0x7f618800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.475+0000 7f618d7fa640 1 -- 172.21.15.116:0/4210937883 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188003b90 con 0x7f61901401c0 2023-11-28T17:09:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.475+0000 7f618d7fa640 1 -- 172.21.15.116:0/4210937883 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6188003cf0 con 0x7f61901401c0 2023-11-28T17:09:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.475+0000 7f6196456640 1 -- 172.21.15.116:0/4210937883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619015a650 con 0x7f61901401c0 2023-11-28T17:09:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.475+0000 7f618d7fa640 1 -- 172.21.15.116:0/4210937883 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188004240 con 0x7f61901401c0 2023-11-28T17:09:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.475+0000 7f6196456640 1 -- 172.21.15.116:0/4210937883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619015b480 con 0x7f61901401c0 2023-11-28T17:09:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.476+0000 7f6196456640 1 -- 172.21.15.116:0/4210937883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61900726b0 con 0x7f61901401c0 2023-11-28T17:09:01.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.477+0000 7f618d7fa640 1 -- 172.21.15.116:0/4210937883 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f61880043a0 con 0x7f61901401c0 2023-11-28T17:09:01.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.478+0000 7f618d7fa640 1 --2- 172.21.15.116:0/4210937883 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f617006b2a0 0x7f617006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.478+0000 7f618d7fa640 1 -- 172.21.15.116:0/4210937883 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6188012070 con 0x7f61901401c0 2023-11-28T17:09:01.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.478+0000 7f618ffff640 1 --2- 172.21.15.116:0/4210937883 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f617006b2a0 0x7f617006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:01.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.479+0000 7f618ffff640 1 --2- 172.21.15.116:0/4210937883 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f617006b2a0 0x7f617006d760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6180007fd0 tx=0x7f6180007960 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:01.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.482+0000 7f618d7fa640 1 -- 172.21.15.116:0/4210937883 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6188055400 con 0x7f61901401c0 2023-11-28T17:09:01.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.661+0000 7f6196456640 1 -- 172.21.15.116:0/4210937883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f619015b9a0 con 0x7f61901401c0 2023-11-28T17:09:01.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.662+0000 7f618d7fa640 1 -- 172.21.15.116:0/4210937883 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16534 (secure 0 0 0) 0x7f619015b9a0 con 0x7f61901401c0 2023-11-28T17:09:01.664 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:01.665 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:00.672792+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:01.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.665+0000 7f6196456640 1 -- 172.21.15.116:0/4210937883 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f617006b2a0 msgr2=0x7f617006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:01.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.665+0000 7f6196456640 1 --2- 172.21.15.116:0/4210937883 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f617006b2a0 0x7f617006d760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6180007fd0 tx=0x7f6180007960 comp rx=0 tx=0).stop 2023-11-28T17:09:01.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.665+0000 7f6196456640 1 -- 172.21.15.116:0/4210937883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61901401c0 msgr2=0x7f619013e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:01.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.665+0000 7f6196456640 1 --2- 172.21.15.116:0/4210937883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61901401c0 0x7f619013e000 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f618800cda0 tx=0x7f618800a660 comp rx=0 tx=0).stop 2023-11-28T17:09:01.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.665+0000 7f6196456640 1 -- 172.21.15.116:0/4210937883 shutdown_connections 2023-11-28T17:09:01.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.665+0000 7f6196456640 1 --2- 172.21.15.116:0/4210937883 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f617006b2a0 0x7f617006d760 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.665+0000 7f6196456640 1 --2- 172.21.15.116:0/4210937883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61901401c0 0x7f619013e000 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.665+0000 7f6196456640 1 --2- 172.21.15.116:0/4210937883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6190071710 0x7f6190159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.665+0000 7f6196456640 1 --2- 172.21.15.116:0/4210937883 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f619006c790 0x7f619013c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.665+0000 7f6196456640 1 -- 172.21.15.116:0/4210937883 >> 172.21.15.116:0/4210937883 conn(0x7f619006a4c0 msgr2=0x7f61900b3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:01.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.665+0000 7f6196456640 1 -- 172.21.15.116:0/4210937883 shutdown_connections 2023-11-28T17:09:01.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.665+0000 7f6196456640 1 -- 172.21.15.116:0/4210937883 wait complete. 2023-11-28T17:09:01.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:01.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.891+0000 7f564adff640 1 Processor -- start 2023-11-28T17:09:01.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.891+0000 7f564adff640 1 -- start start 2023-11-28T17:09:01.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.893+0000 7f564adff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f564415d3b0 0x7f56440abd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.893+0000 7f564adff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56440ce470 0x7f5644071230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.893+0000 7f564adff640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56440cc7d0 con 0x7f56440ceb70 2023-11-28T17:09:01.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.893+0000 7f564adff640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56440ce920 con 0x7f564415d3b0 2023-11-28T17:09:01.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.893+0000 7f564adff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5644071770 con 0x7f56440ce470 2023-11-28T17:09:01.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.893+0000 7f5649dfd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f56440ceb70 0x7f56440cef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45562/0 (socket says 172.21.15.116:45562) 2023-11-28T17:09:01.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.893+0000 7f5649dfd640 1 -- 172.21.15.116:0/840988931 learned_addr learned my addr 172.21.15.116:0/840988931 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:01.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f56495fc640 1 --2- 172.21.15.116:0/840988931 >> v2:172.21.15.156:3300/0 conn(0x7f564415d3b0 0x7f56440abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:01.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f564a5fe640 1 --2- 172.21.15.116:0/840988931 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56440ce470 0x7f5644071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:01.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 -- 172.21.15.116:0/840988931 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779597356 0 0) 0x7f56440cc7d0 con 0x7f56440ceb70 2023-11-28T17:09:01.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 -- 172.21.15.116:0/840988931 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5630003520 con 0x7f56440ceb70 2023-11-28T17:09:01.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 -- 172.21.15.116:0/840988931 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3026477576 0 0) 0x7f5630003520 con 0x7f56440ceb70 2023-11-28T17:09:01.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 -- 172.21.15.116:0/840988931 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56440ce470 msgr2=0x7f5644071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:01.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 --2- 172.21.15.116:0/840988931 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56440ce470 0x7f5644071230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 -- 172.21.15.116:0/840988931 >> v2:172.21.15.156:3300/0 conn(0x7f564415d3b0 msgr2=0x7f56440abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:01.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 --2- 172.21.15.116:0/840988931 >> v2:172.21.15.156:3300/0 conn(0x7f564415d3b0 0x7f56440abd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 -- 172.21.15.116:0/840988931 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5644071a30 con 0x7f56440ceb70 2023-11-28T17:09:01.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 -- 172.21.15.116:0/840988931 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f5640001670 con 0x7f56440ceb70 2023-11-28T17:09:01.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 -- 172.21.15.116:0/840988931 >> v1:172.21.15.116:6789/0 conn(0x7f56440ceb70 legacy=0x7f56440cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:01.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 --2- 172.21.15.116:0/840988931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5630003970 0x7f5630003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 --2- 172.21.15.116:0/840988931 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56440ce470 0x7f56300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 --2- 172.21.15.116:0/840988931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5630004d00 0x7f5630005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 -- 172.21.15.116:0/840988931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56440ce920 con 0x7f5630003970 2023-11-28T17:09:01.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 -- 172.21.15.116:0/840988931 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5644071770 con 0x7f5630004d00 2023-11-28T17:09:01.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.894+0000 7f5648dfb640 1 -- 172.21.15.116:0/840988931 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563000b9b0 con 0x7f56440ce470 2023-11-28T17:09:01.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.895+0000 7f5649dfd640 1 --2- 172.21.15.116:0/840988931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5630003970 0x7f5630003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:01.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.895+0000 7f564a5fe640 1 --2- 172.21.15.116:0/840988931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5630004d00 0x7f5630005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:01.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.895+0000 7f5649dfd640 1 -- 172.21.15.116:0/840988931 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56440ce470 msgr2=0x7f56300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:01.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.895+0000 7f56495fc640 1 --2- 172.21.15.116:0/840988931 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56440ce470 0x7f56300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:01.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.895+0000 7f5649dfd640 1 --2- 172.21.15.116:0/840988931 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56440ce470 0x7f56300047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.895+0000 7f5649dfd640 1 -- 172.21.15.116:0/840988931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5630004d00 msgr2=0x7f5630005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:01.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.895+0000 7f5649dfd640 1 --2- 172.21.15.116:0/840988931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5630004d00 0x7f5630005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.895+0000 7f5649dfd640 1 -- 172.21.15.116:0/840988931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5630003520 con 0x7f5630003970 2023-11-28T17:09:01.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.895+0000 7f564a5fe640 1 --2- 172.21.15.116:0/840988931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5630004d00 0x7f5630005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:01.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.896+0000 7f5649dfd640 1 --2- 172.21.15.116:0/840988931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5630003970 0x7f5630003d90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f564000c240 tx=0x7f56300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2fb3d7c4d4f5c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:01.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.896+0000 7f5648dfb640 1 -- 172.21.15.116:0/840988931 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564002cb00 con 0x7f5630003970 2023-11-28T17:09:01.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.896+0000 7f5648dfb640 1 -- 172.21.15.116:0/840988931 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564002cc60 con 0x7f5630003970 2023-11-28T17:09:01.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.896+0000 7f5648dfb640 1 -- 172.21.15.116:0/840988931 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5640005af0 con 0x7f5630003970 2023-11-28T17:09:01.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.897+0000 7f564adff640 1 -- 172.21.15.116:0/840988931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5630003970 msgr2=0x7f5630003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:01.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.897+0000 7f564adff640 1 --2- 172.21.15.116:0/840988931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5630003970 0x7f5630003d90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f564000c240 tx=0x7f56300057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.897+0000 7f564a5fe640 1 -- 172.21.15.116:0/840988931 reap_dead start 2023-11-28T17:09:01.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.897+0000 7f564adff640 1 -- 172.21.15.116:0/840988931 shutdown_connections 2023-11-28T17:09:01.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.897+0000 7f564adff640 1 -- 172.21.15.116:0/840988931 >> 172.21.15.116:0/840988931 conn(0x7f56440b0950 msgr2=0x7f56440b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:01.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.897+0000 7f564adff640 1 -- 172.21.15.116:0/840988931 shutdown_connections 2023-11-28T17:09:01.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.897+0000 7f564adff640 1 -- 172.21.15.116:0/840988931 wait complete. 2023-11-28T17:09:01.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.899+0000 7f564adff640 1 Processor -- start 2023-11-28T17:09:01.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.899+0000 7f564adff640 1 -- start start 2023-11-28T17:09:01.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.900+0000 7f564adff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5630003970 0x7f564415b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.900+0000 7f564adff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f564409c420 0x7f564414f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.900+0000 7f564adff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56441f30f0 0x7f56440af2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.900+0000 7f564adff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56441f2910 con 0x7f56441f30f0 2023-11-28T17:09:01.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.900+0000 7f564adff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564415b7f0 con 0x7f564409c420 2023-11-28T17:09:01.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.900+0000 7f564adff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564415bad0 con 0x7f5630003970 2023-11-28T17:09:01.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.900+0000 7f564a5fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56441f30f0 0x7f56440af2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:01.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.900+0000 7f564a5fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56441f30f0 0x7f56440af2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43602/0 (socket says 172.21.15.116:43602) 2023-11-28T17:09:01.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.900+0000 7f564a5fe640 1 -- 172.21.15.116:0/3038917522 learned_addr learned my addr 172.21.15.116:0/3038917522 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:01.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.901+0000 7f5649dfd640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5630003970 0x7f564415b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:01.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.901+0000 7f56495fc640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f564409c420 0x7f564414f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:01.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.901+0000 7f564a5fe640 1 -- 172.21.15.116:0/3038917522 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5630003970 msgr2=0x7f564415b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:01.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.901+0000 7f564a5fe640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5630003970 0x7f564415b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.901+0000 7f564a5fe640 1 -- 172.21.15.116:0/3038917522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f564409c420 msgr2=0x7f564414f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:01.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.901+0000 7f564a5fe640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f564409c420 0x7f564414f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:01.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.901+0000 7f564a5fe640 1 -- 172.21.15.116:0/3038917522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564415bdb0 con 0x7f56441f30f0 2023-11-28T17:09:01.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.901+0000 7f5649dfd640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5630003970 0x7f564415b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:01.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.901+0000 7f56495fc640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f564409c420 0x7f564414f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:01.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.901+0000 7f564a5fe640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56441f30f0 0x7f56440af2a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f56440ae810 tx=0x7f563c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:01.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.901+0000 7f563a7fc640 1 -- 172.21.15.116:0/3038917522 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c017400 con 0x7f56441f30f0 2023-11-28T17:09:01.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.901+0000 7f563a7fc640 1 -- 172.21.15.116:0/3038917522 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563c011040 con 0x7f56441f30f0 2023-11-28T17:09:01.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.901+0000 7f563a7fc640 1 -- 172.21.15.116:0/3038917522 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c0177c0 con 0x7f56441f30f0 2023-11-28T17:09:01.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.901+0000 7f564adff640 1 -- 172.21.15.116:0/3038917522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56441f2bd0 con 0x7f56441f30f0 2023-11-28T17:09:01.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.902+0000 7f564adff640 1 -- 172.21.15.116:0/3038917522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564414fc60 con 0x7f56441f30f0 2023-11-28T17:09:01.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.903+0000 7f564adff640 1 -- 172.21.15.116:0/3038917522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56441f2dc0 con 0x7f56441f30f0 2023-11-28T17:09:01.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.904+0000 7f563a7fc640 1 -- 172.21.15.116:0/3038917522 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f563c0203b0 con 0x7f56441f30f0 2023-11-28T17:09:01.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.904+0000 7f563a7fc640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f562806b2a0 0x7f562806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:01.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.905+0000 7f563a7fc640 1 -- 172.21.15.116:0/3038917522 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f563c089f30 con 0x7f56441f30f0 2023-11-28T17:09:01.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.905+0000 7f5649dfd640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f562806b2a0 0x7f562806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:01.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.905+0000 7f5649dfd640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f562806b2a0 0x7f562806d760 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5640034000 tx=0x7f5640007000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:01.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:01.909+0000 7f563a7fc640 1 -- 172.21.15.116:0/3038917522 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f563c054d90 con 0x7f56441f30f0 2023-11-28T17:09:02.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.091+0000 7f564adff640 1 -- 172.21.15.116:0/3038917522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f564409d0b0 con 0x7f56441f30f0 2023-11-28T17:09:02.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.093+0000 7f563a7fc640 1 -- 172.21.15.116:0/3038917522 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16521 (secure 0 0 0) 0x7f563c0583b0 con 0x7f56441f30f0 2023-11-28T17:09:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:02.095 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:01.970957+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:02.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.096+0000 7f564adff640 1 -- 172.21.15.116:0/3038917522 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f562806b2a0 msgr2=0x7f562806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:02.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.096+0000 7f564adff640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f562806b2a0 0x7f562806d760 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5640034000 tx=0x7f5640007000 comp rx=0 tx=0).stop 2023-11-28T17:09:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.096+0000 7f564adff640 1 -- 172.21.15.116:0/3038917522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56441f30f0 msgr2=0x7f56440af2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.096+0000 7f564adff640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56441f30f0 0x7f56440af2a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f56440ae810 tx=0x7f563c00f4a0 comp rx=0 tx=0).stop 2023-11-28T17:09:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.096+0000 7f564adff640 1 -- 172.21.15.116:0/3038917522 shutdown_connections 2023-11-28T17:09:02.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.096+0000 7f564adff640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f562806b2a0 0x7f562806d760 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:02.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.096+0000 7f564adff640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56441f30f0 0x7f56440af2a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:02.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.096+0000 7f564adff640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f564409c420 0x7f564414f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:02.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.096+0000 7f564adff640 1 --2- 172.21.15.116:0/3038917522 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5630003970 0x7f564415b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:02.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.096+0000 7f564adff640 1 -- 172.21.15.116:0/3038917522 >> 172.21.15.116:0/3038917522 conn(0x7f56440b0950 msgr2=0x7f56440cef50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:02.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.097+0000 7f564adff640 1 -- 172.21.15.116:0/3038917522 shutdown_connections 2023-11-28T17:09:02.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.097+0000 7f564adff640 1 -- 172.21.15.116:0/3038917522 wait complete. 2023-11-28T17:09:02.116 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T17:09:02.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.329+0000 7f360d7e5640 1 Processor -- start 2023-11-28T17:09:02.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.329+0000 7f360d7e5640 1 -- start start 2023-11-28T17:09:02.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.331+0000 7f360d7e5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f360815b0c0 0x7f360815ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:02.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.331+0000 7f360d7e5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36080cd320 con 0x7f36080ce940 2023-11-28T17:09:02.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.331+0000 7f360d7e5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36080ced80 con 0x7f3608072250 2023-11-28T17:09:02.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.331+0000 7f360d7e5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36080cc710 con 0x7f360815b0c0 2023-11-28T17:09:02.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.331+0000 7f35fe7fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f36080ce940 0x7f3608071b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45568/0 (socket says 172.21.15.116:45568) 2023-11-28T17:09:02.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.331+0000 7f35fe7fc640 1 -- 172.21.15.116:0/3242424871 learned_addr learned my addr 172.21.15.116:0/3242424871 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:02.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.331+0000 7f36067fc640 1 -- 172.21.15.116:0/3242424871 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161384131 0 0) 0x7f36080cd320 con 0x7f36080ce940 2023-11-28T17:09:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.331+0000 7f36067fc640 1 -- 172.21.15.116:0/3242424871 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35f4003520 con 0x7f36080ce940 2023-11-28T17:09:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f3606ffd640 1 --2- 172.21.15.116:0/3242424871 >> v2:172.21.15.156:3300/0 conn(0x7f360815b0c0 0x7f360815ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f36067fc640 1 -- 172.21.15.116:0/3242424871 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1851967338 0 0) 0x7f35f4003520 con 0x7f36080ce940 2023-11-28T17:09:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f36067fc640 1 -- 172.21.15.116:0/3242424871 >> v2:172.21.15.156:3300/0 conn(0x7f360815b0c0 msgr2=0x7f360815ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f36067fc640 1 --2- 172.21.15.116:0/3242424871 >> v2:172.21.15.156:3300/0 conn(0x7f360815b0c0 0x7f360815ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f36067fc640 1 -- 172.21.15.116:0/3242424871 >> v1:172.21.15.156:6789/0 conn(0x7f3608072250 legacy=0x7f360815b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f36067fc640 1 -- 172.21.15.116:0/3242424871 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36080cc9f0 con 0x7f36080ce940 2023-11-28T17:09:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f36067fc640 1 -- 172.21.15.116:0/3242424871 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f35f0001670 con 0x7f36080ce940 2023-11-28T17:09:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f36067fc640 1 -- 172.21.15.116:0/3242424871 >> v1:172.21.15.116:6789/0 conn(0x7f36080ce940 legacy=0x7f3608071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:02.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f36067fc640 1 --2- 172.21.15.116:0/3242424871 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35f40039c0 0x7f35f4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f36067fc640 1 --2- 172.21.15.116:0/3242424871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35f40047c0 0x7f35f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f36067fc640 1 --2- 172.21.15.116:0/3242424871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f40052a0 0x7f35f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f36067fc640 1 -- 172.21.15.116:0/3242424871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36080cc710 con 0x7f35f40047c0 2023-11-28T17:09:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f36067fc640 1 -- 172.21.15.116:0/3242424871 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f400c1c0 con 0x7f35f40052a0 2023-11-28T17:09:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f36067fc640 1 -- 172.21.15.116:0/3242424871 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f400c380 con 0x7f35f40039c0 2023-11-28T17:09:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.332+0000 7f35fe7fc640 1 --2- 172.21.15.116:0/3242424871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35f40047c0 0x7f35f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.333+0000 7f3606ffd640 1 --2- 172.21.15.116:0/3242424871 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35f40039c0 0x7f35f4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.333+0000 7f35fe7fc640 1 -- 172.21.15.116:0/3242424871 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35f40039c0 msgr2=0x7f35f4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.333+0000 7f35fe7fc640 1 --2- 172.21.15.116:0/3242424871 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35f40039c0 0x7f35f4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.333+0000 7f35fe7fc640 1 -- 172.21.15.116:0/3242424871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f40052a0 msgr2=0x7f35f400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.333+0000 7f35fe7fc640 1 --2- 172.21.15.116:0/3242424871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f40052a0 0x7f35f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:02.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.333+0000 7f35fe7fc640 1 -- 172.21.15.116:0/3242424871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f4003520 con 0x7f35f40047c0 2023-11-28T17:09:02.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.333+0000 7f36077fe640 1 -- 172.21.15.116:0/3242424871 reap_dead start 2023-11-28T17:09:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.334+0000 7f35fe7fc640 1 --2- 172.21.15.116:0/3242424871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35f40047c0 0x7f35f4004c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f35f000c0e0 tx=0x7f35f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abfc771467eca708 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.334+0000 7f36067fc640 1 -- 172.21.15.116:0/3242424871 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0005ad0 con 0x7f35f40047c0 2023-11-28T17:09:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.334+0000 7f36067fc640 1 -- 172.21.15.116:0/3242424871 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f0005c30 con 0x7f35f40047c0 2023-11-28T17:09:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.334+0000 7f36067fc640 1 -- 172.21.15.116:0/3242424871 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f002ca50 con 0x7f35f40047c0 2023-11-28T17:09:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.335+0000 7f360d7e5640 1 -- 172.21.15.116:0/3242424871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35f40047c0 msgr2=0x7f35f4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.335+0000 7f360d7e5640 1 --2- 172.21.15.116:0/3242424871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35f40047c0 0x7f35f4004c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f35f000c0e0 tx=0x7f35f40057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.335+0000 7f360d7e5640 1 -- 172.21.15.116:0/3242424871 shutdown_connections 2023-11-28T17:09:02.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.335+0000 7f360d7e5640 1 --2- 172.21.15.116:0/3242424871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35f40047c0 0x7f35f4004c40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.335+0000 7f360d7e5640 1 -- 172.21.15.116:0/3242424871 >> 172.21.15.116:0/3242424871 conn(0x7f36080b0cf0 msgr2=0x7f36080b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.335+0000 7f360d7e5640 1 -- 172.21.15.116:0/3242424871 shutdown_connections 2023-11-28T17:09:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.335+0000 7f360d7e5640 1 -- 172.21.15.116:0/3242424871 wait complete. 2023-11-28T17:09:02.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.337+0000 7f360d7e5640 1 Processor -- start 2023-11-28T17:09:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.337+0000 7f360d7e5640 1 -- start start 2023-11-28T17:09:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.338+0000 7f360d7e5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f40047c0 0x7f360814d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.338+0000 7f360d7e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360814f210 0x7f3608147460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.338+0000 7f360d7e5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f360815d8a0 0x7f360815dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.338+0000 7f360d7e5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360814b4f0 con 0x7f360814f210 2023-11-28T17:09:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.338+0000 7f360d7e5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360815e240 con 0x7f35f40047c0 2023-11-28T17:09:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.338+0000 7f360d7e5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f360814dc10 con 0x7f360815d8a0 2023-11-28T17:09:02.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.338+0000 7f35fe7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360814f210 0x7f3608147460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.338+0000 7f35fe7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360814f210 0x7f3608147460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43620/0 (socket says 172.21.15.116:43620) 2023-11-28T17:09:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.338+0000 7f35fe7fc640 1 -- 172.21.15.116:0/644892504 learned_addr learned my addr 172.21.15.116:0/644892504 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.339+0000 7f3606ffd640 1 --2- 172.21.15.116:0/644892504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f40047c0 0x7f360814d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.339+0000 7f36077fe640 1 --2- 172.21.15.116:0/644892504 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f360815d8a0 0x7f360815dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.339+0000 7f35fe7fc640 1 -- 172.21.15.116:0/644892504 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f360815d8a0 msgr2=0x7f360815dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.339+0000 7f35fe7fc640 1 --2- 172.21.15.116:0/644892504 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f360815d8a0 0x7f360815dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.339+0000 7f35fe7fc640 1 -- 172.21.15.116:0/644892504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f40047c0 msgr2=0x7f360814d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.339+0000 7f35fe7fc640 1 --2- 172.21.15.116:0/644892504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f40047c0 0x7f360814d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.339+0000 7f35fe7fc640 1 -- 172.21.15.116:0/644892504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360815b620 con 0x7f360814f210 2023-11-28T17:09:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.339+0000 7f35fe7fc640 1 --2- 172.21.15.116:0/644892504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360814f210 0x7f3608147460 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f35f0000c00 tx=0x7f35f002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.339+0000 7f35fffff640 1 -- 172.21.15.116:0/644892504 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f003c040 con 0x7f360814f210 2023-11-28T17:09:02.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.339+0000 7f35fffff640 1 -- 172.21.15.116:0/644892504 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f002cd00 con 0x7f360814f210 2023-11-28T17:09:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.339+0000 7f35fffff640 1 -- 172.21.15.116:0/644892504 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f00072b0 con 0x7f360814f210 2023-11-28T17:09:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.340+0000 7f360d7e5640 1 -- 172.21.15.116:0/644892504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f360814def0 con 0x7f360814f210 2023-11-28T17:09:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.340+0000 7f360d7e5640 1 -- 172.21.15.116:0/644892504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f360814e4b0 con 0x7f360814f210 2023-11-28T17:09:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.341+0000 7f360d7e5640 1 -- 172.21.15.116:0/644892504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3608150af0 con 0x7f360814f210 2023-11-28T17:09:02.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.342+0000 7f35fffff640 1 -- 172.21.15.116:0/644892504 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f35f0032070 con 0x7f360814f210 2023-11-28T17:09:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.343+0000 7f35fffff640 1 --2- 172.21.15.116:0/644892504 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f35e806b2d0 0x7f35e806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.343+0000 7f35fffff640 1 -- 172.21.15.116:0/644892504 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35f00ab0d0 con 0x7f360814f210 2023-11-28T17:09:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.343+0000 7f3606ffd640 1 --2- 172.21.15.116:0/644892504 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f35e806b2d0 0x7f35e806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:02.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.343+0000 7f3606ffd640 1 --2- 172.21.15.116:0/644892504 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f35e806b2d0 0x7f35e806d790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f35f8003fd0 tx=0x7f35f800b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:02.348 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.347+0000 7f35fffff640 1 -- 172.21.15.116:0/644892504 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35f0075ea0 con 0x7f360814f210 2023-11-28T17:09:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:02.578+0000 7f360d7e5640 1 -- 172.21.15.116:0/644892504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3608069360 con 0x7f360814f210 2023-11-28T17:09:03.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.283+0000 7f35fffff640 1 -- 172.21.15.116:0/644892504 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f35f00794c0 con 0x7f360814f210 2023-11-28T17:09:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.287+0000 7f360d7e5640 1 -- 172.21.15.116:0/644892504 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f35e806b2d0 msgr2=0x7f35e806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.287+0000 7f360d7e5640 1 --2- 172.21.15.116:0/644892504 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f35e806b2d0 0x7f35e806d790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f35f8003fd0 tx=0x7f35f800b040 comp rx=0 tx=0).stop 2023-11-28T17:09:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.287+0000 7f360d7e5640 1 -- 172.21.15.116:0/644892504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360814f210 msgr2=0x7f3608147460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.287+0000 7f360d7e5640 1 --2- 172.21.15.116:0/644892504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360814f210 0x7f3608147460 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f35f0000c00 tx=0x7f35f002e070 comp rx=0 tx=0).stop 2023-11-28T17:09:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.287+0000 7f360d7e5640 1 -- 172.21.15.116:0/644892504 shutdown_connections 2023-11-28T17:09:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.287+0000 7f360d7e5640 1 --2- 172.21.15.116:0/644892504 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f35e806b2d0 0x7f35e806d790 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.287+0000 7f360d7e5640 1 --2- 172.21.15.116:0/644892504 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f360815d8a0 0x7f360815dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.287+0000 7f360d7e5640 1 --2- 172.21.15.116:0/644892504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f360814f210 0x7f3608147460 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.287+0000 7f360d7e5640 1 --2- 172.21.15.116:0/644892504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35f40047c0 0x7f360814d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.287+0000 7f360d7e5640 1 -- 172.21.15.116:0/644892504 >> 172.21.15.116:0/644892504 conn(0x7f36080b0cf0 msgr2=0x7f360806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:03.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.287+0000 7f360d7e5640 1 -- 172.21.15.116:0/644892504 shutdown_connections 2023-11-28T17:09:03.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.288+0000 7f360d7e5640 1 -- 172.21.15.116:0/644892504 wait complete. 2023-11-28T17:09:03.309 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T17:09:03.310 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T17:09:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.524+0000 7f0ef43a5640 1 Processor -- start 2023-11-28T17:09:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.525+0000 7f0ef43a5640 1 -- start start 2023-11-28T17:09:03.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.526+0000 7f0ef43a5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0eec15d480 0x7f0eec0d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:03.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.526+0000 7f0ef43a5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0eec0d01e0 0x7f0eec160270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:03.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.526+0000 7f0ef43a5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eec15ddc0 con 0x7f0eec0d1310 2023-11-28T17:09:03.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.526+0000 7f0ef43a5640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eec15e0a0 con 0x7f0eec0d01e0 2023-11-28T17:09:03.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.526+0000 7f0ef43a5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eec15e380 con 0x7f0eec15d480 2023-11-28T17:09:03.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.526+0000 7f0ef2942640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0eec0d01e0 0x7f0eec160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:03.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.526+0000 7f0ef2942640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0eec0d01e0 0x7f0eec160270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43626/0 (socket says 172.21.15.116:43626) 2023-11-28T17:09:03.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.526+0000 7f0ef2942640 1 -- 172.21.15.116:0/1322571205 learned_addr learned my addr 172.21.15.116:0/1322571205 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:03.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.527+0000 7f0ef2141640 1 --2- 172.21.15.116:0/1322571205 >> v2:172.21.15.156:3300/0 conn(0x7f0eec15d480 0x7f0eec0d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:03.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.527+0000 7f0ef2942640 1 -- 172.21.15.116:0/1322571205 >> v2:172.21.15.156:3300/0 conn(0x7f0eec15d480 msgr2=0x7f0eec0d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:03.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.527+0000 7f0ef2942640 1 --2- 172.21.15.116:0/1322571205 >> v2:172.21.15.156:3300/0 conn(0x7f0eec15d480 0x7f0eec0d0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:03.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.527+0000 7f0ef2942640 1 -- 172.21.15.116:0/1322571205 >> v1:172.21.15.116:6789/0 conn(0x7f0eec0d1310 legacy=0x7f0eec0cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:03.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.527+0000 7f0ef2942640 1 -- 172.21.15.116:0/1322571205 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec15e660 con 0x7f0eec0d01e0 2023-11-28T17:09:03.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.527+0000 7f0ef113f640 1 -- 172.21.15.116:0/1322571205 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796790640 0 0) 0x7f0eec15ddc0 con 0x7f0eec0d1310 2023-11-28T17:09:03.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.527+0000 7f0ef2141640 1 --2- 172.21.15.116:0/1322571205 >> v2:172.21.15.156:3300/0 conn(0x7f0eec15d480 0x7f0eec0d0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:03.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.528+0000 7f0ef2942640 1 --2- 172.21.15.116:0/1322571205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eec0d01e0 0x7f0eec160270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0b5d30 tx=0x7f0ee402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444686d8d36510b6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.528+0000 7f0ef113f640 1 -- 172.21.15.116:0/1322571205 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4030860 con 0x7f0eec0d01e0 2023-11-28T17:09:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.528+0000 7f0ef113f640 1 -- 172.21.15.116:0/1322571205 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee40309c0 con 0x7f0eec0d01e0 2023-11-28T17:09:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.528+0000 7f0ef113f640 1 -- 172.21.15.116:0/1322571205 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4030d50 con 0x7f0eec0d01e0 2023-11-28T17:09:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.529+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1322571205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eec0d01e0 msgr2=0x7f0eec160270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.529+0000 7f0ef43a5640 1 --2- 172.21.15.116:0/1322571205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eec0d01e0 0x7f0eec160270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0b5d30 tx=0x7f0ee402b9a0 comp rx=0 tx=0).stop 2023-11-28T17:09:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.529+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1322571205 shutdown_connections 2023-11-28T17:09:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.529+0000 7f0ef43a5640 1 --2- 172.21.15.116:0/1322571205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eec0d01e0 0x7f0eec160270 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.529+0000 7f0ef43a5640 1 --2- 172.21.15.116:0/1322571205 >> v2:172.21.15.156:3300/0 conn(0x7f0eec15d480 0x7f0eec0d0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.529+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1322571205 >> 172.21.15.116:0/1322571205 conn(0x7f0eec09c540 msgr2=0x7f0eec0d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.529+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1322571205 shutdown_connections 2023-11-28T17:09:03.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.529+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1322571205 wait complete. 2023-11-28T17:09:03.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.532+0000 7f0ef43a5640 1 Processor -- start 2023-11-28T17:09:03.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.532+0000 7f0ef43a5640 1 -- start start 2023-11-28T17:09:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.533+0000 7f0ef43a5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0eec0d01e0 0x7f0eec1f24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.533+0000 7f0ef43a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eec0d1310 0x7f0eec160520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.533+0000 7f0ef43a5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0eec15d480 0x7f0eec160a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.533+0000 7f0ef43a5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec151ab0 con 0x7f0eec0d1310 2023-11-28T17:09:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.533+0000 7f0ef43a5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec1f6930 con 0x7f0eec0d01e0 2023-11-28T17:09:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.533+0000 7f0ef43a5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec0c9400 con 0x7f0eec15d480 2023-11-28T17:09:03.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.534+0000 7f0ef1940640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eec0d1310 0x7f0eec160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.534+0000 7f0ef1940640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eec0d1310 0x7f0eec160520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43630/0 (socket says 172.21.15.116:43630) 2023-11-28T17:09:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.534+0000 7f0ef1940640 1 -- 172.21.15.116:0/1507258070 learned_addr learned my addr 172.21.15.116:0/1507258070 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.534+0000 7f0ef2141640 1 --2- 172.21.15.116:0/1507258070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0eec0d01e0 0x7f0eec1f24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.534+0000 7f0ef2942640 1 --2- 172.21.15.116:0/1507258070 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0eec15d480 0x7f0eec160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:03.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.534+0000 7f0ef1940640 1 -- 172.21.15.116:0/1507258070 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0eec15d480 msgr2=0x7f0eec160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.534+0000 7f0ef1940640 1 --2- 172.21.15.116:0/1507258070 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0eec15d480 0x7f0eec160a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.534+0000 7f0ef1940640 1 -- 172.21.15.116:0/1507258070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0eec0d01e0 msgr2=0x7f0eec1f24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.534+0000 7f0ef1940640 1 --2- 172.21.15.116:0/1507258070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0eec0d01e0 0x7f0eec1f24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.534+0000 7f0ef1940640 1 -- 172.21.15.116:0/1507258070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee40094d0 con 0x7f0eec0d1310 2023-11-28T17:09:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.534+0000 7f0ef1940640 1 --2- 172.21.15.116:0/1507258070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eec0d1310 0x7f0eec160520 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0ed800c440 tx=0x7f0ed800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:03.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.534+0000 7f0eeaffd640 1 -- 172.21.15.116:0/1507258070 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8005480 con 0x7f0eec0d1310 2023-11-28T17:09:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.535+0000 7f0eeaffd640 1 -- 172.21.15.116:0/1507258070 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed80035f0 con 0x7f0eec0d1310 2023-11-28T17:09:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.535+0000 7f0eeaffd640 1 -- 172.21.15.116:0/1507258070 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80039a0 con 0x7f0eec0d1310 2023-11-28T17:09:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.535+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1507258070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eec1f29e0 con 0x7f0eec0d1310 2023-11-28T17:09:03.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.535+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1507258070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eec1f62e0 con 0x7f0eec0d1310 2023-11-28T17:09:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.536+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1507258070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eec067980 con 0x7f0eec0d1310 2023-11-28T17:09:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.537+0000 7f0eeaffd640 1 -- 172.21.15.116:0/1507258070 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0ed8003b00 con 0x7f0eec0d1310 2023-11-28T17:09:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.538+0000 7f0eeaffd640 1 --2- 172.21.15.116:0/1507258070 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ed006b2a0 0x7f0ed006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.538+0000 7f0eeaffd640 1 -- 172.21.15.116:0/1507258070 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ed8089120 con 0x7f0eec0d1310 2023-11-28T17:09:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.538+0000 7f0ef2141640 1 --2- 172.21.15.116:0/1507258070 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ed006b2a0 0x7f0ed006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.539+0000 7f0ef2141640 1 --2- 172.21.15.116:0/1507258070 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ed006b2a0 0x7f0ed006d760 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0ee00075b0 tx=0x7f0ee0007540 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:03.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.542+0000 7f0eeaffd640 1 -- 172.21.15.116:0/1507258070 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ed8053f80 con 0x7f0eec0d1310 2023-11-28T17:09:03.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:03.770+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1507258070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f0eec1f6140 con 0x7f0eec0d1310 2023-11-28T17:09:04.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.295+0000 7f0eeaffd640 1 -- 172.21.15.116:0/1507258070 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0eec1f6140 con 0x7f0eec0d1310 2023-11-28T17:09:04.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.299+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1507258070 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ed006b2a0 msgr2=0x7f0ed006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:04.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.299+0000 7f0ef43a5640 1 --2- 172.21.15.116:0/1507258070 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ed006b2a0 0x7f0ed006d760 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0ee00075b0 tx=0x7f0ee0007540 comp rx=0 tx=0).stop 2023-11-28T17:09:04.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.299+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1507258070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eec0d1310 msgr2=0x7f0eec160520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:04.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.299+0000 7f0ef43a5640 1 --2- 172.21.15.116:0/1507258070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eec0d1310 0x7f0eec160520 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0ed800c440 tx=0x7f0ed800c900 comp rx=0 tx=0).stop 2023-11-28T17:09:04.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.299+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1507258070 shutdown_connections 2023-11-28T17:09:04.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.299+0000 7f0ef43a5640 1 --2- 172.21.15.116:0/1507258070 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ed006b2a0 0x7f0ed006d760 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:04.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.299+0000 7f0ef43a5640 1 --2- 172.21.15.116:0/1507258070 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0eec15d480 0x7f0eec160a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:04.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.299+0000 7f0ef43a5640 1 --2- 172.21.15.116:0/1507258070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0eec0d1310 0x7f0eec160520 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:04.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.299+0000 7f0ef43a5640 1 --2- 172.21.15.116:0/1507258070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0eec0d01e0 0x7f0eec1f24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:04.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.299+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1507258070 >> 172.21.15.116:0/1507258070 conn(0x7f0eec09c540 msgr2=0x7f0eec0cdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:04.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.299+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1507258070 shutdown_connections 2023-11-28T17:09:04.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.300+0000 7f0ef43a5640 1 -- 172.21.15.116:0/1507258070 wait complete. 2023-11-28T17:09:04.321 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T17:09:04.322 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:09:04.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.536+0000 7fb0be7b3640 1 Processor -- start 2023-11-28T17:09:04.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.536+0000 7fb0be7b3640 1 -- start start 2023-11-28T17:09:04.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.538+0000 7fb0be7b3640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0b815d3b0 0x7fb0b80abd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:04.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.538+0000 7fb0be7b3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0b80ce470 0x7fb0b8071230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:04.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.538+0000 7fb0be7b3640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b80cc7d0 con 0x7fb0b80ceb70 2023-11-28T17:09:04.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.538+0000 7fb0be7b3640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b80ce920 con 0x7fb0b80ce470 2023-11-28T17:09:04.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.538+0000 7fb0be7b3640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b8071770 con 0x7fb0b815d3b0 2023-11-28T17:09:04.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.538+0000 7fb0bdfb2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0b80ce470 0x7fb0b8071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:04.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.538+0000 7fb0bdfb2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0b80ce470 0x7fb0b8071230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43636/0 (socket says 172.21.15.116:43636) 2023-11-28T17:09:04.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.538+0000 7fb0bdfb2640 1 -- 172.21.15.116:0/1078993998 learned_addr learned my addr 172.21.15.116:0/1078993998 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:04.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.538+0000 7fb0bcfb0640 1 --2- 172.21.15.116:0/1078993998 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0b815d3b0 0x7fb0b80abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:04.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.539+0000 7fb0bdfb2640 1 -- 172.21.15.116:0/1078993998 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0b815d3b0 msgr2=0x7fb0b80abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:04.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.539+0000 7fb0bdfb2640 1 --2- 172.21.15.116:0/1078993998 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0b815d3b0 0x7fb0b80abd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:04.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.539+0000 7fb0bdfb2640 1 -- 172.21.15.116:0/1078993998 >> v1:172.21.15.156:6789/0 conn(0x7fb0b80ceb70 legacy=0x7fb0b80cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:04.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.539+0000 7fb0bdfb2640 1 -- 172.21.15.116:0/1078993998 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b8071a30 con 0x7fb0b80ce470 2023-11-28T17:09:04.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.540+0000 7fb0bdfb2640 1 --2- 172.21.15.116:0/1078993998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b80ce470 0x7fb0b8071230 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009e50 tx=0x7fb0b002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfb7e843c70765a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:04.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.540+0000 7fb0affff640 1 -- 172.21.15.116:0/1078993998 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b002c9e0 con 0x7fb0b80ce470 2023-11-28T17:09:04.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.540+0000 7fb0affff640 1 -- 172.21.15.116:0/1078993998 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b002cb40 con 0x7fb0b80ce470 2023-11-28T17:09:04.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.540+0000 7fb0affff640 1 -- 172.21.15.116:0/1078993998 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b00343f0 con 0x7fb0b80ce470 2023-11-28T17:09:04.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.541+0000 7fb0be7b3640 1 -- 172.21.15.116:0/1078993998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b80ce470 msgr2=0x7fb0b8071230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:04.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.541+0000 7fb0be7b3640 1 --2- 172.21.15.116:0/1078993998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b80ce470 0x7fb0b8071230 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009e50 tx=0x7fb0b002be40 comp rx=0 tx=0).stop 2023-11-28T17:09:04.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.541+0000 7fb0be7b3640 1 -- 172.21.15.116:0/1078993998 shutdown_connections 2023-11-28T17:09:04.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.541+0000 7fb0be7b3640 1 --2- 172.21.15.116:0/1078993998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b80ce470 0x7fb0b8071230 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:04.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.541+0000 7fb0be7b3640 1 --2- 172.21.15.116:0/1078993998 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0b815d3b0 0x7fb0b80abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:04.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.541+0000 7fb0be7b3640 1 -- 172.21.15.116:0/1078993998 >> 172.21.15.116:0/1078993998 conn(0x7fb0b80b0950 msgr2=0x7fb0b80b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:04.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.541+0000 7fb0be7b3640 1 -- 172.21.15.116:0/1078993998 shutdown_connections 2023-11-28T17:09:04.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.541+0000 7fb0be7b3640 1 -- 172.21.15.116:0/1078993998 wait complete. 2023-11-28T17:09:04.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.542+0000 7fb0be7b3640 1 Processor -- start 2023-11-28T17:09:04.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.543+0000 7fb0be7b3640 1 -- start start 2023-11-28T17:09:04.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.543+0000 7fb0be7b3640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0b80ce470 0x7fb0b81e74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:04.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.543+0000 7fb0be7b3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0b80ceb70 0x7fb0b815b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:04.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.543+0000 7fb0be7b3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b815d3b0 0x7fb0b815a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.543+0000 7fb0be7b3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b81e7a20 con 0x7fb0b815d3b0 2023-11-28T17:09:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.543+0000 7fb0be7b3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b81e6740 con 0x7fb0b80ceb70 2023-11-28T17:09:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.543+0000 7fb0be7b3640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b81e6f50 con 0x7fb0b80ce470 2023-11-28T17:09:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.544+0000 7fb0bdfb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b815d3b0 0x7fb0b815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.544+0000 7fb0bdfb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b815d3b0 0x7fb0b815a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43650/0 (socket says 172.21.15.116:43650) 2023-11-28T17:09:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.544+0000 7fb0bdfb2640 1 -- 172.21.15.116:0/2611468498 learned_addr learned my addr 172.21.15.116:0/2611468498 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.544+0000 7fb0bcfb0640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0b80ceb70 0x7fb0b815b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.544+0000 7fb0bd7b1640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0b80ce470 0x7fb0b81e74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.544+0000 7fb0bdfb2640 1 -- 172.21.15.116:0/2611468498 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0b80ce470 msgr2=0x7fb0b81e74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.544+0000 7fb0bdfb2640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0b80ce470 0x7fb0b81e74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.544+0000 7fb0bdfb2640 1 -- 172.21.15.116:0/2611468498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0b80ceb70 msgr2=0x7fb0b815b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.544+0000 7fb0bdfb2640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0b80ceb70 0x7fb0b815b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.544+0000 7fb0bdfb2640 1 -- 172.21.15.116:0/2611468498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b0009b30 con 0x7fb0b815d3b0 2023-11-28T17:09:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.544+0000 7fb0bd7b1640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0b80ce470 0x7fb0b81e74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.544+0000 7fb0bcfb0640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0b80ceb70 0x7fb0b815b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.544+0000 7fb0bdfb2640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b815d3b0 0x7fb0b815a410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0036040 tx=0x7fb0b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.545+0000 7fb0adffb640 1 -- 172.21.15.116:0/2611468498 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0009510 con 0x7fb0b815d3b0 2023-11-28T17:09:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.545+0000 7fb0adffb640 1 -- 172.21.15.116:0/2611468498 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b003e450 con 0x7fb0b815d3b0 2023-11-28T17:09:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.545+0000 7fb0be7b3640 1 -- 172.21.15.116:0/2611468498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b815fb00 con 0x7fb0b815d3b0 2023-11-28T17:09:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.545+0000 7fb0adffb640 1 -- 172.21.15.116:0/2611468498 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b003e7e0 con 0x7fb0b815d3b0 2023-11-28T17:09:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.545+0000 7fb0be7b3640 1 -- 172.21.15.116:0/2611468498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b81e6c70 con 0x7fb0b815d3b0 2023-11-28T17:09:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.546+0000 7fb0be7b3640 1 -- 172.21.15.116:0/2611468498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b809d0b0 con 0x7fb0b815d3b0 2023-11-28T17:09:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.547+0000 7fb0adffb640 1 -- 172.21.15.116:0/2611468498 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb0b003e940 con 0x7fb0b815d3b0 2023-11-28T17:09:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.548+0000 7fb0adffb640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb09c06b2f0 0x7fb09c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.548+0000 7fb0adffb640 1 -- 172.21.15.116:0/2611468498 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0b00ac890 con 0x7fb0b815d3b0 2023-11-28T17:09:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.548+0000 7fb0bd7b1640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb09c06b2f0 0x7fb09c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:04.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.549+0000 7fb0bd7b1640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb09c06b2f0 0x7fb09c06d7b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4003eb0 tx=0x7fb0b4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:04.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.552+0000 7fb0adffb640 1 -- 172.21.15.116:0/2611468498 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0b0077770 con 0x7fb0b815d3b0 2023-11-28T17:09:04.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.777+0000 7fb0be7b3640 1 -- 172.21.15.116:0/2611468498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0b80b0370 con 0x7fb0b815d3b0 2023-11-28T17:09:04.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.778+0000 7fb0adffb640 1 -- 172.21.15.116:0/2611468498 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7fb0b007ad90 con 0x7fb0b815d3b0 2023-11-28T17:09:04.829 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:04.829 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3138968863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":3138968863},{"type":"v1","addr":"172.21.15.156:6835","nonce":3138968863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/733981412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":733981412},{"type":"v1","addr":"172.21.15.116:6836","nonce":733981412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2751219705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2751219705},{"type":"v1","addr":"172.21.15.116:6837","nonce":2751219705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:09:00.673159+0000","modified":"2023-11-28T17:09:03.771842+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6834/3268354971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3268354971},{"type":"v1","addr":"172.21.15.156:6834","nonce":3268354971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T17:09:04.831 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2023-11-28T17:09:04.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.781+0000 7fb0be7b3640 1 -- 172.21.15.116:0/2611468498 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb09c06b2f0 msgr2=0x7fb09c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:04.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.781+0000 7fb0be7b3640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb09c06b2f0 0x7fb09c06d7b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4003eb0 tx=0x7fb0b4009040 comp rx=0 tx=0).stop 2023-11-28T17:09:04.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.781+0000 7fb0be7b3640 1 -- 172.21.15.116:0/2611468498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b815d3b0 msgr2=0x7fb0b815a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:04.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.781+0000 7fb0be7b3640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b815d3b0 0x7fb0b815a410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0036040 tx=0x7fb0b0004a40 comp rx=0 tx=0).stop 2023-11-28T17:09:04.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.781+0000 7fb0be7b3640 1 -- 172.21.15.116:0/2611468498 shutdown_connections 2023-11-28T17:09:04.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.781+0000 7fb0be7b3640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb09c06b2f0 0x7fb09c06d7b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:04.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.781+0000 7fb0be7b3640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0b815d3b0 0x7fb0b815a410 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:04.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.781+0000 7fb0be7b3640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0b80ceb70 0x7fb0b815b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:04.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.781+0000 7fb0be7b3640 1 --2- 172.21.15.116:0/2611468498 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0b80ce470 0x7fb0b81e74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:04.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.781+0000 7fb0be7b3640 1 -- 172.21.15.116:0/2611468498 >> 172.21.15.116:0/2611468498 conn(0x7fb0b80b0950 msgr2=0x7fb0b80ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:04.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.781+0000 7fb0be7b3640 1 -- 172.21.15.116:0/2611468498 shutdown_connections 2023-11-28T17:09:04.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:04.782+0000 7fb0be7b3640 1 -- 172.21.15.116:0/2611468498 wait complete. 2023-11-28T17:09:04.833 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.050+0000 7f1c048ca640 1 Processor -- start 2023-11-28T17:09:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.050+0000 7f1c048ca640 1 -- start start 2023-11-28T17:09:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.052+0000 7f1c048ca640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1c0015ae20 0x7f1c0006ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.052+0000 7f1c048ca640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c000cf2a0 0x7f1c000ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.052+0000 7f1c048ca640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c00070560 con 0x7f1c000cdf90 2023-11-28T17:09:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.052+0000 7f1c048ca640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c00070840 con 0x7f1c0015ae20 2023-11-28T17:09:05.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.052+0000 7f1c048ca640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c00070b20 con 0x7f1c000cf2a0 2023-11-28T17:09:05.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.052+0000 7f1bff7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1c0015ae20 0x7f1c0006ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:05.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.052+0000 7f1bff7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1c0015ae20 0x7f1c0006ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43652/0 (socket says 172.21.15.116:43652) 2023-11-28T17:09:05.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.052+0000 7f1bff7fe640 1 -- 172.21.15.116:0/1485149974 learned_addr learned my addr 172.21.15.116:0/1485149974 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:05.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.052+0000 7f1bff7fe640 1 -- 172.21.15.116:0/1485149974 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c000cf2a0 msgr2=0x7f1c000ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:05.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.052+0000 7f1bff7fe640 1 --2- 172.21.15.116:0/1485149974 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c000cf2a0 0x7f1c000ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.052+0000 7f1bff7fe640 1 -- 172.21.15.116:0/1485149974 >> v1:172.21.15.156:6789/0 conn(0x7f1c000cdf90 legacy=0x7f1c000ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.052+0000 7f1bff7fe640 1 -- 172.21.15.116:0/1485149974 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c00070e00 con 0x7f1c0015ae20 2023-11-28T17:09:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.053+0000 7f1bff7fe640 1 --2- 172.21.15.116:0/1485149974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0015ae20 0x7f1c0006ccd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0009e80 tx=0x7f1bf002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5746aa6de3a4edeb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.054+0000 7f1bfe7fc640 1 -- 172.21.15.116:0/1485149974 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf002ca70 con 0x7f1c0015ae20 2023-11-28T17:09:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.054+0000 7f1bfe7fc640 1 -- 172.21.15.116:0/1485149974 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf002cbd0 con 0x7f1c0015ae20 2023-11-28T17:09:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.054+0000 7f1bfe7fc640 1 -- 172.21.15.116:0/1485149974 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf00343f0 con 0x7f1c0015ae20 2023-11-28T17:09:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.054+0000 7f1c048ca640 1 -- 172.21.15.116:0/1485149974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0015ae20 msgr2=0x7f1c0006ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.054+0000 7f1c048ca640 1 --2- 172.21.15.116:0/1485149974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0015ae20 0x7f1c0006ccd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0009e80 tx=0x7f1bf002bed0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.055+0000 7f1c048ca640 1 -- 172.21.15.116:0/1485149974 shutdown_connections 2023-11-28T17:09:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.055+0000 7f1c048ca640 1 --2- 172.21.15.116:0/1485149974 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c000cf2a0 0x7f1c000ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.055+0000 7f1c048ca640 1 --2- 172.21.15.116:0/1485149974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0015ae20 0x7f1c0006ccd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.055+0000 7f1c048ca640 1 -- 172.21.15.116:0/1485149974 >> 172.21.15.116:0/1485149974 conn(0x7f1c000b0950 msgr2=0x7f1c000b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.055+0000 7f1c048ca640 1 -- 172.21.15.116:0/1485149974 shutdown_connections 2023-11-28T17:09:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.055+0000 7f1c048ca640 1 -- 172.21.15.116:0/1485149974 wait complete. 2023-11-28T17:09:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.056+0000 7f1c048ca640 1 Processor -- start 2023-11-28T17:09:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.057+0000 7f1c048ca640 1 -- start start 2023-11-28T17:09:05.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.057+0000 7f1c048ca640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c000cdf90 0x7f1c00157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:05.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.057+0000 7f1c048ca640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c000cf2a0 0x7f1c00145350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:05.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.057+0000 7f1c048ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0015ae20 0x7f1c00159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:05.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.057+0000 7f1c048ca640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c00145890 con 0x7f1c0015ae20 2023-11-28T17:09:05.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.058+0000 7f1c048ca640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c00145cb0 con 0x7f1c000cdf90 2023-11-28T17:09:05.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.058+0000 7f1c048ca640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c00145f90 con 0x7f1c000cf2a0 2023-11-28T17:09:05.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.058+0000 7f1bfffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0015ae20 0x7f1c00159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:05.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.058+0000 7f1bfffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0015ae20 0x7f1c00159fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43654/0 (socket says 172.21.15.116:43654) 2023-11-28T17:09:05.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.058+0000 7f1bfffff640 1 -- 172.21.15.116:0/2744774846 learned_addr learned my addr 172.21.15.116:0/2744774846 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:05.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.058+0000 7f1bff7fe640 1 --2- 172.21.15.116:0/2744774846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c000cdf90 0x7f1c00157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:05.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.058+0000 7f1bfeffd640 1 --2- 172.21.15.116:0/2744774846 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c000cf2a0 0x7f1c00145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:05.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.058+0000 7f1bfffff640 1 -- 172.21.15.116:0/2744774846 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c000cf2a0 msgr2=0x7f1c00145350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:05.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.058+0000 7f1bfffff640 1 --2- 172.21.15.116:0/2744774846 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c000cf2a0 0x7f1c00145350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.058+0000 7f1bfffff640 1 -- 172.21.15.116:0/2744774846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c000cdf90 msgr2=0x7f1c00157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:05.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.058+0000 7f1bfffff640 1 --2- 172.21.15.116:0/2744774846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c000cdf90 0x7f1c00157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.058+0000 7f1bfffff640 1 -- 172.21.15.116:0/2744774846 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf0009b30 con 0x7f1c0015ae20 2023-11-28T17:09:05.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.059+0000 7f1bfffff640 1 --2- 172.21.15.116:0/2744774846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0015ae20 0x7f1c00159fd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800b3c0 tx=0x7f1bf800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:05.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.059+0000 7f1bf7fff640 1 -- 172.21.15.116:0/2744774846 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8011830 con 0x7f1c0015ae20 2023-11-28T17:09:05.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.059+0000 7f1bf7fff640 1 -- 172.21.15.116:0/2744774846 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf80119b0 con 0x7f1c0015ae20 2023-11-28T17:09:05.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.059+0000 7f1bf7fff640 1 -- 172.21.15.116:0/2744774846 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8011d60 con 0x7f1c0015ae20 2023-11-28T17:09:05.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.059+0000 7f1c048ca640 1 -- 172.21.15.116:0/2744774846 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0015b380 con 0x7f1c0015ae20 2023-11-28T17:09:05.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.059+0000 7f1c048ca640 1 -- 172.21.15.116:0/2744774846 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c00146550 con 0x7f1c0015ae20 2023-11-28T17:09:05.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.060+0000 7f1c048ca640 1 -- 172.21.15.116:0/2744774846 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0009d0b0 con 0x7f1c0015ae20 2023-11-28T17:09:05.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.061+0000 7f1bf7fff640 1 -- 172.21.15.116:0/2744774846 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f1bf801b460 con 0x7f1c0015ae20 2023-11-28T17:09:05.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.062+0000 7f1bf7fff640 1 --2- 172.21.15.116:0/2744774846 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1be006b2a0 0x7f1be006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:05.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.062+0000 7f1bf7fff640 1 -- 172.21.15.116:0/2744774846 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1bf8089950 con 0x7f1c0015ae20 2023-11-28T17:09:05.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.062+0000 7f1bff7fe640 1 --2- 172.21.15.116:0/2744774846 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1be006b2a0 0x7f1be006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:05.064 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.063+0000 7f1bff7fe640 1 --2- 172.21.15.116:0/2744774846 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1be006b2a0 0x7f1be006d760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0035000 tx=0x7f1bf0030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:05.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.066+0000 7f1bf7fff640 1 -- 172.21.15.116:0/2744774846 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1bf8058240 con 0x7f1c0015ae20 2023-11-28T17:09:05.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.256+0000 7f1c048ca640 1 -- 172.21.15.116:0/2744774846 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c0004e960 con 0x7f1c0015ae20 2023-11-28T17:09:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.258+0000 7f1bf7fff640 1 -- 172.21.15.116:0/2744774846 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16521 (secure 0 0 0) 0x7f1bf8057dd0 con 0x7f1c0015ae20 2023-11-28T17:09:05.260 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:05.260 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:01.970957+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.261+0000 7f1c048ca640 1 -- 172.21.15.116:0/2744774846 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1be006b2a0 msgr2=0x7f1be006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.261+0000 7f1c048ca640 1 --2- 172.21.15.116:0/2744774846 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1be006b2a0 0x7f1be006d760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0035000 tx=0x7f1bf0030000 comp rx=0 tx=0).stop 2023-11-28T17:09:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.261+0000 7f1c048ca640 1 -- 172.21.15.116:0/2744774846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0015ae20 msgr2=0x7f1c00159fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.261+0000 7f1c048ca640 1 --2- 172.21.15.116:0/2744774846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0015ae20 0x7f1c00159fd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800b3c0 tx=0x7f1bf800b880 comp rx=0 tx=0).stop 2023-11-28T17:09:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.261+0000 7f1c048ca640 1 -- 172.21.15.116:0/2744774846 shutdown_connections 2023-11-28T17:09:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.261+0000 7f1c048ca640 1 --2- 172.21.15.116:0/2744774846 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1be006b2a0 0x7f1be006d760 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.261+0000 7f1c048ca640 1 --2- 172.21.15.116:0/2744774846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c0015ae20 0x7f1c00159fd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.261+0000 7f1c048ca640 1 --2- 172.21.15.116:0/2744774846 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c000cf2a0 0x7f1c00145350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.261+0000 7f1c048ca640 1 --2- 172.21.15.116:0/2744774846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c000cdf90 0x7f1c00157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.261+0000 7f1c048ca640 1 -- 172.21.15.116:0/2744774846 >> 172.21.15.116:0/2744774846 conn(0x7f1c000b0950 msgr2=0x7f1c000b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.261+0000 7f1c048ca640 1 -- 172.21.15.116:0/2744774846 shutdown_connections 2023-11-28T17:09:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.262+0000 7f1c048ca640 1 -- 172.21.15.116:0/2744774846 wait complete. 2023-11-28T17:09:05.282 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T17:09:05.282 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:09:05.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.494+0000 7ff096e60640 1 Processor -- start 2023-11-28T17:09:05.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.495+0000 7ff096e60640 1 -- start start 2023-11-28T17:09:05.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.496+0000 7ff096e60640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0900cf2a0 0x7ff0900701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:05.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.496+0000 7ff096e60640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff09015ae90 0x7ff0900ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:05.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.496+0000 7ff096e60640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0900cd8e0 con 0x7ff090159f30 2023-11-28T17:09:05.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.496+0000 7ff096e60640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff090070730 con 0x7ff09015ae90 2023-11-28T17:09:05.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.496+0000 7ff096e60640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff090070ee0 con 0x7ff0900cf2a0 2023-11-28T17:09:05.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.497+0000 7ff09665f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff09015ae90 0x7ff0900ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:05.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.497+0000 7ff09665f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff09015ae90 0x7ff0900ce330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43656/0 (socket says 172.21.15.116:43656) 2023-11-28T17:09:05.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.497+0000 7ff09665f640 1 -- 172.21.15.116:0/3399939172 learned_addr learned my addr 172.21.15.116:0/3399939172 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:05.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.497+0000 7ff09665f640 1 -- 172.21.15.116:0/3399939172 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0900cf2a0 msgr2=0x7ff0900701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:05.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.497+0000 7ff09665f640 1 --2- 172.21.15.116:0/3399939172 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0900cf2a0 0x7ff0900701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.497+0000 7ff09665f640 1 -- 172.21.15.116:0/3399939172 >> v1:172.21.15.156:6789/0 conn(0x7ff090159f30 legacy=0x7ff0900ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:05.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.497+0000 7ff09665f640 1 -- 172.21.15.116:0/3399939172 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09015b310 con 0x7ff09015ae90 2023-11-28T17:09:05.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.498+0000 7ff09665f640 1 --2- 172.21.15.116:0/3399939172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09015ae90 0x7ff0900ce330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff088009e50 tx=0x7ff08802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84910ebc0c7ac9de server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:05.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.498+0000 7ff094e5c640 1 -- 172.21.15.116:0/3399939172 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08802cb20 con 0x7ff09015ae90 2023-11-28T17:09:05.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.498+0000 7ff094e5c640 1 -- 172.21.15.116:0/3399939172 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08802cc80 con 0x7ff09015ae90 2023-11-28T17:09:05.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.498+0000 7ff094e5c640 1 -- 172.21.15.116:0/3399939172 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0880343f0 con 0x7ff09015ae90 2023-11-28T17:09:05.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.499+0000 7ff096e60640 1 -- 172.21.15.116:0/3399939172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09015ae90 msgr2=0x7ff0900ce330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:05.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.499+0000 7ff096e60640 1 --2- 172.21.15.116:0/3399939172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09015ae90 0x7ff0900ce330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff088009e50 tx=0x7ff08802bf80 comp rx=0 tx=0).stop 2023-11-28T17:09:05.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.499+0000 7ff096e60640 1 -- 172.21.15.116:0/3399939172 shutdown_connections 2023-11-28T17:09:05.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.499+0000 7ff096e60640 1 --2- 172.21.15.116:0/3399939172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff09015ae90 0x7ff0900ce330 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.499+0000 7ff096e60640 1 --2- 172.21.15.116:0/3399939172 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0900cf2a0 0x7ff0900701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.499+0000 7ff096e60640 1 -- 172.21.15.116:0/3399939172 >> 172.21.15.116:0/3399939172 conn(0x7ff0900b0950 msgr2=0x7ff0900b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:05.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.499+0000 7ff096e60640 1 -- 172.21.15.116:0/3399939172 shutdown_connections 2023-11-28T17:09:05.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.499+0000 7ff096e60640 1 -- 172.21.15.116:0/3399939172 wait complete. 2023-11-28T17:09:05.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.501+0000 7ff096e60640 1 Processor -- start 2023-11-28T17:09:05.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.501+0000 7ff096e60640 1 -- start start 2023-11-28T17:09:05.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.502+0000 7ff096e60640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0900cf2a0 0x7ff0900ad390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:05.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.502+0000 7ff096e60640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff090159f30 0x7ff09006ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:05.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.502+0000 7ff096e60640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff09015ae90 0x7ff09014e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:05.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.502+0000 7ff096e60640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09009a960 con 0x7ff090159f30 2023-11-28T17:09:05.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.502+0000 7ff096e60640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09006de20 con 0x7ff0900cf2a0 2023-11-28T17:09:05.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.502+0000 7ff096e60640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09009ad80 con 0x7ff09015ae90 2023-11-28T17:09:05.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.502+0000 7ff09565d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff090159f30 0x7ff09006ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:05.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.502+0000 7ff09565d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff090159f30 0x7ff09006ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43664/0 (socket says 172.21.15.116:43664) 2023-11-28T17:09:05.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.502+0000 7ff09565d640 1 -- 172.21.15.116:0/2032638961 learned_addr learned my addr 172.21.15.116:0/2032638961 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:05.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.502+0000 7ff09565d640 1 -- 172.21.15.116:0/2032638961 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff09015ae90 msgr2=0x7ff09014e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:05.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.502+0000 7ff09665f640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff09015ae90 0x7ff09014e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:05.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.503+0000 7ff095e5e640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0900cf2a0 0x7ff0900ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:05.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.503+0000 7ff09565d640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff09015ae90 0x7ff09014e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.503+0000 7ff09565d640 1 -- 172.21.15.116:0/2032638961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0900cf2a0 msgr2=0x7ff0900ad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:05.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.503+0000 7ff09565d640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0900cf2a0 0x7ff0900ad390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.503+0000 7ff09565d640 1 -- 172.21.15.116:0/2032638961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff088009b30 con 0x7ff090159f30 2023-11-28T17:09:05.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.503+0000 7ff095e5e640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0900cf2a0 0x7ff0900ad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:05.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.503+0000 7ff09565d640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff090159f30 0x7ff09006ce90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff08400b390 tx=0x7ff08400b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:05.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.503+0000 7ff0827fc640 1 -- 172.21.15.116:0/2032638961 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084011830 con 0x7ff090159f30 2023-11-28T17:09:05.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.503+0000 7ff0827fc640 1 -- 172.21.15.116:0/2032638961 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff084011990 con 0x7ff090159f30 2023-11-28T17:09:05.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.503+0000 7ff0827fc640 1 -- 172.21.15.116:0/2032638961 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084011d40 con 0x7ff090159f30 2023-11-28T17:09:05.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.503+0000 7ff096e60640 1 -- 172.21.15.116:0/2032638961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff090099a00 con 0x7ff090159f30 2023-11-28T17:09:05.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.503+0000 7ff096e60640 1 -- 172.21.15.116:0/2032638961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff090099fc0 con 0x7ff090159f30 2023-11-28T17:09:05.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.505+0000 7ff096e60640 1 -- 172.21.15.116:0/2032638961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff09009d0b0 con 0x7ff090159f30 2023-11-28T17:09:05.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.505+0000 7ff0827fc640 1 -- 172.21.15.116:0/2032638961 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff08401a400 con 0x7ff090159f30 2023-11-28T17:09:05.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.506+0000 7ff0827fc640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff07406b2a0 0x7ff07406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:05.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.506+0000 7ff0827fc640 1 -- 172.21.15.116:0/2032638961 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff084089bb0 con 0x7ff090159f30 2023-11-28T17:09:05.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.506+0000 7ff095e5e640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff07406b2a0 0x7ff07406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:05.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.507+0000 7ff095e5e640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff07406b2a0 0x7ff07406d760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff08c000f60 tx=0x7ff08c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:05.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.510+0000 7ff0827fc640 1 -- 172.21.15.116:0/2032638961 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff084054ab0 con 0x7ff090159f30 2023-11-28T17:09:05.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.695+0000 7ff096e60640 1 -- 172.21.15.116:0/2032638961 --> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff090070c00 con 0x7ff07406b2a0 2023-11-28T17:09:05.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.706+0000 7ff0827fc640 1 -- 172.21.15.116:0/2032638961 <== mgr.4098 v2:172.21.15.116:6832/1077400178 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264338 (secure 0 0 0) 0x7ff08c0013e0 con 0x7ff07406b2a0 2023-11-28T17:09:05.711 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:05.713 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-28T17:09:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.712+0000 7ff096e60640 1 -- 172.21.15.116:0/2032638961 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff07406b2a0 msgr2=0x7ff07406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.712+0000 7ff096e60640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff07406b2a0 0x7ff07406d760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff08c000f60 tx=0x7ff08c009040 comp rx=0 tx=0).stop 2023-11-28T17:09:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.712+0000 7ff096e60640 1 -- 172.21.15.116:0/2032638961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff090159f30 msgr2=0x7ff09006ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.712+0000 7ff096e60640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff090159f30 0x7ff09006ce90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff08400b390 tx=0x7ff08400b850 comp rx=0 tx=0).stop 2023-11-28T17:09:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.712+0000 7ff096e60640 1 -- 172.21.15.116:0/2032638961 shutdown_connections 2023-11-28T17:09:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.712+0000 7ff096e60640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff07406b2a0 0x7ff07406d760 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.712+0000 7ff096e60640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff09015ae90 0x7ff09014e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.712+0000 7ff096e60640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff090159f30 0x7ff09006ce90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.712+0000 7ff096e60640 1 --2- 172.21.15.116:0/2032638961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0900cf2a0 0x7ff0900ad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.712+0000 7ff096e60640 1 -- 172.21.15.116:0/2032638961 >> 172.21.15.116:0/2032638961 conn(0x7ff0900b0950 msgr2=0x7ff0900733f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.712+0000 7ff096e60640 1 -- 172.21.15.116:0/2032638961 shutdown_connections 2023-11-28T17:09:05.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.712+0000 7ff096e60640 1 -- 172.21.15.116:0/2032638961 wait complete. 2023-11-28T17:09:05.731 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2023-11-28T17:09:05.040444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":162,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235648,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739072,"statfs":{"total":773094113280,"available":772852809728,"internally_reserved":0,"allocated":2891776,"data_stored":1752836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.819595"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513632+0000","last_change":"2023-11-28T17:09:01.014584+0000","last_active":"2023-11-28T17:09:02.513632+0000","last_peered":"2023-11-28T17:09:02.513632+0000","last_clean":"2023-11-28T17:09:02.513632+0000","last_became_active":"2023-11-28T17:09:01.014291+0000","last_became_peered":"2023-11-28T17:09:01.014291+0000","last_unstale":"2023-11-28T17:09:02.513632+0000","last_undegraded":"2023-11-28T17:09:02.513632+0000","last_fullsized":"2023-11-28T17:09:02.513632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:22.787341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223151+0000","last_change":"2023-11-28T17:09:01.009980+0000","last_active":"2023-11-28T17:09:02.223151+0000","last_peered":"2023-11-28T17:09:02.223151+0000","last_clean":"2023-11-28T17:09:02.223151+0000","last_became_active":"2023-11-28T17:09:01.009702+0000","last_became_peered":"2023-11-28T17:09:01.009702+0000","last_unstale":"2023-11-28T17:09:02.223151+0000","last_undegraded":"2023-11-28T17:09:02.223151+0000","last_fullsized":"2023-11-28T17:09:02.223151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:06:59.199376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222772+0000","last_change":"2023-11-28T17:09:01.010483+0000","last_active":"2023-11-28T17:09:02.222772+0000","last_peered":"2023-11-28T17:09:02.222772+0000","last_clean":"2023-11-28T17:09:02.222772+0000","last_became_active":"2023-11-28T17:09:01.010032+0000","last_became_peered":"2023-11-28T17:09:01.010032+0000","last_unstale":"2023-11-28T17:09:02.222772+0000","last_undegraded":"2023-11-28T17:09:02.222772+0000","last_fullsized":"2023-11-28T17:09:02.222772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:46:24.200692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513159+0000","last_change":"2023-11-28T17:09:00.998473+0000","last_active":"2023-11-28T17:09:02.513159+0000","last_peered":"2023-11-28T17:09:02.513159+0000","last_clean":"2023-11-28T17:09:02.513159+0000","last_became_active":"2023-11-28T17:09:00.998206+0000","last_became_peered":"2023-11-28T17:09:00.998206+0000","last_unstale":"2023-11-28T17:09:02.513159+0000","last_undegraded":"2023-11-28T17:09:02.513159+0000","last_fullsized":"2023-11-28T17:09:02.513159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:31.569896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512712+0000","last_change":"2023-11-28T17:09:01.019053+0000","last_active":"2023-11-28T17:09:02.512712+0000","last_peered":"2023-11-28T17:09:02.512712+0000","last_clean":"2023-11-28T17:09:02.512712+0000","last_became_active":"2023-11-28T17:09:01.018689+0000","last_became_peered":"2023-11-28T17:09:01.018689+0000","last_unstale":"2023-11-28T17:09:02.512712+0000","last_undegraded":"2023-11-28T17:09:02.512712+0000","last_fullsized":"2023-11-28T17:09:02.512712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:06:59.199376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513060+0000","last_change":"2023-11-28T17:09:01.000056+0000","last_active":"2023-11-28T17:09:02.513060+0000","last_peered":"2023-11-28T17:09:02.513060+0000","last_clean":"2023-11-28T17:09:02.513060+0000","last_became_active":"2023-11-28T17:09:00.999547+0000","last_became_peered":"2023-11-28T17:09:00.999547+0000","last_unstale":"2023-11-28T17:09:02.513060+0000","last_undegraded":"2023-11-28T17:09:02.513060+0000","last_fullsized":"2023-11-28T17:09:02.513060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:12.314413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.511764+0000","last_change":"2023-11-28T17:09:01.012699+0000","last_active":"2023-11-28T17:09:02.511764+0000","last_peered":"2023-11-28T17:09:02.511764+0000","last_clean":"2023-11-28T17:09:02.511764+0000","last_became_active":"2023-11-28T17:09:01.012442+0000","last_became_peered":"2023-11-28T17:09:01.012442+0000","last_unstale":"2023-11-28T17:09:02.511764+0000","last_undegraded":"2023-11-28T17:09:02.511764+0000","last_fullsized":"2023-11-28T17:09:02.511764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:46:24.200692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223215+0000","last_change":"2023-11-28T17:09:01.016792+0000","last_active":"2023-11-28T17:09:02.223215+0000","last_peered":"2023-11-28T17:09:02.223215+0000","last_clean":"2023-11-28T17:09:02.223215+0000","last_became_active":"2023-11-28T17:09:01.016034+0000","last_became_peered":"2023-11-28T17:09:01.016034+0000","last_unstale":"2023-11-28T17:09:02.223215+0000","last_undegraded":"2023-11-28T17:09:02.223215+0000","last_fullsized":"2023-11-28T17:09:02.223215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:52.224230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515192+0000","last_change":"2023-11-28T17:09:01.007271+0000","last_active":"2023-11-28T17:09:02.515192+0000","last_peered":"2023-11-28T17:09:02.515192+0000","last_clean":"2023-11-28T17:09:02.515192+0000","last_became_active":"2023-11-28T17:09:01.006929+0000","last_became_peered":"2023-11-28T17:09:01.006929+0000","last_unstale":"2023-11-28T17:09:02.515192+0000","last_undegraded":"2023-11-28T17:09:02.515192+0000","last_fullsized":"2023-11-28T17:09:02.515192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:20.109182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515425+0000","last_change":"2023-11-28T17:09:01.012176+0000","last_active":"2023-11-28T17:09:02.515425+0000","last_peered":"2023-11-28T17:09:02.515425+0000","last_clean":"2023-11-28T17:09:02.515425+0000","last_became_active":"2023-11-28T17:09:01.011821+0000","last_became_peered":"2023-11-28T17:09:01.011821+0000","last_unstale":"2023-11-28T17:09:02.515425+0000","last_undegraded":"2023-11-28T17:09:02.515425+0000","last_fullsized":"2023-11-28T17:09:02.515425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:55.626548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513365+0000","last_change":"2023-11-28T17:09:01.008348+0000","last_active":"2023-11-28T17:09:02.513365+0000","last_peered":"2023-11-28T17:09:02.513365+0000","last_clean":"2023-11-28T17:09:02.513365+0000","last_became_active":"2023-11-28T17:09:01.008029+0000","last_became_peered":"2023-11-28T17:09:01.008029+0000","last_unstale":"2023-11-28T17:09:02.513365+0000","last_undegraded":"2023-11-28T17:09:02.513365+0000","last_fullsized":"2023-11-28T17:09:02.513365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:52.224230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512353+0000","last_change":"2023-11-28T17:09:01.013836+0000","last_active":"2023-11-28T17:09:02.512353+0000","last_peered":"2023-11-28T17:09:02.512353+0000","last_clean":"2023-11-28T17:09:02.512353+0000","last_became_active":"2023-11-28T17:09:01.011870+0000","last_became_peered":"2023-11-28T17:09:01.011870+0000","last_unstale":"2023-11-28T17:09:02.512353+0000","last_undegraded":"2023-11-28T17:09:02.512353+0000","last_fullsized":"2023-11-28T17:09:02.512353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:23:27.238395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988425+0000","last_change":"2023-11-28T17:09:01.007854+0000","last_active":"2023-11-28T17:09:01.988425+0000","last_peered":"2023-11-28T17:09:01.988425+0000","last_clean":"2023-11-28T17:09:01.988425+0000","last_became_active":"2023-11-28T17:09:01.007561+0000","last_became_peered":"2023-11-28T17:09:01.007561+0000","last_unstale":"2023-11-28T17:09:01.988425+0000","last_undegraded":"2023-11-28T17:09:01.988425+0000","last_fullsized":"2023-11-28T17:09:01.988425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:53.375742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986992+0000","last_change":"2023-11-28T17:09:01.006142+0000","last_active":"2023-11-28T17:09:01.986992+0000","last_peered":"2023-11-28T17:09:01.986992+0000","last_clean":"2023-11-28T17:09:01.986992+0000","last_became_active":"2023-11-28T17:09:01.005823+0000","last_became_peered":"2023-11-28T17:09:01.005823+0000","last_unstale":"2023-11-28T17:09:01.986992+0000","last_undegraded":"2023-11-28T17:09:01.986992+0000","last_fullsized":"2023-11-28T17:09:01.986992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:32.388198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222442+0000","last_change":"2023-11-28T17:09:01.016320+0000","last_active":"2023-11-28T17:09:02.222442+0000","last_peered":"2023-11-28T17:09:02.222442+0000","last_clean":"2023-11-28T17:09:02.222442+0000","last_became_active":"2023-11-28T17:09:01.016012+0000","last_became_peered":"2023-11-28T17:09:01.016012+0000","last_unstale":"2023-11-28T17:09:02.222442+0000","last_undegraded":"2023-11-28T17:09:02.222442+0000","last_fullsized":"2023-11-28T17:09:02.222442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:44.651188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986526+0000","last_change":"2023-11-28T17:09:00.996901+0000","last_active":"2023-11-28T17:09:01.986526+0000","last_peered":"2023-11-28T17:09:01.986526+0000","last_clean":"2023-11-28T17:09:01.986526+0000","last_became_active":"2023-11-28T17:09:00.996627+0000","last_became_peered":"2023-11-28T17:09:00.996627+0000","last_unstale":"2023-11-28T17:09:01.986526+0000","last_undegraded":"2023-11-28T17:09:01.986526+0000","last_fullsized":"2023-11-28T17:09:01.986526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:45.964344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988654+0000","last_change":"2023-11-28T17:09:01.015527+0000","last_active":"2023-11-28T17:09:01.988654+0000","last_peered":"2023-11-28T17:09:01.988654+0000","last_clean":"2023-11-28T17:09:01.988654+0000","last_became_active":"2023-11-28T17:09:01.015238+0000","last_became_peered":"2023-11-28T17:09:01.015238+0000","last_unstale":"2023-11-28T17:09:01.988654+0000","last_undegraded":"2023-11-28T17:09:01.988654+0000","last_fullsized":"2023-11-28T17:09:01.988654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:44:13.494591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512305+0000","last_change":"2023-11-28T17:09:01.015000+0000","last_active":"2023-11-28T17:09:02.512305+0000","last_peered":"2023-11-28T17:09:02.512305+0000","last_clean":"2023-11-28T17:09:02.512305+0000","last_became_active":"2023-11-28T17:09:01.014649+0000","last_became_peered":"2023-11-28T17:09:01.014649+0000","last_unstale":"2023-11-28T17:09:02.512305+0000","last_undegraded":"2023-11-28T17:09:02.512305+0000","last_fullsized":"2023-11-28T17:09:02.512305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:31.569896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988495+0000","last_change":"2023-11-28T17:09:00.997869+0000","last_active":"2023-11-28T17:09:01.988495+0000","last_peered":"2023-11-28T17:09:01.988495+0000","last_clean":"2023-11-28T17:09:01.988495+0000","last_became_active":"2023-11-28T17:09:00.997614+0000","last_became_peered":"2023-11-28T17:09:00.997614+0000","last_unstale":"2023-11-28T17:09:01.988495+0000","last_undegraded":"2023-11-28T17:09:01.988495+0000","last_fullsized":"2023-11-28T17:09:01.988495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:25:26.955817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513938+0000","last_change":"2023-11-28T17:09:01.011584+0000","last_active":"2023-11-28T17:09:02.513938+0000","last_peered":"2023-11-28T17:09:02.513938+0000","last_clean":"2023-11-28T17:09:02.513938+0000","last_became_active":"2023-11-28T17:09:01.010359+0000","last_became_peered":"2023-11-28T17:09:01.010359+0000","last_unstale":"2023-11-28T17:09:02.513938+0000","last_undegraded":"2023-11-28T17:09:02.513938+0000","last_fullsized":"2023-11-28T17:09:02.513938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:45.964344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.989818+0000","last_change":"2023-11-28T17:09:00.999976+0000","last_active":"2023-11-28T17:09:01.989818+0000","last_peered":"2023-11-28T17:09:01.989818+0000","last_clean":"2023-11-28T17:09:01.989818+0000","last_became_active":"2023-11-28T17:09:00.999647+0000","last_became_peered":"2023-11-28T17:09:00.999647+0000","last_unstale":"2023-11-28T17:09:01.989818+0000","last_undegraded":"2023-11-28T17:09:01.989818+0000","last_fullsized":"2023-11-28T17:09:01.989818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:44.651188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.514610+0000","last_change":"2023-11-28T17:09:01.009510+0000","last_active":"2023-11-28T17:09:02.514610+0000","last_peered":"2023-11-28T17:09:02.514610+0000","last_clean":"2023-11-28T17:09:02.514610+0000","last_became_active":"2023-11-28T17:09:01.009184+0000","last_became_peered":"2023-11-28T17:09:01.009184+0000","last_unstale":"2023-11-28T17:09:02.514610+0000","last_undegraded":"2023-11-28T17:09:02.514610+0000","last_fullsized":"2023-11-28T17:09:02.514610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:12.314413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222577+0000","last_change":"2023-11-28T17:09:01.015602+0000","last_active":"2023-11-28T17:09:02.222577+0000","last_peered":"2023-11-28T17:09:02.222577+0000","last_clean":"2023-11-28T17:09:02.222577+0000","last_became_active":"2023-11-28T17:09:01.012170+0000","last_became_peered":"2023-11-28T17:09:01.012170+0000","last_unstale":"2023-11-28T17:09:02.222577+0000","last_undegraded":"2023-11-28T17:09:02.222577+0000","last_fullsized":"2023-11-28T17:09:02.222577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:07.593383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986512+0000","last_change":"2023-11-28T17:09:01.016062+0000","last_active":"2023-11-28T17:09:01.986512+0000","last_peered":"2023-11-28T17:09:01.986512+0000","last_clean":"2023-11-28T17:09:01.986512+0000","last_became_active":"2023-11-28T17:09:01.015802+0000","last_became_peered":"2023-11-28T17:09:01.015802+0000","last_unstale":"2023-11-28T17:09:01.986512+0000","last_undegraded":"2023-11-28T17:09:01.986512+0000","last_fullsized":"2023-11-28T17:09:01.986512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:34.895213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223456+0000","last_change":"2023-11-28T17:09:01.016389+0000","last_active":"2023-11-28T17:09:02.223456+0000","last_peered":"2023-11-28T17:09:02.223456+0000","last_clean":"2023-11-28T17:09:02.223456+0000","last_became_active":"2023-11-28T17:09:01.011680+0000","last_became_peered":"2023-11-28T17:09:01.011680+0000","last_unstale":"2023-11-28T17:09:02.223456+0000","last_undegraded":"2023-11-28T17:09:02.223456+0000","last_fullsized":"2023-11-28T17:09:02.223456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:22.787341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515917+0000","last_change":"2023-11-28T17:09:01.006281+0000","last_active":"2023-11-28T17:09:02.515917+0000","last_peered":"2023-11-28T17:09:02.515917+0000","last_clean":"2023-11-28T17:09:02.515917+0000","last_became_active":"2023-11-28T17:09:01.000480+0000","last_became_peered":"2023-11-28T17:09:01.000480+0000","last_unstale":"2023-11-28T17:09:02.515917+0000","last_undegraded":"2023-11-28T17:09:02.515917+0000","last_fullsized":"2023-11-28T17:09:02.515917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:25.324961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.989120+0000","last_change":"2023-11-28T17:09:01.002090+0000","last_active":"2023-11-28T17:09:01.989120+0000","last_peered":"2023-11-28T17:09:01.989120+0000","last_clean":"2023-11-28T17:09:01.989120+0000","last_became_active":"2023-11-28T17:09:01.001728+0000","last_became_peered":"2023-11-28T17:09:01.001728+0000","last_unstale":"2023-11-28T17:09:01.989120+0000","last_undegraded":"2023-11-28T17:09:01.989120+0000","last_fullsized":"2023-11-28T17:09:01.989120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:11.188332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988810+0000","last_change":"2023-11-28T17:09:01.013978+0000","last_active":"2023-11-28T17:09:01.988810+0000","last_peered":"2023-11-28T17:09:01.988810+0000","last_clean":"2023-11-28T17:09:01.988810+0000","last_became_active":"2023-11-28T17:09:01.013646+0000","last_became_peered":"2023-11-28T17:09:01.013646+0000","last_unstale":"2023-11-28T17:09:01.988810+0000","last_undegraded":"2023-11-28T17:09:01.988810+0000","last_fullsized":"2023-11-28T17:09:01.988810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:00.035340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222920+0000","last_change":"2023-11-28T17:09:01.013635+0000","last_active":"2023-11-28T17:09:02.222920+0000","last_peered":"2023-11-28T17:09:02.222920+0000","last_clean":"2023-11-28T17:09:02.222920+0000","last_became_active":"2023-11-28T17:09:01.013310+0000","last_became_peered":"2023-11-28T17:09:01.013310+0000","last_unstale":"2023-11-28T17:09:02.222920+0000","last_undegraded":"2023-11-28T17:09:02.222920+0000","last_fullsized":"2023-11-28T17:09:02.222920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:23:27.238395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987149+0000","last_change":"2023-11-28T17:09:01.016976+0000","last_active":"2023-11-28T17:09:01.987149+0000","last_peered":"2023-11-28T17:09:01.987149+0000","last_clean":"2023-11-28T17:09:01.987149+0000","last_became_active":"2023-11-28T17:09:01.016690+0000","last_became_peered":"2023-11-28T17:09:01.016690+0000","last_unstale":"2023-11-28T17:09:01.987149+0000","last_undegraded":"2023-11-28T17:09:01.987149+0000","last_fullsized":"2023-11-28T17:09:01.987149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:55.626548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.511885+0000","last_change":"2023-11-28T17:09:01.012474+0000","last_active":"2023-11-28T17:09:02.511885+0000","last_peered":"2023-11-28T17:09:02.511885+0000","last_clean":"2023-11-28T17:09:02.511885+0000","last_became_active":"2023-11-28T17:09:01.012226+0000","last_became_peered":"2023-11-28T17:09:01.012226+0000","last_unstale":"2023-11-28T17:09:02.511885+0000","last_undegraded":"2023-11-28T17:09:02.511885+0000","last_fullsized":"2023-11-28T17:09:02.511885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:36.438412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512524+0000","last_change":"2023-11-28T17:09:00.994555+0000","last_active":"2023-11-28T17:09:02.512524+0000","last_peered":"2023-11-28T17:09:02.512524+0000","last_clean":"2023-11-28T17:09:02.512524+0000","last_became_active":"2023-11-28T17:09:00.993870+0000","last_became_peered":"2023-11-28T17:09:00.993870+0000","last_unstale":"2023-11-28T17:09:02.512524+0000","last_undegraded":"2023-11-28T17:09:02.512524+0000","last_fullsized":"2023-11-28T17:09:02.512524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:36.438412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.514893+0000","last_change":"2023-11-28T17:09:01.018035+0000","last_active":"2023-11-28T17:09:02.514893+0000","last_peered":"2023-11-28T17:09:02.514893+0000","last_clean":"2023-11-28T17:09:02.514893+0000","last_became_active":"2023-11-28T17:09:01.017735+0000","last_became_peered":"2023-11-28T17:09:01.017735+0000","last_unstale":"2023-11-28T17:09:02.514893+0000","last_undegraded":"2023-11-28T17:09:02.514893+0000","last_fullsized":"2023-11-28T17:09:02.514893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:32.388198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.271905+0000","last_change":"2023-11-28T17:09:00.266501+0000","last_active":"2023-11-28T17:09:01.271905+0000","last_peered":"2023-11-28T17:09:01.271905+0000","last_clean":"2023-11-28T17:09:01.271905+0000","last_became_active":"2023-11-28T17:09:00.266177+0000","last_became_peered":"2023-11-28T17:09:00.266177+0000","last_unstale":"2023-11-28T17:09:01.271905+0000","last_undegraded":"2023-11-28T17:09:01.271905+0000","last_fullsized":"2023-11-28T17:09:01.271905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:08.376086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.990190+0000","last_change":"2023-11-28T17:09:01.008791+0000","last_active":"2023-11-28T17:09:01.990190+0000","last_peered":"2023-11-28T17:09:01.990190+0000","last_clean":"2023-11-28T17:09:01.990190+0000","last_became_active":"2023-11-28T17:09:01.008266+0000","last_became_peered":"2023-11-28T17:09:01.008266+0000","last_unstale":"2023-11-28T17:09:01.990190+0000","last_undegraded":"2023-11-28T17:09:01.990190+0000","last_fullsized":"2023-11-28T17:09:01.990190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:50.317730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.275936+0000","last_change":"2023-11-28T17:09:00.269760+0000","last_active":"2023-11-28T17:09:01.275936+0000","last_peered":"2023-11-28T17:09:01.275936+0000","last_clean":"2023-11-28T17:09:01.275936+0000","last_became_active":"2023-11-28T17:09:00.269082+0000","last_became_peered":"2023-11-28T17:09:00.269082+0000","last_unstale":"2023-11-28T17:09:01.275936+0000","last_undegraded":"2023-11-28T17:09:01.275936+0000","last_fullsized":"2023-11-28T17:09:01.275936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:51.213109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.986115+0000","last_change":"2023-11-28T17:09:00.262912+0000","last_active":"2023-11-28T17:09:00.986115+0000","last_peered":"2023-11-28T17:09:00.986115+0000","last_clean":"2023-11-28T17:09:00.986115+0000","last_became_active":"2023-11-28T17:09:00.260568+0000","last_became_peered":"2023-11-28T17:09:00.260568+0000","last_unstale":"2023-11-28T17:09:00.986115+0000","last_undegraded":"2023-11-28T17:09:00.986115+0000","last_fullsized":"2023-11-28T17:09:00.986115+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:30.558775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223014+0000","last_change":"2023-11-28T17:09:01.012823+0000","last_active":"2023-11-28T17:09:02.223014+0000","last_peered":"2023-11-28T17:09:02.223014+0000","last_clean":"2023-11-28T17:09:02.223014+0000","last_became_active":"2023-11-28T17:09:01.012514+0000","last_became_peered":"2023-11-28T17:09:01.012514+0000","last_unstale":"2023-11-28T17:09:02.223014+0000","last_undegraded":"2023-11-28T17:09:02.223014+0000","last_fullsized":"2023-11-28T17:09:02.223014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:24.354633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.985898+0000","last_change":"2023-11-28T17:09:00.262108+0000","last_active":"2023-11-28T17:09:00.985898+0000","last_peered":"2023-11-28T17:09:00.985898+0000","last_clean":"2023-11-28T17:09:00.985898+0000","last_became_active":"2023-11-28T17:09:00.261621+0000","last_became_peered":"2023-11-28T17:09:00.261621+0000","last_unstale":"2023-11-28T17:09:00.985898+0000","last_undegraded":"2023-11-28T17:09:00.985898+0000","last_fullsized":"2023-11-28T17:09:00.985898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:14:52.843064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988589+0000","last_change":"2023-11-28T17:09:01.011726+0000","last_active":"2023-11-28T17:09:01.988589+0000","last_peered":"2023-11-28T17:09:01.988589+0000","last_clean":"2023-11-28T17:09:01.988589+0000","last_became_active":"2023-11-28T17:09:01.010904+0000","last_became_peered":"2023-11-28T17:09:01.010904+0000","last_unstale":"2023-11-28T17:09:01.988589+0000","last_undegraded":"2023-11-28T17:09:01.988589+0000","last_fullsized":"2023-11-28T17:09:01.988589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:14:53.854185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.990133+0000","last_change":"2023-11-28T17:09:00.262360+0000","last_active":"2023-11-28T17:09:00.990133+0000","last_peered":"2023-11-28T17:09:00.990133+0000","last_clean":"2023-11-28T17:09:00.990133+0000","last_became_active":"2023-11-28T17:09:00.261943+0000","last_became_peered":"2023-11-28T17:09:00.261943+0000","last_unstale":"2023-11-28T17:09:00.990133+0000","last_undegraded":"2023-11-28T17:09:00.990133+0000","last_fullsized":"2023-11-28T17:09:00.990133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:52.998404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515909+0000","last_change":"2023-11-28T17:09:01.007941+0000","last_active":"2023-11-28T17:09:02.515909+0000","last_peered":"2023-11-28T17:09:02.515909+0000","last_clean":"2023-11-28T17:09:02.515909+0000","last_became_active":"2023-11-28T17:09:01.002743+0000","last_became_peered":"2023-11-28T17:09:01.002743+0000","last_unstale":"2023-11-28T17:09:02.515909+0000","last_undegraded":"2023-11-28T17:09:02.515909+0000","last_fullsized":"2023-11-28T17:09:02.515909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:20.109182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.997425+0000","last_change":"2023-11-28T17:09:00.259873+0000","last_active":"2023-11-28T17:09:00.997425+0000","last_peered":"2023-11-28T17:09:00.997425+0000","last_clean":"2023-11-28T17:09:00.997425+0000","last_became_active":"2023-11-28T17:09:00.259506+0000","last_became_peered":"2023-11-28T17:09:00.259506+0000","last_unstale":"2023-11-28T17:09:00.997425+0000","last_undegraded":"2023-11-28T17:09:00.997425+0000","last_fullsized":"2023-11-28T17:09:00.997425+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:19.098061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223644+0000","last_change":"2023-11-28T17:09:01.017611+0000","last_active":"2023-11-28T17:09:02.223644+0000","last_peered":"2023-11-28T17:09:02.223644+0000","last_clean":"2023-11-28T17:09:02.223644+0000","last_became_active":"2023-11-28T17:09:01.017269+0000","last_became_peered":"2023-11-28T17:09:01.017269+0000","last_unstale":"2023-11-28T17:09:02.223644+0000","last_undegraded":"2023-11-28T17:09:02.223644+0000","last_fullsized":"2023-11-28T17:09:02.223644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:45.964344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.272853+0000","last_change":"2023-11-28T17:09:00.264904+0000","last_active":"2023-11-28T17:09:01.272853+0000","last_peered":"2023-11-28T17:09:01.272853+0000","last_clean":"2023-11-28T17:09:01.272853+0000","last_became_active":"2023-11-28T17:09:00.264645+0000","last_became_peered":"2023-11-28T17:09:00.264645+0000","last_unstale":"2023-11-28T17:09:01.272853+0000","last_undegraded":"2023-11-28T17:09:01.272853+0000","last_fullsized":"2023-11-28T17:09:01.272853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:25:59.069890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987426+0000","last_change":"2023-11-28T17:09:01.005502+0000","last_active":"2023-11-28T17:09:01.987426+0000","last_peered":"2023-11-28T17:09:01.987426+0000","last_clean":"2023-11-28T17:09:01.987426+0000","last_became_active":"2023-11-28T17:09:01.004783+0000","last_became_peered":"2023-11-28T17:09:01.004783+0000","last_unstale":"2023-11-28T17:09:01.987426+0000","last_undegraded":"2023-11-28T17:09:01.987426+0000","last_fullsized":"2023-11-28T17:09:01.987426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:10:50.736315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276609+0000","last_change":"2023-11-28T17:09:00.263414+0000","last_active":"2023-11-28T17:09:01.276609+0000","last_peered":"2023-11-28T17:09:01.276609+0000","last_clean":"2023-11-28T17:09:01.276609+0000","last_became_active":"2023-11-28T17:09:00.260801+0000","last_became_peered":"2023-11-28T17:09:00.260801+0000","last_unstale":"2023-11-28T17:09:01.276609+0000","last_undegraded":"2023-11-28T17:09:01.276609+0000","last_fullsized":"2023-11-28T17:09:01.276609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:31:18.731288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513670+0000","last_change":"2023-11-28T17:09:01.019149+0000","last_active":"2023-11-28T17:09:02.513670+0000","last_peered":"2023-11-28T17:09:02.513670+0000","last_clean":"2023-11-28T17:09:02.513670+0000","last_became_active":"2023-11-28T17:09:01.018834+0000","last_became_peered":"2023-11-28T17:09:01.018834+0000","last_unstale":"2023-11-28T17:09:02.513670+0000","last_undegraded":"2023-11-28T17:09:02.513670+0000","last_fullsized":"2023-11-28T17:09:02.513670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:34.895213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.275750+0000","last_change":"2023-11-28T17:09:00.271771+0000","last_active":"2023-11-28T17:09:01.275750+0000","last_peered":"2023-11-28T17:09:01.275750+0000","last_clean":"2023-11-28T17:09:01.275750+0000","last_became_active":"2023-11-28T17:09:00.269930+0000","last_became_peered":"2023-11-28T17:09:00.269930+0000","last_unstale":"2023-11-28T17:09:01.275750+0000","last_undegraded":"2023-11-28T17:09:01.275750+0000","last_fullsized":"2023-11-28T17:09:01.275750+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:33.502573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987179+0000","last_change":"2023-11-28T17:09:00.999606+0000","last_active":"2023-11-28T17:09:01.987179+0000","last_peered":"2023-11-28T17:09:01.987179+0000","last_clean":"2023-11-28T17:09:01.987179+0000","last_became_active":"2023-11-28T17:09:00.999346+0000","last_became_peered":"2023-11-28T17:09:00.999346+0000","last_unstale":"2023-11-28T17:09:01.987179+0000","last_undegraded":"2023-11-28T17:09:01.987179+0000","last_fullsized":"2023-11-28T17:09:01.987179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:25.324961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.991340+0000","last_change":"2023-11-28T17:09:00.269952+0000","last_active":"2023-11-28T17:09:00.991340+0000","last_peered":"2023-11-28T17:09:00.991340+0000","last_clean":"2023-11-28T17:09:00.991340+0000","last_became_active":"2023-11-28T17:09:00.269539+0000","last_became_peered":"2023-11-28T17:09:00.269539+0000","last_unstale":"2023-11-28T17:09:00.991340+0000","last_undegraded":"2023-11-28T17:09:00.991340+0000","last_fullsized":"2023-11-28T17:09:00.991340+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:58.974841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987385+0000","last_change":"2023-11-28T17:09:01.009043+0000","last_active":"2023-11-28T17:09:01.987385+0000","last_peered":"2023-11-28T17:09:01.987385+0000","last_clean":"2023-11-28T17:09:01.987385+0000","last_became_active":"2023-11-28T17:09:01.008739+0000","last_became_peered":"2023-11-28T17:09:01.008739+0000","last_unstale":"2023-11-28T17:09:01.987385+0000","last_undegraded":"2023-11-28T17:09:01.987385+0000","last_fullsized":"2023-11-28T17:09:01.987385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:49.417143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.999207+0000","last_change":"2023-11-28T17:09:00.270474+0000","last_active":"2023-11-28T17:09:00.999207+0000","last_peered":"2023-11-28T17:09:00.999207+0000","last_clean":"2023-11-28T17:09:00.999207+0000","last_became_active":"2023-11-28T17:09:00.270277+0000","last_became_peered":"2023-11-28T17:09:00.270277+0000","last_unstale":"2023-11-28T17:09:00.999207+0000","last_undegraded":"2023-11-28T17:09:00.999207+0000","last_fullsized":"2023-11-28T17:09:00.999207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:52.364620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.511767+0000","last_change":"2023-11-28T17:09:01.016931+0000","last_active":"2023-11-28T17:09:02.511767+0000","last_peered":"2023-11-28T17:09:02.511767+0000","last_clean":"2023-11-28T17:09:02.511767+0000","last_became_active":"2023-11-28T17:09:01.016673+0000","last_became_peered":"2023-11-28T17:09:01.016673+0000","last_unstale":"2023-11-28T17:09:02.511767+0000","last_undegraded":"2023-11-28T17:09:02.511767+0000","last_fullsized":"2023-11-28T17:09:02.511767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:36.438412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.274020+0000","last_change":"2023-11-28T17:09:00.262584+0000","last_active":"2023-11-28T17:09:01.274020+0000","last_peered":"2023-11-28T17:09:01.274020+0000","last_clean":"2023-11-28T17:09:01.274020+0000","last_became_active":"2023-11-28T17:09:00.262259+0000","last_became_peered":"2023-11-28T17:09:00.262259+0000","last_unstale":"2023-11-28T17:09:01.274020+0000","last_undegraded":"2023-11-28T17:09:01.274020+0000","last_fullsized":"2023-11-28T17:09:01.274020+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:31.377077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986947+0000","last_change":"2023-11-28T17:09:01.003188+0000","last_active":"2023-11-28T17:09:01.986947+0000","last_peered":"2023-11-28T17:09:01.986947+0000","last_clean":"2023-11-28T17:09:01.986947+0000","last_became_active":"2023-11-28T17:09:01.000643+0000","last_became_peered":"2023-11-28T17:09:01.000643+0000","last_unstale":"2023-11-28T17:09:01.986947+0000","last_undegraded":"2023-11-28T17:09:01.986947+0000","last_fullsized":"2023-11-28T17:09:01.986947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:49:27.242284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273771+0000","last_change":"2023-11-28T17:09:00.272398+0000","last_active":"2023-11-28T17:09:01.273771+0000","last_peered":"2023-11-28T17:09:01.273771+0000","last_clean":"2023-11-28T17:09:01.273771+0000","last_became_active":"2023-11-28T17:09:00.271617+0000","last_became_peered":"2023-11-28T17:09:00.271617+0000","last_unstale":"2023-11-28T17:09:01.273771+0000","last_undegraded":"2023-11-28T17:09:01.273771+0000","last_fullsized":"2023-11-28T17:09:01.273771+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:10.177210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.516054+0000","last_change":"2023-11-28T17:09:01.012920+0000","last_active":"2023-11-28T17:09:02.516054+0000","last_peered":"2023-11-28T17:09:02.516054+0000","last_clean":"2023-11-28T17:09:02.516054+0000","last_became_active":"2023-11-28T17:09:01.012651+0000","last_became_peered":"2023-11-28T17:09:01.012651+0000","last_unstale":"2023-11-28T17:09:02.516054+0000","last_undegraded":"2023-11-28T17:09:02.516054+0000","last_fullsized":"2023-11-28T17:09:02.516054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:53.375742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273900+0000","last_change":"2023-11-28T17:09:00.271952+0000","last_active":"2023-11-28T17:09:01.273900+0000","last_peered":"2023-11-28T17:09:01.273900+0000","last_clean":"2023-11-28T17:09:01.273900+0000","last_became_active":"2023-11-28T17:09:00.271641+0000","last_became_peered":"2023-11-28T17:09:00.271641+0000","last_unstale":"2023-11-28T17:09:01.273900+0000","last_undegraded":"2023-11-28T17:09:01.273900+0000","last_fullsized":"2023-11-28T17:09:01.273900+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:52.364620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513991+0000","last_change":"2023-11-28T17:09:01.014412+0000","last_active":"2023-11-28T17:09:02.513991+0000","last_peered":"2023-11-28T17:09:02.513991+0000","last_clean":"2023-11-28T17:09:02.513991+0000","last_became_active":"2023-11-28T17:09:01.014145+0000","last_became_peered":"2023-11-28T17:09:01.014145+0000","last_unstale":"2023-11-28T17:09:02.513991+0000","last_undegraded":"2023-11-28T17:09:02.513991+0000","last_fullsized":"2023-11-28T17:09:02.513991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:49:27.242284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273346+0000","last_change":"2023-11-28T17:09:00.264424+0000","last_active":"2023-11-28T17:09:01.273346+0000","last_peered":"2023-11-28T17:09:01.273346+0000","last_clean":"2023-11-28T17:09:01.273346+0000","last_became_active":"2023-11-28T17:09:00.263195+0000","last_became_peered":"2023-11-28T17:09:00.263195+0000","last_unstale":"2023-11-28T17:09:01.273346+0000","last_undegraded":"2023-11-28T17:09:01.273346+0000","last_fullsized":"2023-11-28T17:09:01.273346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:35.427291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513525+0000","last_change":"2023-11-28T17:09:01.019072+0000","last_active":"2023-11-28T17:09:02.513525+0000","last_peered":"2023-11-28T17:09:02.513525+0000","last_clean":"2023-11-28T17:09:02.513525+0000","last_became_active":"2023-11-28T17:09:01.018785+0000","last_became_peered":"2023-11-28T17:09:01.018785+0000","last_unstale":"2023-11-28T17:09:02.513525+0000","last_undegraded":"2023-11-28T17:09:02.513525+0000","last_fullsized":"2023-11-28T17:09:02.513525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:59.985963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.984805+0000","last_change":"2023-11-28T17:09:00.268057+0000","last_active":"2023-11-28T17:09:00.984805+0000","last_peered":"2023-11-28T17:09:00.984805+0000","last_clean":"2023-11-28T17:09:00.984805+0000","last_became_active":"2023-11-28T17:09:00.264298+0000","last_became_peered":"2023-11-28T17:09:00.264298+0000","last_unstale":"2023-11-28T17:09:00.984805+0000","last_undegraded":"2023-11-28T17:09:00.984805+0000","last_fullsized":"2023-11-28T17:09:00.984805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:35.427291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988219+0000","last_change":"2023-11-28T17:09:01.009062+0000","last_active":"2023-11-28T17:09:01.988219+0000","last_peered":"2023-11-28T17:09:01.988219+0000","last_clean":"2023-11-28T17:09:01.988219+0000","last_became_active":"2023-11-28T17:09:01.008759+0000","last_became_peered":"2023-11-28T17:09:01.008759+0000","last_unstale":"2023-11-28T17:09:01.988219+0000","last_undegraded":"2023-11-28T17:09:01.988219+0000","last_fullsized":"2023-11-28T17:09:01.988219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:09:50.534329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.986684+0000","last_change":"2023-11-28T17:08:51.392782+0000","last_active":"2023-11-28T17:09:00.986684+0000","last_peered":"2023-11-28T17:09:00.986684+0000","last_clean":"2023-11-28T17:09:00.986684+0000","last_became_active":"2023-11-28T17:08:51.390415+0000","last_became_peered":"2023-11-28T17:08:51.390415+0000","last_unstale":"2023-11-28T17:09:00.986684+0000","last_undegraded":"2023-11-28T17:09:00.986684+0000","last_fullsized":"2023-11-28T17:09:00.986684+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_clean_scrub_stamp":"2023-11-28T17:08:49.967682+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:03:32.638685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.985042+0000","last_change":"2023-11-28T17:09:00.267795+0000","last_active":"2023-11-28T17:09:00.985042+0000","last_peered":"2023-11-28T17:09:00.985042+0000","last_clean":"2023-11-28T17:09:00.985042+0000","last_became_active":"2023-11-28T17:09:00.264764+0000","last_became_peered":"2023-11-28T17:09:00.264764+0000","last_unstale":"2023-11-28T17:09:00.985042+0000","last_undegraded":"2023-11-28T17:09:00.985042+0000","last_fullsized":"2023-11-28T17:09:00.985042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:59.024219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988457+0000","last_change":"2023-11-28T17:09:01.004564+0000","last_active":"2023-11-28T17:09:01.988457+0000","last_peered":"2023-11-28T17:09:01.988457+0000","last_clean":"2023-11-28T17:09:01.988457+0000","last_became_active":"2023-11-28T17:09:01.004189+0000","last_became_peered":"2023-11-28T17:09:01.004189+0000","last_unstale":"2023-11-28T17:09:01.988457+0000","last_undegraded":"2023-11-28T17:09:01.988457+0000","last_fullsized":"2023-11-28T17:09:01.988457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:09.387207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.986785+0000","last_change":"2023-11-28T17:09:00.269480+0000","last_active":"2023-11-28T17:09:00.986785+0000","last_peered":"2023-11-28T17:09:00.986785+0000","last_clean":"2023-11-28T17:09:00.986785+0000","last_became_active":"2023-11-28T17:09:00.269032+0000","last_became_peered":"2023-11-28T17:09:00.269032+0000","last_unstale":"2023-11-28T17:09:00.986785+0000","last_undegraded":"2023-11-28T17:09:00.986785+0000","last_fullsized":"2023-11-28T17:09:00.986785+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:49.306608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987055+0000","last_change":"2023-11-28T17:09:01.007809+0000","last_active":"2023-11-28T17:09:01.987055+0000","last_peered":"2023-11-28T17:09:01.987055+0000","last_clean":"2023-11-28T17:09:01.987055+0000","last_became_active":"2023-11-28T17:09:01.007508+0000","last_became_peered":"2023-11-28T17:09:01.007508+0000","last_unstale":"2023-11-28T17:09:01.987055+0000","last_undegraded":"2023-11-28T17:09:01.987055+0000","last_fullsized":"2023-11-28T17:09:01.987055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:20.109182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.001078+0000","last_change":"2023-11-28T17:09:00.271320+0000","last_active":"2023-11-28T17:09:01.001078+0000","last_peered":"2023-11-28T17:09:01.001078+0000","last_clean":"2023-11-28T17:09:01.001078+0000","last_became_active":"2023-11-28T17:09:00.271019+0000","last_became_peered":"2023-11-28T17:09:00.271019+0000","last_unstale":"2023-11-28T17:09:01.001078+0000","last_undegraded":"2023-11-28T17:09:01.001078+0000","last_fullsized":"2023-11-28T17:09:01.001078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:41:19.939145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222456+0000","last_change":"2023-11-28T17:09:01.009687+0000","last_active":"2023-11-28T17:09:02.222456+0000","last_peered":"2023-11-28T17:09:02.222456+0000","last_clean":"2023-11-28T17:09:02.222456+0000","last_became_active":"2023-11-28T17:09:01.009329+0000","last_became_peered":"2023-11-28T17:09:01.009329+0000","last_unstale":"2023-11-28T17:09:02.222456+0000","last_undegraded":"2023-11-28T17:09:02.222456+0000","last_fullsized":"2023-11-28T17:09:02.222456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:50.317730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.991649+0000","last_change":"2023-11-28T17:09:00.269333+0000","last_active":"2023-11-28T17:09:00.991649+0000","last_peered":"2023-11-28T17:09:00.991649+0000","last_clean":"2023-11-28T17:09:00.991649+0000","last_became_active":"2023-11-28T17:09:00.267481+0000","last_became_peered":"2023-11-28T17:09:00.267481+0000","last_unstale":"2023-11-28T17:09:00.991649+0000","last_undegraded":"2023-11-28T17:09:00.991649+0000","last_fullsized":"2023-11-28T17:09:00.991649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:21.776220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512505+0000","last_change":"2023-11-28T17:09:01.012458+0000","last_active":"2023-11-28T17:09:02.512505+0000","last_peered":"2023-11-28T17:09:02.512505+0000","last_clean":"2023-11-28T17:09:02.512505+0000","last_became_active":"2023-11-28T17:09:01.011811+0000","last_became_peered":"2023-11-28T17:09:01.011811+0000","last_unstale":"2023-11-28T17:09:02.512505+0000","last_undegraded":"2023-11-28T17:09:02.512505+0000","last_fullsized":"2023-11-28T17:09:02.512505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:24.354633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.999956+0000","last_change":"2023-11-28T17:09:00.263517+0000","last_active":"2023-11-28T17:09:00.999956+0000","last_peered":"2023-11-28T17:09:00.999956+0000","last_clean":"2023-11-28T17:09:00.999956+0000","last_became_active":"2023-11-28T17:09:00.263243+0000","last_became_peered":"2023-11-28T17:09:00.263243+0000","last_unstale":"2023-11-28T17:09:00.999956+0000","last_undegraded":"2023-11-28T17:09:00.999956+0000","last_fullsized":"2023-11-28T17:09:00.999956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:10.177210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515382+0000","last_change":"2023-11-28T17:09:00.997629+0000","last_active":"2023-11-28T17:09:02.515382+0000","last_peered":"2023-11-28T17:09:02.515382+0000","last_clean":"2023-11-28T17:09:02.515382+0000","last_became_active":"2023-11-28T17:09:00.997360+0000","last_became_peered":"2023-11-28T17:09:00.997360+0000","last_unstale":"2023-11-28T17:09:02.515382+0000","last_undegraded":"2023-11-28T17:09:02.515382+0000","last_fullsized":"2023-11-28T17:09:02.515382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:32.388198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.988631+0000","last_change":"2023-11-28T17:09:00.268166+0000","last_active":"2023-11-28T17:09:00.988631+0000","last_peered":"2023-11-28T17:09:00.988631+0000","last_clean":"2023-11-28T17:09:00.988631+0000","last_became_active":"2023-11-28T17:09:00.267609+0000","last_became_peered":"2023-11-28T17:09:00.267609+0000","last_unstale":"2023-11-28T17:09:00.988631+0000","last_undegraded":"2023-11-28T17:09:00.988631+0000","last_fullsized":"2023-11-28T17:09:00.988631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:23:26.227274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.989182+0000","last_change":"2023-11-28T17:09:01.017987+0000","last_active":"2023-11-28T17:09:01.989182+0000","last_peered":"2023-11-28T17:09:01.989182+0000","last_clean":"2023-11-28T17:09:01.989182+0000","last_became_active":"2023-11-28T17:09:01.017491+0000","last_became_peered":"2023-11-28T17:09:01.017491+0000","last_unstale":"2023-11-28T17:09:01.989182+0000","last_undegraded":"2023-11-28T17:09:01.989182+0000","last_fullsized":"2023-11-28T17:09:01.989182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:09.387207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.996053+0000","last_change":"2023-11-28T17:09:00.267582+0000","last_active":"2023-11-28T17:09:00.996053+0000","last_peered":"2023-11-28T17:09:00.996053+0000","last_clean":"2023-11-28T17:09:00.996053+0000","last_became_active":"2023-11-28T17:09:00.267245+0000","last_became_peered":"2023-11-28T17:09:00.267245+0000","last_unstale":"2023-11-28T17:09:00.996053+0000","last_undegraded":"2023-11-28T17:09:00.996053+0000","last_fullsized":"2023-11-28T17:09:00.996053+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:59.024219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.990398+0000","last_change":"2023-11-28T17:09:00.997521+0000","last_active":"2023-11-28T17:09:01.990398+0000","last_peered":"2023-11-28T17:09:01.990398+0000","last_clean":"2023-11-28T17:09:01.990398+0000","last_became_active":"2023-11-28T17:09:00.997041+0000","last_became_peered":"2023-11-28T17:09:00.997041+0000","last_unstale":"2023-11-28T17:09:01.990398+0000","last_undegraded":"2023-11-28T17:09:01.990398+0000","last_fullsized":"2023-11-28T17:09:01.990398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:07.593383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276430+0000","last_change":"2023-11-28T17:09:00.264837+0000","last_active":"2023-11-28T17:09:01.276430+0000","last_peered":"2023-11-28T17:09:01.276430+0000","last_clean":"2023-11-28T17:09:01.276430+0000","last_became_active":"2023-11-28T17:09:00.264570+0000","last_became_peered":"2023-11-28T17:09:00.264570+0000","last_unstale":"2023-11-28T17:09:01.276430+0000","last_undegraded":"2023-11-28T17:09:01.276430+0000","last_fullsized":"2023-11-28T17:09:01.276430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:23.343512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.990475+0000","last_change":"2023-11-28T17:09:01.006414+0000","last_active":"2023-11-28T17:09:01.990475+0000","last_peered":"2023-11-28T17:09:01.990475+0000","last_clean":"2023-11-28T17:09:01.990475+0000","last_became_active":"2023-11-28T17:09:01.005570+0000","last_became_peered":"2023-11-28T17:09:01.005570+0000","last_unstale":"2023-11-28T17:09:01.990475+0000","last_undegraded":"2023-11-28T17:09:01.990475+0000","last_fullsized":"2023-11-28T17:09:01.990475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:46:24.200692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.998379+0000","last_change":"2023-11-28T17:09:00.257189+0000","last_active":"2023-11-28T17:09:00.998379+0000","last_peered":"2023-11-28T17:09:00.998379+0000","last_clean":"2023-11-28T17:09:00.998379+0000","last_became_active":"2023-11-28T17:09:00.256862+0000","last_became_peered":"2023-11-28T17:09:00.256862+0000","last_unstale":"2023-11-28T17:09:00.998379+0000","last_undegraded":"2023-11-28T17:09:00.998379+0000","last_fullsized":"2023-11-28T17:09:00.998379+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:24.313840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987313+0000","last_change":"2023-11-28T17:09:01.006232+0000","last_active":"2023-11-28T17:09:01.987313+0000","last_peered":"2023-11-28T17:09:01.987313+0000","last_clean":"2023-11-28T17:09:01.987313+0000","last_became_active":"2023-11-28T17:09:01.005930+0000","last_became_peered":"2023-11-28T17:09:01.005930+0000","last_unstale":"2023-11-28T17:09:01.987313+0000","last_undegraded":"2023-11-28T17:09:01.987313+0000","last_fullsized":"2023-11-28T17:09:01.987313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:06:27.104365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.991766+0000","last_change":"2023-11-28T17:09:00.266555+0000","last_active":"2023-11-28T17:09:00.991766+0000","last_peered":"2023-11-28T17:09:00.991766+0000","last_clean":"2023-11-28T17:09:00.991766+0000","last_became_active":"2023-11-28T17:09:00.266279+0000","last_became_peered":"2023-11-28T17:09:00.266279+0000","last_unstale":"2023-11-28T17:09:00.991766+0000","last_undegraded":"2023-11-28T17:09:00.991766+0000","last_fullsized":"2023-11-28T17:09:00.991766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:19.098061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988255+0000","last_change":"2023-11-28T17:09:01.007494+0000","last_active":"2023-11-28T17:09:01.988255+0000","last_peered":"2023-11-28T17:09:01.988255+0000","last_clean":"2023-11-28T17:09:01.988255+0000","last_became_active":"2023-11-28T17:09:01.007121+0000","last_became_peered":"2023-11-28T17:09:01.007121+0000","last_unstale":"2023-11-28T17:09:01.988255+0000","last_undegraded":"2023-11-28T17:09:01.988255+0000","last_fullsized":"2023-11-28T17:09:01.988255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:11.188332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.984942+0000","last_change":"2023-11-28T17:09:00.267476+0000","last_active":"2023-11-28T17:09:00.984942+0000","last_peered":"2023-11-28T17:09:00.984942+0000","last_clean":"2023-11-28T17:09:00.984942+0000","last_became_active":"2023-11-28T17:09:00.267138+0000","last_became_peered":"2023-11-28T17:09:00.267138+0000","last_unstale":"2023-11-28T17:09:00.984942+0000","last_undegraded":"2023-11-28T17:09:00.984942+0000","last_fullsized":"2023-11-28T17:09:00.984942+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:11.303292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512236+0000","last_change":"2023-11-28T17:09:01.008773+0000","last_active":"2023-11-28T17:09:02.512236+0000","last_peered":"2023-11-28T17:09:02.512236+0000","last_clean":"2023-11-28T17:09:02.512236+0000","last_became_active":"2023-11-28T17:09:01.008241+0000","last_became_peered":"2023-11-28T17:09:01.008241+0000","last_unstale":"2023-11-28T17:09:02.512236+0000","last_undegraded":"2023-11-28T17:09:02.512236+0000","last_fullsized":"2023-11-28T17:09:02.512236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:12.314413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276631+0000","last_change":"2023-11-28T17:09:00.267671+0000","last_active":"2023-11-28T17:09:01.276631+0000","last_peered":"2023-11-28T17:09:01.276631+0000","last_clean":"2023-11-28T17:09:01.276631+0000","last_became_active":"2023-11-28T17:09:00.264445+0000","last_became_peered":"2023-11-28T17:09:00.264445+0000","last_unstale":"2023-11-28T17:09:01.276631+0000","last_undegraded":"2023-11-28T17:09:01.276631+0000","last_fullsized":"2023-11-28T17:09:01.276631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:52.364620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987244+0000","last_change":"2023-11-28T17:09:01.003981+0000","last_active":"2023-11-28T17:09:01.987244+0000","last_peered":"2023-11-28T17:09:01.987244+0000","last_clean":"2023-11-28T17:09:01.987244+0000","last_became_active":"2023-11-28T17:09:01.003508+0000","last_became_peered":"2023-11-28T17:09:01.003508+0000","last_unstale":"2023-11-28T17:09:01.987244+0000","last_undegraded":"2023-11-28T17:09:01.987244+0000","last_fullsized":"2023-11-28T17:09:01.987244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:48:31.212857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988121+0000","last_change":"2023-11-28T17:09:01.012859+0000","last_active":"2023-11-28T17:09:01.988121+0000","last_peered":"2023-11-28T17:09:01.988121+0000","last_clean":"2023-11-28T17:09:01.988121+0000","last_became_active":"2023-11-28T17:09:01.010930+0000","last_became_peered":"2023-11-28T17:09:01.010930+0000","last_unstale":"2023-11-28T17:09:01.988121+0000","last_undegraded":"2023-11-28T17:09:01.988121+0000","last_fullsized":"2023-11-28T17:09:01.988121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:46:01.399129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.268661+0000","last_change":"2023-11-28T17:09:00.259200+0000","last_active":"2023-11-28T17:09:01.268661+0000","last_peered":"2023-11-28T17:09:01.268661+0000","last_clean":"2023-11-28T17:09:01.268661+0000","last_became_active":"2023-11-28T17:09:00.258886+0000","last_became_peered":"2023-11-28T17:09:00.258886+0000","last_unstale":"2023-11-28T17:09:01.268661+0000","last_undegraded":"2023-11-28T17:09:01.268661+0000","last_fullsized":"2023-11-28T17:09:01.268661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:43.640066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.514743+0000","last_change":"2023-11-28T17:09:01.015028+0000","last_active":"2023-11-28T17:09:02.514743+0000","last_peered":"2023-11-28T17:09:02.514743+0000","last_clean":"2023-11-28T17:09:02.514743+0000","last_became_active":"2023-11-28T17:09:01.014670+0000","last_became_peered":"2023-11-28T17:09:01.014670+0000","last_unstale":"2023-11-28T17:09:02.514743+0000","last_undegraded":"2023-11-28T17:09:02.514743+0000","last_fullsized":"2023-11-28T17:09:02.514743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:31.569896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.268602+0000","last_change":"2023-11-28T17:09:00.263220+0000","last_active":"2023-11-28T17:09:01.268602+0000","last_peered":"2023-11-28T17:09:01.268602+0000","last_clean":"2023-11-28T17:09:01.268602+0000","last_became_active":"2023-11-28T17:09:00.262930+0000","last_became_peered":"2023-11-28T17:09:00.262930+0000","last_unstale":"2023-11-28T17:09:01.268602+0000","last_undegraded":"2023-11-28T17:09:01.268602+0000","last_fullsized":"2023-11-28T17:09:01.268602+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:06.582262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223461+0000","last_change":"2023-11-28T17:09:01.014772+0000","last_active":"2023-11-28T17:09:02.223461+0000","last_peered":"2023-11-28T17:09:02.223461+0000","last_clean":"2023-11-28T17:09:02.223461+0000","last_became_active":"2023-11-28T17:09:01.013249+0000","last_became_peered":"2023-11-28T17:09:01.013249+0000","last_unstale":"2023-11-28T17:09:02.223461+0000","last_undegraded":"2023-11-28T17:09:02.223461+0000","last_fullsized":"2023-11-28T17:09:02.223461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:34.895213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273335+0000","last_change":"2023-11-28T17:09:00.270822+0000","last_active":"2023-11-28T17:09:01.273335+0000","last_peered":"2023-11-28T17:09:01.273335+0000","last_clean":"2023-11-28T17:09:01.273335+0000","last_became_active":"2023-11-28T17:09:00.268995+0000","last_became_peered":"2023-11-28T17:09:00.268995+0000","last_unstale":"2023-11-28T17:09:01.273335+0000","last_undegraded":"2023-11-28T17:09:01.273335+0000","last_fullsized":"2023-11-28T17:09:01.273335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:24.313840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223456+0000","last_change":"2023-11-28T17:09:01.012994+0000","last_active":"2023-11-28T17:09:02.223456+0000","last_peered":"2023-11-28T17:09:02.223456+0000","last_clean":"2023-11-28T17:09:02.223456+0000","last_became_active":"2023-11-28T17:09:01.012496+0000","last_became_peered":"2023-11-28T17:09:01.012496+0000","last_unstale":"2023-11-28T17:09:02.223456+0000","last_undegraded":"2023-11-28T17:09:02.223456+0000","last_fullsized":"2023-11-28T17:09:02.223456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:59.985963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276064+0000","last_change":"2023-11-28T17:09:00.265200+0000","last_active":"2023-11-28T17:09:01.276064+0000","last_peered":"2023-11-28T17:09:01.276064+0000","last_clean":"2023-11-28T17:09:01.276064+0000","last_became_active":"2023-11-28T17:09:00.264913+0000","last_became_peered":"2023-11-28T17:09:00.264913+0000","last_unstale":"2023-11-28T17:09:01.276064+0000","last_undegraded":"2023-11-28T17:09:01.276064+0000","last_fullsized":"2023-11-28T17:09:01.276064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:24.313840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.446}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.875}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3069999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5740000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.97}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.696}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":806912,"data_stored":661370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.774}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.595}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2869999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2320000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.032}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":73194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4390000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1679999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.075}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.49}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.222}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7390000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8660000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5940000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.78}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.665}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.788}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8839999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8600000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":70966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.425}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T17:09:05.738 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:09:05.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.990+0000 7f2b9efdc640 1 Processor -- start 2023-11-28T17:09:05.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.990+0000 7f2b9efdc640 1 -- start start 2023-11-28T17:09:05.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.992+0000 7f2b9efdc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b98157b70 0x7f2b980b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:05.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.992+0000 7f2b9efdc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2b980b4a80 0x7f2b9813f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:05.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.992+0000 7f2b9efdc640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b9815dac0 con 0x7f2b980b4620 2023-11-28T17:09:05.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.992+0000 7f2b9efdc640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b9815e920 con 0x7f2b980b4a80 2023-11-28T17:09:05.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.992+0000 7f2b9efdc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9815dd10 con 0x7f2b98157b70 2023-11-28T17:09:05.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.993+0000 7f2b9cd78640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b98157b70 0x7f2b980b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:05.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.993+0000 7f2b9cd78640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b98157b70 0x7f2b980b2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:55768/0 (socket says 172.21.15.116:55768) 2023-11-28T17:09:05.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.993+0000 7f2b9cd78640 1 -- 172.21.15.116:0/1794176738 learned_addr learned my addr 172.21.15.116:0/1794176738 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:05.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.993+0000 7f2b9d579640 1 --2- 172.21.15.116:0/1794176738 >> v2:172.21.15.156:3300/0 conn(0x7f2b980b4a80 0x7f2b9813f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:05.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.993+0000 7f2b9cd78640 1 -- 172.21.15.116:0/1794176738 >> v2:172.21.15.156:3300/0 conn(0x7f2b980b4a80 msgr2=0x7f2b9813f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:05.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.993+0000 7f2b9cd78640 1 --2- 172.21.15.116:0/1794176738 >> v2:172.21.15.156:3300/0 conn(0x7f2b980b4a80 0x7f2b9813f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.993+0000 7f2b9cd78640 1 -- 172.21.15.116:0/1794176738 >> v1:172.21.15.156:6789/0 conn(0x7f2b980b4620 legacy=0x7f2b9806d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:05.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.993+0000 7f2b9cd78640 1 -- 172.21.15.116:0/1794176738 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9815dff0 con 0x7f2b98157b70 2023-11-28T17:09:05.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.995+0000 7f2b9cd78640 1 --2- 172.21.15.116:0/1794176738 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b98157b70 0x7f2b980b2f70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2b94009e80 tx=0x7f2b9402bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c1643ec8334139 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:05.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.995+0000 7f2b8f7fe640 1 -- 172.21.15.116:0/1794176738 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9402caf0 con 0x7f2b98157b70 2023-11-28T17:09:05.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.995+0000 7f2b8f7fe640 1 -- 172.21.15.116:0/1794176738 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9402cc50 con 0x7f2b98157b70 2023-11-28T17:09:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.995+0000 7f2b8f7fe640 1 -- 172.21.15.116:0/1794176738 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b940343f0 con 0x7f2b98157b70 2023-11-28T17:09:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.996+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1794176738 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b98157b70 msgr2=0x7f2b980b2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.996+0000 7f2b9efdc640 1 --2- 172.21.15.116:0/1794176738 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b98157b70 0x7f2b980b2f70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2b94009e80 tx=0x7f2b9402bf50 comp rx=0 tx=0).stop 2023-11-28T17:09:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.996+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1794176738 shutdown_connections 2023-11-28T17:09:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.996+0000 7f2b9efdc640 1 --2- 172.21.15.116:0/1794176738 >> v2:172.21.15.156:3300/0 conn(0x7f2b980b4a80 0x7f2b9813f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.996+0000 7f2b9efdc640 1 --2- 172.21.15.116:0/1794176738 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b98157b70 0x7f2b980b2f70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:05.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.996+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1794176738 >> 172.21.15.116:0/1794176738 conn(0x7f2b980cfec0 msgr2=0x7f2b980d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:05.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.996+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1794176738 shutdown_connections 2023-11-28T17:09:05.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.996+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1794176738 wait complete. 2023-11-28T17:09:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.998+0000 7f2b9efdc640 1 Processor -- start 2023-11-28T17:09:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.998+0000 7f2b9efdc640 1 -- start start 2023-11-28T17:09:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9efdc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b980b4620 0x7f2b9813caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9efdc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b98157b70 0x7f2b9806aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9efdc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b9806afa0 0x7f2b980a8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9efdc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9806b430 con 0x7f2b9806afa0 2023-11-28T17:09:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9efdc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9813d030 con 0x7f2b98157b70 2023-11-28T17:09:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9efdc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b980a94c0 con 0x7f2b980b4620 2023-11-28T17:09:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9d579640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b9806afa0 0x7f2b980a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9d579640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b9806afa0 0x7f2b980a8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43674/0 (socket says 172.21.15.116:43674) 2023-11-28T17:09:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9d579640 1 -- 172.21.15.116:0/1149194998 learned_addr learned my addr 172.21.15.116:0/1149194998 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9cd78640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b980b4620 0x7f2b9813caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b8ffff640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b98157b70 0x7f2b9806aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9d579640 1 -- 172.21.15.116:0/1149194998 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b980b4620 msgr2=0x7f2b9813caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9d579640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b980b4620 0x7f2b9813caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9d579640 1 -- 172.21.15.116:0/1149194998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b98157b70 msgr2=0x7f2b9806aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9d579640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b98157b70 0x7f2b9806aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b9d579640 1 -- 172.21.15.116:0/1149194998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b94009b30 con 0x7f2b9806afa0 2023-11-28T17:09:06.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:05.999+0000 7f2b8ffff640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b98157b70 0x7f2b9806aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:06.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.000+0000 7f2b9d579640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b9806afa0 0x7f2b980a8710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000e930 tx=0x7f2b9000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.000+0000 7f2b8d7fa640 1 -- 172.21.15.116:0/1149194998 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90009690 con 0x7f2b9806afa0 2023-11-28T17:09:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.000+0000 7f2b8d7fa640 1 -- 172.21.15.116:0/1149194998 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9000c980 con 0x7f2b9806afa0 2023-11-28T17:09:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.000+0000 7f2b8d7fa640 1 -- 172.21.15.116:0/1149194998 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9000cd30 con 0x7f2b9806afa0 2023-11-28T17:09:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.000+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1149194998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b980a97a0 con 0x7f2b9806afa0 2023-11-28T17:09:06.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.000+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1149194998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b980a9d60 con 0x7f2b9806afa0 2023-11-28T17:09:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.001+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1149194998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b98067460 con 0x7f2b9806afa0 2023-11-28T17:09:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.002+0000 7f2b8d7fa640 1 -- 172.21.15.116:0/1149194998 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2b90013470 con 0x7f2b9806afa0 2023-11-28T17:09:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.003+0000 7f2b8d7fa640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2b8406b2a0 0x7f2b8406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.003+0000 7f2b8d7fa640 1 -- 172.21.15.116:0/1149194998 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b90089e80 con 0x7f2b9806afa0 2023-11-28T17:09:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.003+0000 7f2b9cd78640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2b8406b2a0 0x7f2b8406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:06.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.004+0000 7f2b9cd78640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2b8406b2a0 0x7f2b8406d760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2b94035000 tx=0x7f2b94030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:06.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.008+0000 7f2b8d7fa640 1 -- 172.21.15.116:0/1149194998 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b90054d10 con 0x7f2b9806afa0 2023-11-28T17:09:06.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.196+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1149194998 --> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b9815e5b0 con 0x7f2b8406b2a0 2023-11-28T17:09:06.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.216+0000 7f2b8d7fa640 1 -- 172.21.15.116:0/1149194998 <== mgr.4098 v2:172.21.15.116:6832/1077400178 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264338 (secure 0 0 0) 0x7f2b94009510 con 0x7f2b8406b2a0 2023-11-28T17:09:06.220 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:06.222 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-28T17:09:06.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.221+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1149194998 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2b8406b2a0 msgr2=0x7f2b8406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.221+0000 7f2b9efdc640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2b8406b2a0 0x7f2b8406d760 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2b94035000 tx=0x7f2b94030000 comp rx=0 tx=0).stop 2023-11-28T17:09:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.221+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1149194998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b9806afa0 msgr2=0x7f2b980a8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.221+0000 7f2b9efdc640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b9806afa0 0x7f2b980a8710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000e930 tx=0x7f2b9000edf0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.222+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1149194998 shutdown_connections 2023-11-28T17:09:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.222+0000 7f2b9efdc640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2b8406b2a0 0x7f2b8406d760 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.222+0000 7f2b9efdc640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b9806afa0 0x7f2b980a8710 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.222+0000 7f2b9efdc640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b98157b70 0x7f2b9806aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.222+0000 7f2b9efdc640 1 --2- 172.21.15.116:0/1149194998 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b980b4620 0x7f2b9813caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.222+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1149194998 >> 172.21.15.116:0/1149194998 conn(0x7f2b980cfec0 msgr2=0x7f2b9815b690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.222+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1149194998 shutdown_connections 2023-11-28T17:09:06.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.222+0000 7f2b9efdc640 1 -- 172.21.15.116:0/1149194998 wait complete. 2023-11-28T17:09:06.243 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2023-11-28T17:09:05.040444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":162,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235648,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739072,"statfs":{"total":773094113280,"available":772852809728,"internally_reserved":0,"allocated":2891776,"data_stored":1752836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.819595"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513632+0000","last_change":"2023-11-28T17:09:01.014584+0000","last_active":"2023-11-28T17:09:02.513632+0000","last_peered":"2023-11-28T17:09:02.513632+0000","last_clean":"2023-11-28T17:09:02.513632+0000","last_became_active":"2023-11-28T17:09:01.014291+0000","last_became_peered":"2023-11-28T17:09:01.014291+0000","last_unstale":"2023-11-28T17:09:02.513632+0000","last_undegraded":"2023-11-28T17:09:02.513632+0000","last_fullsized":"2023-11-28T17:09:02.513632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:22.787341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223151+0000","last_change":"2023-11-28T17:09:01.009980+0000","last_active":"2023-11-28T17:09:02.223151+0000","last_peered":"2023-11-28T17:09:02.223151+0000","last_clean":"2023-11-28T17:09:02.223151+0000","last_became_active":"2023-11-28T17:09:01.009702+0000","last_became_peered":"2023-11-28T17:09:01.009702+0000","last_unstale":"2023-11-28T17:09:02.223151+0000","last_undegraded":"2023-11-28T17:09:02.223151+0000","last_fullsized":"2023-11-28T17:09:02.223151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:06:59.199376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222772+0000","last_change":"2023-11-28T17:09:01.010483+0000","last_active":"2023-11-28T17:09:02.222772+0000","last_peered":"2023-11-28T17:09:02.222772+0000","last_clean":"2023-11-28T17:09:02.222772+0000","last_became_active":"2023-11-28T17:09:01.010032+0000","last_became_peered":"2023-11-28T17:09:01.010032+0000","last_unstale":"2023-11-28T17:09:02.222772+0000","last_undegraded":"2023-11-28T17:09:02.222772+0000","last_fullsized":"2023-11-28T17:09:02.222772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:46:24.200692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513159+0000","last_change":"2023-11-28T17:09:00.998473+0000","last_active":"2023-11-28T17:09:02.513159+0000","last_peered":"2023-11-28T17:09:02.513159+0000","last_clean":"2023-11-28T17:09:02.513159+0000","last_became_active":"2023-11-28T17:09:00.998206+0000","last_became_peered":"2023-11-28T17:09:00.998206+0000","last_unstale":"2023-11-28T17:09:02.513159+0000","last_undegraded":"2023-11-28T17:09:02.513159+0000","last_fullsized":"2023-11-28T17:09:02.513159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:31.569896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512712+0000","last_change":"2023-11-28T17:09:01.019053+0000","last_active":"2023-11-28T17:09:02.512712+0000","last_peered":"2023-11-28T17:09:02.512712+0000","last_clean":"2023-11-28T17:09:02.512712+0000","last_became_active":"2023-11-28T17:09:01.018689+0000","last_became_peered":"2023-11-28T17:09:01.018689+0000","last_unstale":"2023-11-28T17:09:02.512712+0000","last_undegraded":"2023-11-28T17:09:02.512712+0000","last_fullsized":"2023-11-28T17:09:02.512712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:06:59.199376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513060+0000","last_change":"2023-11-28T17:09:01.000056+0000","last_active":"2023-11-28T17:09:02.513060+0000","last_peered":"2023-11-28T17:09:02.513060+0000","last_clean":"2023-11-28T17:09:02.513060+0000","last_became_active":"2023-11-28T17:09:00.999547+0000","last_became_peered":"2023-11-28T17:09:00.999547+0000","last_unstale":"2023-11-28T17:09:02.513060+0000","last_undegraded":"2023-11-28T17:09:02.513060+0000","last_fullsized":"2023-11-28T17:09:02.513060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:12.314413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.511764+0000","last_change":"2023-11-28T17:09:01.012699+0000","last_active":"2023-11-28T17:09:02.511764+0000","last_peered":"2023-11-28T17:09:02.511764+0000","last_clean":"2023-11-28T17:09:02.511764+0000","last_became_active":"2023-11-28T17:09:01.012442+0000","last_became_peered":"2023-11-28T17:09:01.012442+0000","last_unstale":"2023-11-28T17:09:02.511764+0000","last_undegraded":"2023-11-28T17:09:02.511764+0000","last_fullsized":"2023-11-28T17:09:02.511764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:46:24.200692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223215+0000","last_change":"2023-11-28T17:09:01.016792+0000","last_active":"2023-11-28T17:09:02.223215+0000","last_peered":"2023-11-28T17:09:02.223215+0000","last_clean":"2023-11-28T17:09:02.223215+0000","last_became_active":"2023-11-28T17:09:01.016034+0000","last_became_peered":"2023-11-28T17:09:01.016034+0000","last_unstale":"2023-11-28T17:09:02.223215+0000","last_undegraded":"2023-11-28T17:09:02.223215+0000","last_fullsized":"2023-11-28T17:09:02.223215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:52.224230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515192+0000","last_change":"2023-11-28T17:09:01.007271+0000","last_active":"2023-11-28T17:09:02.515192+0000","last_peered":"2023-11-28T17:09:02.515192+0000","last_clean":"2023-11-28T17:09:02.515192+0000","last_became_active":"2023-11-28T17:09:01.006929+0000","last_became_peered":"2023-11-28T17:09:01.006929+0000","last_unstale":"2023-11-28T17:09:02.515192+0000","last_undegraded":"2023-11-28T17:09:02.515192+0000","last_fullsized":"2023-11-28T17:09:02.515192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:20.109182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515425+0000","last_change":"2023-11-28T17:09:01.012176+0000","last_active":"2023-11-28T17:09:02.515425+0000","last_peered":"2023-11-28T17:09:02.515425+0000","last_clean":"2023-11-28T17:09:02.515425+0000","last_became_active":"2023-11-28T17:09:01.011821+0000","last_became_peered":"2023-11-28T17:09:01.011821+0000","last_unstale":"2023-11-28T17:09:02.515425+0000","last_undegraded":"2023-11-28T17:09:02.515425+0000","last_fullsized":"2023-11-28T17:09:02.515425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:55.626548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513365+0000","last_change":"2023-11-28T17:09:01.008348+0000","last_active":"2023-11-28T17:09:02.513365+0000","last_peered":"2023-11-28T17:09:02.513365+0000","last_clean":"2023-11-28T17:09:02.513365+0000","last_became_active":"2023-11-28T17:09:01.008029+0000","last_became_peered":"2023-11-28T17:09:01.008029+0000","last_unstale":"2023-11-28T17:09:02.513365+0000","last_undegraded":"2023-11-28T17:09:02.513365+0000","last_fullsized":"2023-11-28T17:09:02.513365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:52.224230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512353+0000","last_change":"2023-11-28T17:09:01.013836+0000","last_active":"2023-11-28T17:09:02.512353+0000","last_peered":"2023-11-28T17:09:02.512353+0000","last_clean":"2023-11-28T17:09:02.512353+0000","last_became_active":"2023-11-28T17:09:01.011870+0000","last_became_peered":"2023-11-28T17:09:01.011870+0000","last_unstale":"2023-11-28T17:09:02.512353+0000","last_undegraded":"2023-11-28T17:09:02.512353+0000","last_fullsized":"2023-11-28T17:09:02.512353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:23:27.238395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988425+0000","last_change":"2023-11-28T17:09:01.007854+0000","last_active":"2023-11-28T17:09:01.988425+0000","last_peered":"2023-11-28T17:09:01.988425+0000","last_clean":"2023-11-28T17:09:01.988425+0000","last_became_active":"2023-11-28T17:09:01.007561+0000","last_became_peered":"2023-11-28T17:09:01.007561+0000","last_unstale":"2023-11-28T17:09:01.988425+0000","last_undegraded":"2023-11-28T17:09:01.988425+0000","last_fullsized":"2023-11-28T17:09:01.988425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:53.375742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986992+0000","last_change":"2023-11-28T17:09:01.006142+0000","last_active":"2023-11-28T17:09:01.986992+0000","last_peered":"2023-11-28T17:09:01.986992+0000","last_clean":"2023-11-28T17:09:01.986992+0000","last_became_active":"2023-11-28T17:09:01.005823+0000","last_became_peered":"2023-11-28T17:09:01.005823+0000","last_unstale":"2023-11-28T17:09:01.986992+0000","last_undegraded":"2023-11-28T17:09:01.986992+0000","last_fullsized":"2023-11-28T17:09:01.986992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:32.388198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222442+0000","last_change":"2023-11-28T17:09:01.016320+0000","last_active":"2023-11-28T17:09:02.222442+0000","last_peered":"2023-11-28T17:09:02.222442+0000","last_clean":"2023-11-28T17:09:02.222442+0000","last_became_active":"2023-11-28T17:09:01.016012+0000","last_became_peered":"2023-11-28T17:09:01.016012+0000","last_unstale":"2023-11-28T17:09:02.222442+0000","last_undegraded":"2023-11-28T17:09:02.222442+0000","last_fullsized":"2023-11-28T17:09:02.222442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:44.651188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986526+0000","last_change":"2023-11-28T17:09:00.996901+0000","last_active":"2023-11-28T17:09:01.986526+0000","last_peered":"2023-11-28T17:09:01.986526+0000","last_clean":"2023-11-28T17:09:01.986526+0000","last_became_active":"2023-11-28T17:09:00.996627+0000","last_became_peered":"2023-11-28T17:09:00.996627+0000","last_unstale":"2023-11-28T17:09:01.986526+0000","last_undegraded":"2023-11-28T17:09:01.986526+0000","last_fullsized":"2023-11-28T17:09:01.986526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:45.964344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988654+0000","last_change":"2023-11-28T17:09:01.015527+0000","last_active":"2023-11-28T17:09:01.988654+0000","last_peered":"2023-11-28T17:09:01.988654+0000","last_clean":"2023-11-28T17:09:01.988654+0000","last_became_active":"2023-11-28T17:09:01.015238+0000","last_became_peered":"2023-11-28T17:09:01.015238+0000","last_unstale":"2023-11-28T17:09:01.988654+0000","last_undegraded":"2023-11-28T17:09:01.988654+0000","last_fullsized":"2023-11-28T17:09:01.988654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:44:13.494591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512305+0000","last_change":"2023-11-28T17:09:01.015000+0000","last_active":"2023-11-28T17:09:02.512305+0000","last_peered":"2023-11-28T17:09:02.512305+0000","last_clean":"2023-11-28T17:09:02.512305+0000","last_became_active":"2023-11-28T17:09:01.014649+0000","last_became_peered":"2023-11-28T17:09:01.014649+0000","last_unstale":"2023-11-28T17:09:02.512305+0000","last_undegraded":"2023-11-28T17:09:02.512305+0000","last_fullsized":"2023-11-28T17:09:02.512305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:31.569896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988495+0000","last_change":"2023-11-28T17:09:00.997869+0000","last_active":"2023-11-28T17:09:01.988495+0000","last_peered":"2023-11-28T17:09:01.988495+0000","last_clean":"2023-11-28T17:09:01.988495+0000","last_became_active":"2023-11-28T17:09:00.997614+0000","last_became_peered":"2023-11-28T17:09:00.997614+0000","last_unstale":"2023-11-28T17:09:01.988495+0000","last_undegraded":"2023-11-28T17:09:01.988495+0000","last_fullsized":"2023-11-28T17:09:01.988495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:25:26.955817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513938+0000","last_change":"2023-11-28T17:09:01.011584+0000","last_active":"2023-11-28T17:09:02.513938+0000","last_peered":"2023-11-28T17:09:02.513938+0000","last_clean":"2023-11-28T17:09:02.513938+0000","last_became_active":"2023-11-28T17:09:01.010359+0000","last_became_peered":"2023-11-28T17:09:01.010359+0000","last_unstale":"2023-11-28T17:09:02.513938+0000","last_undegraded":"2023-11-28T17:09:02.513938+0000","last_fullsized":"2023-11-28T17:09:02.513938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:45.964344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.989818+0000","last_change":"2023-11-28T17:09:00.999976+0000","last_active":"2023-11-28T17:09:01.989818+0000","last_peered":"2023-11-28T17:09:01.989818+0000","last_clean":"2023-11-28T17:09:01.989818+0000","last_became_active":"2023-11-28T17:09:00.999647+0000","last_became_peered":"2023-11-28T17:09:00.999647+0000","last_unstale":"2023-11-28T17:09:01.989818+0000","last_undegraded":"2023-11-28T17:09:01.989818+0000","last_fullsized":"2023-11-28T17:09:01.989818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:44.651188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.514610+0000","last_change":"2023-11-28T17:09:01.009510+0000","last_active":"2023-11-28T17:09:02.514610+0000","last_peered":"2023-11-28T17:09:02.514610+0000","last_clean":"2023-11-28T17:09:02.514610+0000","last_became_active":"2023-11-28T17:09:01.009184+0000","last_became_peered":"2023-11-28T17:09:01.009184+0000","last_unstale":"2023-11-28T17:09:02.514610+0000","last_undegraded":"2023-11-28T17:09:02.514610+0000","last_fullsized":"2023-11-28T17:09:02.514610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:12.314413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222577+0000","last_change":"2023-11-28T17:09:01.015602+0000","last_active":"2023-11-28T17:09:02.222577+0000","last_peered":"2023-11-28T17:09:02.222577+0000","last_clean":"2023-11-28T17:09:02.222577+0000","last_became_active":"2023-11-28T17:09:01.012170+0000","last_became_peered":"2023-11-28T17:09:01.012170+0000","last_unstale":"2023-11-28T17:09:02.222577+0000","last_undegraded":"2023-11-28T17:09:02.222577+0000","last_fullsized":"2023-11-28T17:09:02.222577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:07.593383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986512+0000","last_change":"2023-11-28T17:09:01.016062+0000","last_active":"2023-11-28T17:09:01.986512+0000","last_peered":"2023-11-28T17:09:01.986512+0000","last_clean":"2023-11-28T17:09:01.986512+0000","last_became_active":"2023-11-28T17:09:01.015802+0000","last_became_peered":"2023-11-28T17:09:01.015802+0000","last_unstale":"2023-11-28T17:09:01.986512+0000","last_undegraded":"2023-11-28T17:09:01.986512+0000","last_fullsized":"2023-11-28T17:09:01.986512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:34.895213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223456+0000","last_change":"2023-11-28T17:09:01.016389+0000","last_active":"2023-11-28T17:09:02.223456+0000","last_peered":"2023-11-28T17:09:02.223456+0000","last_clean":"2023-11-28T17:09:02.223456+0000","last_became_active":"2023-11-28T17:09:01.011680+0000","last_became_peered":"2023-11-28T17:09:01.011680+0000","last_unstale":"2023-11-28T17:09:02.223456+0000","last_undegraded":"2023-11-28T17:09:02.223456+0000","last_fullsized":"2023-11-28T17:09:02.223456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:22.787341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515917+0000","last_change":"2023-11-28T17:09:01.006281+0000","last_active":"2023-11-28T17:09:02.515917+0000","last_peered":"2023-11-28T17:09:02.515917+0000","last_clean":"2023-11-28T17:09:02.515917+0000","last_became_active":"2023-11-28T17:09:01.000480+0000","last_became_peered":"2023-11-28T17:09:01.000480+0000","last_unstale":"2023-11-28T17:09:02.515917+0000","last_undegraded":"2023-11-28T17:09:02.515917+0000","last_fullsized":"2023-11-28T17:09:02.515917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:25.324961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.989120+0000","last_change":"2023-11-28T17:09:01.002090+0000","last_active":"2023-11-28T17:09:01.989120+0000","last_peered":"2023-11-28T17:09:01.989120+0000","last_clean":"2023-11-28T17:09:01.989120+0000","last_became_active":"2023-11-28T17:09:01.001728+0000","last_became_peered":"2023-11-28T17:09:01.001728+0000","last_unstale":"2023-11-28T17:09:01.989120+0000","last_undegraded":"2023-11-28T17:09:01.989120+0000","last_fullsized":"2023-11-28T17:09:01.989120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:11.188332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988810+0000","last_change":"2023-11-28T17:09:01.013978+0000","last_active":"2023-11-28T17:09:01.988810+0000","last_peered":"2023-11-28T17:09:01.988810+0000","last_clean":"2023-11-28T17:09:01.988810+0000","last_became_active":"2023-11-28T17:09:01.013646+0000","last_became_peered":"2023-11-28T17:09:01.013646+0000","last_unstale":"2023-11-28T17:09:01.988810+0000","last_undegraded":"2023-11-28T17:09:01.988810+0000","last_fullsized":"2023-11-28T17:09:01.988810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:00.035340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222920+0000","last_change":"2023-11-28T17:09:01.013635+0000","last_active":"2023-11-28T17:09:02.222920+0000","last_peered":"2023-11-28T17:09:02.222920+0000","last_clean":"2023-11-28T17:09:02.222920+0000","last_became_active":"2023-11-28T17:09:01.013310+0000","last_became_peered":"2023-11-28T17:09:01.013310+0000","last_unstale":"2023-11-28T17:09:02.222920+0000","last_undegraded":"2023-11-28T17:09:02.222920+0000","last_fullsized":"2023-11-28T17:09:02.222920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:23:27.238395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987149+0000","last_change":"2023-11-28T17:09:01.016976+0000","last_active":"2023-11-28T17:09:01.987149+0000","last_peered":"2023-11-28T17:09:01.987149+0000","last_clean":"2023-11-28T17:09:01.987149+0000","last_became_active":"2023-11-28T17:09:01.016690+0000","last_became_peered":"2023-11-28T17:09:01.016690+0000","last_unstale":"2023-11-28T17:09:01.987149+0000","last_undegraded":"2023-11-28T17:09:01.987149+0000","last_fullsized":"2023-11-28T17:09:01.987149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:55.626548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.511885+0000","last_change":"2023-11-28T17:09:01.012474+0000","last_active":"2023-11-28T17:09:02.511885+0000","last_peered":"2023-11-28T17:09:02.511885+0000","last_clean":"2023-11-28T17:09:02.511885+0000","last_became_active":"2023-11-28T17:09:01.012226+0000","last_became_peered":"2023-11-28T17:09:01.012226+0000","last_unstale":"2023-11-28T17:09:02.511885+0000","last_undegraded":"2023-11-28T17:09:02.511885+0000","last_fullsized":"2023-11-28T17:09:02.511885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:36.438412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512524+0000","last_change":"2023-11-28T17:09:00.994555+0000","last_active":"2023-11-28T17:09:02.512524+0000","last_peered":"2023-11-28T17:09:02.512524+0000","last_clean":"2023-11-28T17:09:02.512524+0000","last_became_active":"2023-11-28T17:09:00.993870+0000","last_became_peered":"2023-11-28T17:09:00.993870+0000","last_unstale":"2023-11-28T17:09:02.512524+0000","last_undegraded":"2023-11-28T17:09:02.512524+0000","last_fullsized":"2023-11-28T17:09:02.512524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:36.438412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.514893+0000","last_change":"2023-11-28T17:09:01.018035+0000","last_active":"2023-11-28T17:09:02.514893+0000","last_peered":"2023-11-28T17:09:02.514893+0000","last_clean":"2023-11-28T17:09:02.514893+0000","last_became_active":"2023-11-28T17:09:01.017735+0000","last_became_peered":"2023-11-28T17:09:01.017735+0000","last_unstale":"2023-11-28T17:09:02.514893+0000","last_undegraded":"2023-11-28T17:09:02.514893+0000","last_fullsized":"2023-11-28T17:09:02.514893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:32.388198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.271905+0000","last_change":"2023-11-28T17:09:00.266501+0000","last_active":"2023-11-28T17:09:01.271905+0000","last_peered":"2023-11-28T17:09:01.271905+0000","last_clean":"2023-11-28T17:09:01.271905+0000","last_became_active":"2023-11-28T17:09:00.266177+0000","last_became_peered":"2023-11-28T17:09:00.266177+0000","last_unstale":"2023-11-28T17:09:01.271905+0000","last_undegraded":"2023-11-28T17:09:01.271905+0000","last_fullsized":"2023-11-28T17:09:01.271905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:08.376086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.990190+0000","last_change":"2023-11-28T17:09:01.008791+0000","last_active":"2023-11-28T17:09:01.990190+0000","last_peered":"2023-11-28T17:09:01.990190+0000","last_clean":"2023-11-28T17:09:01.990190+0000","last_became_active":"2023-11-28T17:09:01.008266+0000","last_became_peered":"2023-11-28T17:09:01.008266+0000","last_unstale":"2023-11-28T17:09:01.990190+0000","last_undegraded":"2023-11-28T17:09:01.990190+0000","last_fullsized":"2023-11-28T17:09:01.990190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:50.317730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.275936+0000","last_change":"2023-11-28T17:09:00.269760+0000","last_active":"2023-11-28T17:09:01.275936+0000","last_peered":"2023-11-28T17:09:01.275936+0000","last_clean":"2023-11-28T17:09:01.275936+0000","last_became_active":"2023-11-28T17:09:00.269082+0000","last_became_peered":"2023-11-28T17:09:00.269082+0000","last_unstale":"2023-11-28T17:09:01.275936+0000","last_undegraded":"2023-11-28T17:09:01.275936+0000","last_fullsized":"2023-11-28T17:09:01.275936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:51.213109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.986115+0000","last_change":"2023-11-28T17:09:00.262912+0000","last_active":"2023-11-28T17:09:00.986115+0000","last_peered":"2023-11-28T17:09:00.986115+0000","last_clean":"2023-11-28T17:09:00.986115+0000","last_became_active":"2023-11-28T17:09:00.260568+0000","last_became_peered":"2023-11-28T17:09:00.260568+0000","last_unstale":"2023-11-28T17:09:00.986115+0000","last_undegraded":"2023-11-28T17:09:00.986115+0000","last_fullsized":"2023-11-28T17:09:00.986115+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:30.558775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223014+0000","last_change":"2023-11-28T17:09:01.012823+0000","last_active":"2023-11-28T17:09:02.223014+0000","last_peered":"2023-11-28T17:09:02.223014+0000","last_clean":"2023-11-28T17:09:02.223014+0000","last_became_active":"2023-11-28T17:09:01.012514+0000","last_became_peered":"2023-11-28T17:09:01.012514+0000","last_unstale":"2023-11-28T17:09:02.223014+0000","last_undegraded":"2023-11-28T17:09:02.223014+0000","last_fullsized":"2023-11-28T17:09:02.223014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:24.354633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.985898+0000","last_change":"2023-11-28T17:09:00.262108+0000","last_active":"2023-11-28T17:09:00.985898+0000","last_peered":"2023-11-28T17:09:00.985898+0000","last_clean":"2023-11-28T17:09:00.985898+0000","last_became_active":"2023-11-28T17:09:00.261621+0000","last_became_peered":"2023-11-28T17:09:00.261621+0000","last_unstale":"2023-11-28T17:09:00.985898+0000","last_undegraded":"2023-11-28T17:09:00.985898+0000","last_fullsized":"2023-11-28T17:09:00.985898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:14:52.843064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988589+0000","last_change":"2023-11-28T17:09:01.011726+0000","last_active":"2023-11-28T17:09:01.988589+0000","last_peered":"2023-11-28T17:09:01.988589+0000","last_clean":"2023-11-28T17:09:01.988589+0000","last_became_active":"2023-11-28T17:09:01.010904+0000","last_became_peered":"2023-11-28T17:09:01.010904+0000","last_unstale":"2023-11-28T17:09:01.988589+0000","last_undegraded":"2023-11-28T17:09:01.988589+0000","last_fullsized":"2023-11-28T17:09:01.988589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:14:53.854185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.990133+0000","last_change":"2023-11-28T17:09:00.262360+0000","last_active":"2023-11-28T17:09:00.990133+0000","last_peered":"2023-11-28T17:09:00.990133+0000","last_clean":"2023-11-28T17:09:00.990133+0000","last_became_active":"2023-11-28T17:09:00.261943+0000","last_became_peered":"2023-11-28T17:09:00.261943+0000","last_unstale":"2023-11-28T17:09:00.990133+0000","last_undegraded":"2023-11-28T17:09:00.990133+0000","last_fullsized":"2023-11-28T17:09:00.990133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:52.998404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515909+0000","last_change":"2023-11-28T17:09:01.007941+0000","last_active":"2023-11-28T17:09:02.515909+0000","last_peered":"2023-11-28T17:09:02.515909+0000","last_clean":"2023-11-28T17:09:02.515909+0000","last_became_active":"2023-11-28T17:09:01.002743+0000","last_became_peered":"2023-11-28T17:09:01.002743+0000","last_unstale":"2023-11-28T17:09:02.515909+0000","last_undegraded":"2023-11-28T17:09:02.515909+0000","last_fullsized":"2023-11-28T17:09:02.515909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:20.109182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.997425+0000","last_change":"2023-11-28T17:09:00.259873+0000","last_active":"2023-11-28T17:09:00.997425+0000","last_peered":"2023-11-28T17:09:00.997425+0000","last_clean":"2023-11-28T17:09:00.997425+0000","last_became_active":"2023-11-28T17:09:00.259506+0000","last_became_peered":"2023-11-28T17:09:00.259506+0000","last_unstale":"2023-11-28T17:09:00.997425+0000","last_undegraded":"2023-11-28T17:09:00.997425+0000","last_fullsized":"2023-11-28T17:09:00.997425+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:19.098061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223644+0000","last_change":"2023-11-28T17:09:01.017611+0000","last_active":"2023-11-28T17:09:02.223644+0000","last_peered":"2023-11-28T17:09:02.223644+0000","last_clean":"2023-11-28T17:09:02.223644+0000","last_became_active":"2023-11-28T17:09:01.017269+0000","last_became_peered":"2023-11-28T17:09:01.017269+0000","last_unstale":"2023-11-28T17:09:02.223644+0000","last_undegraded":"2023-11-28T17:09:02.223644+0000","last_fullsized":"2023-11-28T17:09:02.223644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:45.964344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.272853+0000","last_change":"2023-11-28T17:09:00.264904+0000","last_active":"2023-11-28T17:09:01.272853+0000","last_peered":"2023-11-28T17:09:01.272853+0000","last_clean":"2023-11-28T17:09:01.272853+0000","last_became_active":"2023-11-28T17:09:00.264645+0000","last_became_peered":"2023-11-28T17:09:00.264645+0000","last_unstale":"2023-11-28T17:09:01.272853+0000","last_undegraded":"2023-11-28T17:09:01.272853+0000","last_fullsized":"2023-11-28T17:09:01.272853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:25:59.069890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987426+0000","last_change":"2023-11-28T17:09:01.005502+0000","last_active":"2023-11-28T17:09:01.987426+0000","last_peered":"2023-11-28T17:09:01.987426+0000","last_clean":"2023-11-28T17:09:01.987426+0000","last_became_active":"2023-11-28T17:09:01.004783+0000","last_became_peered":"2023-11-28T17:09:01.004783+0000","last_unstale":"2023-11-28T17:09:01.987426+0000","last_undegraded":"2023-11-28T17:09:01.987426+0000","last_fullsized":"2023-11-28T17:09:01.987426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:10:50.736315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276609+0000","last_change":"2023-11-28T17:09:00.263414+0000","last_active":"2023-11-28T17:09:01.276609+0000","last_peered":"2023-11-28T17:09:01.276609+0000","last_clean":"2023-11-28T17:09:01.276609+0000","last_became_active":"2023-11-28T17:09:00.260801+0000","last_became_peered":"2023-11-28T17:09:00.260801+0000","last_unstale":"2023-11-28T17:09:01.276609+0000","last_undegraded":"2023-11-28T17:09:01.276609+0000","last_fullsized":"2023-11-28T17:09:01.276609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:31:18.731288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513670+0000","last_change":"2023-11-28T17:09:01.019149+0000","last_active":"2023-11-28T17:09:02.513670+0000","last_peered":"2023-11-28T17:09:02.513670+0000","last_clean":"2023-11-28T17:09:02.513670+0000","last_became_active":"2023-11-28T17:09:01.018834+0000","last_became_peered":"2023-11-28T17:09:01.018834+0000","last_unstale":"2023-11-28T17:09:02.513670+0000","last_undegraded":"2023-11-28T17:09:02.513670+0000","last_fullsized":"2023-11-28T17:09:02.513670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:34.895213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.275750+0000","last_change":"2023-11-28T17:09:00.271771+0000","last_active":"2023-11-28T17:09:01.275750+0000","last_peered":"2023-11-28T17:09:01.275750+0000","last_clean":"2023-11-28T17:09:01.275750+0000","last_became_active":"2023-11-28T17:09:00.269930+0000","last_became_peered":"2023-11-28T17:09:00.269930+0000","last_unstale":"2023-11-28T17:09:01.275750+0000","last_undegraded":"2023-11-28T17:09:01.275750+0000","last_fullsized":"2023-11-28T17:09:01.275750+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:33.502573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987179+0000","last_change":"2023-11-28T17:09:00.999606+0000","last_active":"2023-11-28T17:09:01.987179+0000","last_peered":"2023-11-28T17:09:01.987179+0000","last_clean":"2023-11-28T17:09:01.987179+0000","last_became_active":"2023-11-28T17:09:00.999346+0000","last_became_peered":"2023-11-28T17:09:00.999346+0000","last_unstale":"2023-11-28T17:09:01.987179+0000","last_undegraded":"2023-11-28T17:09:01.987179+0000","last_fullsized":"2023-11-28T17:09:01.987179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:25.324961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.991340+0000","last_change":"2023-11-28T17:09:00.269952+0000","last_active":"2023-11-28T17:09:00.991340+0000","last_peered":"2023-11-28T17:09:00.991340+0000","last_clean":"2023-11-28T17:09:00.991340+0000","last_became_active":"2023-11-28T17:09:00.269539+0000","last_became_peered":"2023-11-28T17:09:00.269539+0000","last_unstale":"2023-11-28T17:09:00.991340+0000","last_undegraded":"2023-11-28T17:09:00.991340+0000","last_fullsized":"2023-11-28T17:09:00.991340+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:58.974841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987385+0000","last_change":"2023-11-28T17:09:01.009043+0000","last_active":"2023-11-28T17:09:01.987385+0000","last_peered":"2023-11-28T17:09:01.987385+0000","last_clean":"2023-11-28T17:09:01.987385+0000","last_became_active":"2023-11-28T17:09:01.008739+0000","last_became_peered":"2023-11-28T17:09:01.008739+0000","last_unstale":"2023-11-28T17:09:01.987385+0000","last_undegraded":"2023-11-28T17:09:01.987385+0000","last_fullsized":"2023-11-28T17:09:01.987385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:49.417143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.999207+0000","last_change":"2023-11-28T17:09:00.270474+0000","last_active":"2023-11-28T17:09:00.999207+0000","last_peered":"2023-11-28T17:09:00.999207+0000","last_clean":"2023-11-28T17:09:00.999207+0000","last_became_active":"2023-11-28T17:09:00.270277+0000","last_became_peered":"2023-11-28T17:09:00.270277+0000","last_unstale":"2023-11-28T17:09:00.999207+0000","last_undegraded":"2023-11-28T17:09:00.999207+0000","last_fullsized":"2023-11-28T17:09:00.999207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:52.364620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.511767+0000","last_change":"2023-11-28T17:09:01.016931+0000","last_active":"2023-11-28T17:09:02.511767+0000","last_peered":"2023-11-28T17:09:02.511767+0000","last_clean":"2023-11-28T17:09:02.511767+0000","last_became_active":"2023-11-28T17:09:01.016673+0000","last_became_peered":"2023-11-28T17:09:01.016673+0000","last_unstale":"2023-11-28T17:09:02.511767+0000","last_undegraded":"2023-11-28T17:09:02.511767+0000","last_fullsized":"2023-11-28T17:09:02.511767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:36.438412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.274020+0000","last_change":"2023-11-28T17:09:00.262584+0000","last_active":"2023-11-28T17:09:01.274020+0000","last_peered":"2023-11-28T17:09:01.274020+0000","last_clean":"2023-11-28T17:09:01.274020+0000","last_became_active":"2023-11-28T17:09:00.262259+0000","last_became_peered":"2023-11-28T17:09:00.262259+0000","last_unstale":"2023-11-28T17:09:01.274020+0000","last_undegraded":"2023-11-28T17:09:01.274020+0000","last_fullsized":"2023-11-28T17:09:01.274020+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:31.377077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986947+0000","last_change":"2023-11-28T17:09:01.003188+0000","last_active":"2023-11-28T17:09:01.986947+0000","last_peered":"2023-11-28T17:09:01.986947+0000","last_clean":"2023-11-28T17:09:01.986947+0000","last_became_active":"2023-11-28T17:09:01.000643+0000","last_became_peered":"2023-11-28T17:09:01.000643+0000","last_unstale":"2023-11-28T17:09:01.986947+0000","last_undegraded":"2023-11-28T17:09:01.986947+0000","last_fullsized":"2023-11-28T17:09:01.986947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:49:27.242284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273771+0000","last_change":"2023-11-28T17:09:00.272398+0000","last_active":"2023-11-28T17:09:01.273771+0000","last_peered":"2023-11-28T17:09:01.273771+0000","last_clean":"2023-11-28T17:09:01.273771+0000","last_became_active":"2023-11-28T17:09:00.271617+0000","last_became_peered":"2023-11-28T17:09:00.271617+0000","last_unstale":"2023-11-28T17:09:01.273771+0000","last_undegraded":"2023-11-28T17:09:01.273771+0000","last_fullsized":"2023-11-28T17:09:01.273771+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:10.177210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.516054+0000","last_change":"2023-11-28T17:09:01.012920+0000","last_active":"2023-11-28T17:09:02.516054+0000","last_peered":"2023-11-28T17:09:02.516054+0000","last_clean":"2023-11-28T17:09:02.516054+0000","last_became_active":"2023-11-28T17:09:01.012651+0000","last_became_peered":"2023-11-28T17:09:01.012651+0000","last_unstale":"2023-11-28T17:09:02.516054+0000","last_undegraded":"2023-11-28T17:09:02.516054+0000","last_fullsized":"2023-11-28T17:09:02.516054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:53.375742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273900+0000","last_change":"2023-11-28T17:09:00.271952+0000","last_active":"2023-11-28T17:09:01.273900+0000","last_peered":"2023-11-28T17:09:01.273900+0000","last_clean":"2023-11-28T17:09:01.273900+0000","last_became_active":"2023-11-28T17:09:00.271641+0000","last_became_peered":"2023-11-28T17:09:00.271641+0000","last_unstale":"2023-11-28T17:09:01.273900+0000","last_undegraded":"2023-11-28T17:09:01.273900+0000","last_fullsized":"2023-11-28T17:09:01.273900+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:52.364620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513991+0000","last_change":"2023-11-28T17:09:01.014412+0000","last_active":"2023-11-28T17:09:02.513991+0000","last_peered":"2023-11-28T17:09:02.513991+0000","last_clean":"2023-11-28T17:09:02.513991+0000","last_became_active":"2023-11-28T17:09:01.014145+0000","last_became_peered":"2023-11-28T17:09:01.014145+0000","last_unstale":"2023-11-28T17:09:02.513991+0000","last_undegraded":"2023-11-28T17:09:02.513991+0000","last_fullsized":"2023-11-28T17:09:02.513991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:49:27.242284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273346+0000","last_change":"2023-11-28T17:09:00.264424+0000","last_active":"2023-11-28T17:09:01.273346+0000","last_peered":"2023-11-28T17:09:01.273346+0000","last_clean":"2023-11-28T17:09:01.273346+0000","last_became_active":"2023-11-28T17:09:00.263195+0000","last_became_peered":"2023-11-28T17:09:00.263195+0000","last_unstale":"2023-11-28T17:09:01.273346+0000","last_undegraded":"2023-11-28T17:09:01.273346+0000","last_fullsized":"2023-11-28T17:09:01.273346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:35.427291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513525+0000","last_change":"2023-11-28T17:09:01.019072+0000","last_active":"2023-11-28T17:09:02.513525+0000","last_peered":"2023-11-28T17:09:02.513525+0000","last_clean":"2023-11-28T17:09:02.513525+0000","last_became_active":"2023-11-28T17:09:01.018785+0000","last_became_peered":"2023-11-28T17:09:01.018785+0000","last_unstale":"2023-11-28T17:09:02.513525+0000","last_undegraded":"2023-11-28T17:09:02.513525+0000","last_fullsized":"2023-11-28T17:09:02.513525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:59.985963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.984805+0000","last_change":"2023-11-28T17:09:00.268057+0000","last_active":"2023-11-28T17:09:00.984805+0000","last_peered":"2023-11-28T17:09:00.984805+0000","last_clean":"2023-11-28T17:09:00.984805+0000","last_became_active":"2023-11-28T17:09:00.264298+0000","last_became_peered":"2023-11-28T17:09:00.264298+0000","last_unstale":"2023-11-28T17:09:00.984805+0000","last_undegraded":"2023-11-28T17:09:00.984805+0000","last_fullsized":"2023-11-28T17:09:00.984805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:35.427291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988219+0000","last_change":"2023-11-28T17:09:01.009062+0000","last_active":"2023-11-28T17:09:01.988219+0000","last_peered":"2023-11-28T17:09:01.988219+0000","last_clean":"2023-11-28T17:09:01.988219+0000","last_became_active":"2023-11-28T17:09:01.008759+0000","last_became_peered":"2023-11-28T17:09:01.008759+0000","last_unstale":"2023-11-28T17:09:01.988219+0000","last_undegraded":"2023-11-28T17:09:01.988219+0000","last_fullsized":"2023-11-28T17:09:01.988219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:09:50.534329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.986684+0000","last_change":"2023-11-28T17:08:51.392782+0000","last_active":"2023-11-28T17:09:00.986684+0000","last_peered":"2023-11-28T17:09:00.986684+0000","last_clean":"2023-11-28T17:09:00.986684+0000","last_became_active":"2023-11-28T17:08:51.390415+0000","last_became_peered":"2023-11-28T17:08:51.390415+0000","last_unstale":"2023-11-28T17:09:00.986684+0000","last_undegraded":"2023-11-28T17:09:00.986684+0000","last_fullsized":"2023-11-28T17:09:00.986684+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_clean_scrub_stamp":"2023-11-28T17:08:49.967682+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:03:32.638685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.985042+0000","last_change":"2023-11-28T17:09:00.267795+0000","last_active":"2023-11-28T17:09:00.985042+0000","last_peered":"2023-11-28T17:09:00.985042+0000","last_clean":"2023-11-28T17:09:00.985042+0000","last_became_active":"2023-11-28T17:09:00.264764+0000","last_became_peered":"2023-11-28T17:09:00.264764+0000","last_unstale":"2023-11-28T17:09:00.985042+0000","last_undegraded":"2023-11-28T17:09:00.985042+0000","last_fullsized":"2023-11-28T17:09:00.985042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:59.024219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988457+0000","last_change":"2023-11-28T17:09:01.004564+0000","last_active":"2023-11-28T17:09:01.988457+0000","last_peered":"2023-11-28T17:09:01.988457+0000","last_clean":"2023-11-28T17:09:01.988457+0000","last_became_active":"2023-11-28T17:09:01.004189+0000","last_became_peered":"2023-11-28T17:09:01.004189+0000","last_unstale":"2023-11-28T17:09:01.988457+0000","last_undegraded":"2023-11-28T17:09:01.988457+0000","last_fullsized":"2023-11-28T17:09:01.988457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:09.387207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.986785+0000","last_change":"2023-11-28T17:09:00.269480+0000","last_active":"2023-11-28T17:09:00.986785+0000","last_peered":"2023-11-28T17:09:00.986785+0000","last_clean":"2023-11-28T17:09:00.986785+0000","last_became_active":"2023-11-28T17:09:00.269032+0000","last_became_peered":"2023-11-28T17:09:00.269032+0000","last_unstale":"2023-11-28T17:09:00.986785+0000","last_undegraded":"2023-11-28T17:09:00.986785+0000","last_fullsized":"2023-11-28T17:09:00.986785+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:49.306608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987055+0000","last_change":"2023-11-28T17:09:01.007809+0000","last_active":"2023-11-28T17:09:01.987055+0000","last_peered":"2023-11-28T17:09:01.987055+0000","last_clean":"2023-11-28T17:09:01.987055+0000","last_became_active":"2023-11-28T17:09:01.007508+0000","last_became_peered":"2023-11-28T17:09:01.007508+0000","last_unstale":"2023-11-28T17:09:01.987055+0000","last_undegraded":"2023-11-28T17:09:01.987055+0000","last_fullsized":"2023-11-28T17:09:01.987055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:20.109182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.001078+0000","last_change":"2023-11-28T17:09:00.271320+0000","last_active":"2023-11-28T17:09:01.001078+0000","last_peered":"2023-11-28T17:09:01.001078+0000","last_clean":"2023-11-28T17:09:01.001078+0000","last_became_active":"2023-11-28T17:09:00.271019+0000","last_became_peered":"2023-11-28T17:09:00.271019+0000","last_unstale":"2023-11-28T17:09:01.001078+0000","last_undegraded":"2023-11-28T17:09:01.001078+0000","last_fullsized":"2023-11-28T17:09:01.001078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:41:19.939145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222456+0000","last_change":"2023-11-28T17:09:01.009687+0000","last_active":"2023-11-28T17:09:02.222456+0000","last_peered":"2023-11-28T17:09:02.222456+0000","last_clean":"2023-11-28T17:09:02.222456+0000","last_became_active":"2023-11-28T17:09:01.009329+0000","last_became_peered":"2023-11-28T17:09:01.009329+0000","last_unstale":"2023-11-28T17:09:02.222456+0000","last_undegraded":"2023-11-28T17:09:02.222456+0000","last_fullsized":"2023-11-28T17:09:02.222456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:50.317730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.991649+0000","last_change":"2023-11-28T17:09:00.269333+0000","last_active":"2023-11-28T17:09:00.991649+0000","last_peered":"2023-11-28T17:09:00.991649+0000","last_clean":"2023-11-28T17:09:00.991649+0000","last_became_active":"2023-11-28T17:09:00.267481+0000","last_became_peered":"2023-11-28T17:09:00.267481+0000","last_unstale":"2023-11-28T17:09:00.991649+0000","last_undegraded":"2023-11-28T17:09:00.991649+0000","last_fullsized":"2023-11-28T17:09:00.991649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:21.776220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512505+0000","last_change":"2023-11-28T17:09:01.012458+0000","last_active":"2023-11-28T17:09:02.512505+0000","last_peered":"2023-11-28T17:09:02.512505+0000","last_clean":"2023-11-28T17:09:02.512505+0000","last_became_active":"2023-11-28T17:09:01.011811+0000","last_became_peered":"2023-11-28T17:09:01.011811+0000","last_unstale":"2023-11-28T17:09:02.512505+0000","last_undegraded":"2023-11-28T17:09:02.512505+0000","last_fullsized":"2023-11-28T17:09:02.512505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:24.354633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.999956+0000","last_change":"2023-11-28T17:09:00.263517+0000","last_active":"2023-11-28T17:09:00.999956+0000","last_peered":"2023-11-28T17:09:00.999956+0000","last_clean":"2023-11-28T17:09:00.999956+0000","last_became_active":"2023-11-28T17:09:00.263243+0000","last_became_peered":"2023-11-28T17:09:00.263243+0000","last_unstale":"2023-11-28T17:09:00.999956+0000","last_undegraded":"2023-11-28T17:09:00.999956+0000","last_fullsized":"2023-11-28T17:09:00.999956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:10.177210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515382+0000","last_change":"2023-11-28T17:09:00.997629+0000","last_active":"2023-11-28T17:09:02.515382+0000","last_peered":"2023-11-28T17:09:02.515382+0000","last_clean":"2023-11-28T17:09:02.515382+0000","last_became_active":"2023-11-28T17:09:00.997360+0000","last_became_peered":"2023-11-28T17:09:00.997360+0000","last_unstale":"2023-11-28T17:09:02.515382+0000","last_undegraded":"2023-11-28T17:09:02.515382+0000","last_fullsized":"2023-11-28T17:09:02.515382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:32.388198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.988631+0000","last_change":"2023-11-28T17:09:00.268166+0000","last_active":"2023-11-28T17:09:00.988631+0000","last_peered":"2023-11-28T17:09:00.988631+0000","last_clean":"2023-11-28T17:09:00.988631+0000","last_became_active":"2023-11-28T17:09:00.267609+0000","last_became_peered":"2023-11-28T17:09:00.267609+0000","last_unstale":"2023-11-28T17:09:00.988631+0000","last_undegraded":"2023-11-28T17:09:00.988631+0000","last_fullsized":"2023-11-28T17:09:00.988631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:23:26.227274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.989182+0000","last_change":"2023-11-28T17:09:01.017987+0000","last_active":"2023-11-28T17:09:01.989182+0000","last_peered":"2023-11-28T17:09:01.989182+0000","last_clean":"2023-11-28T17:09:01.989182+0000","last_became_active":"2023-11-28T17:09:01.017491+0000","last_became_peered":"2023-11-28T17:09:01.017491+0000","last_unstale":"2023-11-28T17:09:01.989182+0000","last_undegraded":"2023-11-28T17:09:01.989182+0000","last_fullsized":"2023-11-28T17:09:01.989182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:09.387207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.996053+0000","last_change":"2023-11-28T17:09:00.267582+0000","last_active":"2023-11-28T17:09:00.996053+0000","last_peered":"2023-11-28T17:09:00.996053+0000","last_clean":"2023-11-28T17:09:00.996053+0000","last_became_active":"2023-11-28T17:09:00.267245+0000","last_became_peered":"2023-11-28T17:09:00.267245+0000","last_unstale":"2023-11-28T17:09:00.996053+0000","last_undegraded":"2023-11-28T17:09:00.996053+0000","last_fullsized":"2023-11-28T17:09:00.996053+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:59.024219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.990398+0000","last_change":"2023-11-28T17:09:00.997521+0000","last_active":"2023-11-28T17:09:01.990398+0000","last_peered":"2023-11-28T17:09:01.990398+0000","last_clean":"2023-11-28T17:09:01.990398+0000","last_became_active":"2023-11-28T17:09:00.997041+0000","last_became_peered":"2023-11-28T17:09:00.997041+0000","last_unstale":"2023-11-28T17:09:01.990398+0000","last_undegraded":"2023-11-28T17:09:01.990398+0000","last_fullsized":"2023-11-28T17:09:01.990398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:07.593383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276430+0000","last_change":"2023-11-28T17:09:00.264837+0000","last_active":"2023-11-28T17:09:01.276430+0000","last_peered":"2023-11-28T17:09:01.276430+0000","last_clean":"2023-11-28T17:09:01.276430+0000","last_became_active":"2023-11-28T17:09:00.264570+0000","last_became_peered":"2023-11-28T17:09:00.264570+0000","last_unstale":"2023-11-28T17:09:01.276430+0000","last_undegraded":"2023-11-28T17:09:01.276430+0000","last_fullsized":"2023-11-28T17:09:01.276430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:23.343512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.990475+0000","last_change":"2023-11-28T17:09:01.006414+0000","last_active":"2023-11-28T17:09:01.990475+0000","last_peered":"2023-11-28T17:09:01.990475+0000","last_clean":"2023-11-28T17:09:01.990475+0000","last_became_active":"2023-11-28T17:09:01.005570+0000","last_became_peered":"2023-11-28T17:09:01.005570+0000","last_unstale":"2023-11-28T17:09:01.990475+0000","last_undegraded":"2023-11-28T17:09:01.990475+0000","last_fullsized":"2023-11-28T17:09:01.990475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:46:24.200692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.998379+0000","last_change":"2023-11-28T17:09:00.257189+0000","last_active":"2023-11-28T17:09:00.998379+0000","last_peered":"2023-11-28T17:09:00.998379+0000","last_clean":"2023-11-28T17:09:00.998379+0000","last_became_active":"2023-11-28T17:09:00.256862+0000","last_became_peered":"2023-11-28T17:09:00.256862+0000","last_unstale":"2023-11-28T17:09:00.998379+0000","last_undegraded":"2023-11-28T17:09:00.998379+0000","last_fullsized":"2023-11-28T17:09:00.998379+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:24.313840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987313+0000","last_change":"2023-11-28T17:09:01.006232+0000","last_active":"2023-11-28T17:09:01.987313+0000","last_peered":"2023-11-28T17:09:01.987313+0000","last_clean":"2023-11-28T17:09:01.987313+0000","last_became_active":"2023-11-28T17:09:01.005930+0000","last_became_peered":"2023-11-28T17:09:01.005930+0000","last_unstale":"2023-11-28T17:09:01.987313+0000","last_undegraded":"2023-11-28T17:09:01.987313+0000","last_fullsized":"2023-11-28T17:09:01.987313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:06:27.104365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.991766+0000","last_change":"2023-11-28T17:09:00.266555+0000","last_active":"2023-11-28T17:09:00.991766+0000","last_peered":"2023-11-28T17:09:00.991766+0000","last_clean":"2023-11-28T17:09:00.991766+0000","last_became_active":"2023-11-28T17:09:00.266279+0000","last_became_peered":"2023-11-28T17:09:00.266279+0000","last_unstale":"2023-11-28T17:09:00.991766+0000","last_undegraded":"2023-11-28T17:09:00.991766+0000","last_fullsized":"2023-11-28T17:09:00.991766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:19.098061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988255+0000","last_change":"2023-11-28T17:09:01.007494+0000","last_active":"2023-11-28T17:09:01.988255+0000","last_peered":"2023-11-28T17:09:01.988255+0000","last_clean":"2023-11-28T17:09:01.988255+0000","last_became_active":"2023-11-28T17:09:01.007121+0000","last_became_peered":"2023-11-28T17:09:01.007121+0000","last_unstale":"2023-11-28T17:09:01.988255+0000","last_undegraded":"2023-11-28T17:09:01.988255+0000","last_fullsized":"2023-11-28T17:09:01.988255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:11.188332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.984942+0000","last_change":"2023-11-28T17:09:00.267476+0000","last_active":"2023-11-28T17:09:00.984942+0000","last_peered":"2023-11-28T17:09:00.984942+0000","last_clean":"2023-11-28T17:09:00.984942+0000","last_became_active":"2023-11-28T17:09:00.267138+0000","last_became_peered":"2023-11-28T17:09:00.267138+0000","last_unstale":"2023-11-28T17:09:00.984942+0000","last_undegraded":"2023-11-28T17:09:00.984942+0000","last_fullsized":"2023-11-28T17:09:00.984942+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:11.303292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512236+0000","last_change":"2023-11-28T17:09:01.008773+0000","last_active":"2023-11-28T17:09:02.512236+0000","last_peered":"2023-11-28T17:09:02.512236+0000","last_clean":"2023-11-28T17:09:02.512236+0000","last_became_active":"2023-11-28T17:09:01.008241+0000","last_became_peered":"2023-11-28T17:09:01.008241+0000","last_unstale":"2023-11-28T17:09:02.512236+0000","last_undegraded":"2023-11-28T17:09:02.512236+0000","last_fullsized":"2023-11-28T17:09:02.512236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:12.314413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276631+0000","last_change":"2023-11-28T17:09:00.267671+0000","last_active":"2023-11-28T17:09:01.276631+0000","last_peered":"2023-11-28T17:09:01.276631+0000","last_clean":"2023-11-28T17:09:01.276631+0000","last_became_active":"2023-11-28T17:09:00.264445+0000","last_became_peered":"2023-11-28T17:09:00.264445+0000","last_unstale":"2023-11-28T17:09:01.276631+0000","last_undegraded":"2023-11-28T17:09:01.276631+0000","last_fullsized":"2023-11-28T17:09:01.276631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:52.364620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987244+0000","last_change":"2023-11-28T17:09:01.003981+0000","last_active":"2023-11-28T17:09:01.987244+0000","last_peered":"2023-11-28T17:09:01.987244+0000","last_clean":"2023-11-28T17:09:01.987244+0000","last_became_active":"2023-11-28T17:09:01.003508+0000","last_became_peered":"2023-11-28T17:09:01.003508+0000","last_unstale":"2023-11-28T17:09:01.987244+0000","last_undegraded":"2023-11-28T17:09:01.987244+0000","last_fullsized":"2023-11-28T17:09:01.987244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:48:31.212857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988121+0000","last_change":"2023-11-28T17:09:01.012859+0000","last_active":"2023-11-28T17:09:01.988121+0000","last_peered":"2023-11-28T17:09:01.988121+0000","last_clean":"2023-11-28T17:09:01.988121+0000","last_became_active":"2023-11-28T17:09:01.010930+0000","last_became_peered":"2023-11-28T17:09:01.010930+0000","last_unstale":"2023-11-28T17:09:01.988121+0000","last_undegraded":"2023-11-28T17:09:01.988121+0000","last_fullsized":"2023-11-28T17:09:01.988121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:46:01.399129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.268661+0000","last_change":"2023-11-28T17:09:00.259200+0000","last_active":"2023-11-28T17:09:01.268661+0000","last_peered":"2023-11-28T17:09:01.268661+0000","last_clean":"2023-11-28T17:09:01.268661+0000","last_became_active":"2023-11-28T17:09:00.258886+0000","last_became_peered":"2023-11-28T17:09:00.258886+0000","last_unstale":"2023-11-28T17:09:01.268661+0000","last_undegraded":"2023-11-28T17:09:01.268661+0000","last_fullsized":"2023-11-28T17:09:01.268661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:43.640066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.514743+0000","last_change":"2023-11-28T17:09:01.015028+0000","last_active":"2023-11-28T17:09:02.514743+0000","last_peered":"2023-11-28T17:09:02.514743+0000","last_clean":"2023-11-28T17:09:02.514743+0000","last_became_active":"2023-11-28T17:09:01.014670+0000","last_became_peered":"2023-11-28T17:09:01.014670+0000","last_unstale":"2023-11-28T17:09:02.514743+0000","last_undegraded":"2023-11-28T17:09:02.514743+0000","last_fullsized":"2023-11-28T17:09:02.514743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:31.569896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.268602+0000","last_change":"2023-11-28T17:09:00.263220+0000","last_active":"2023-11-28T17:09:01.268602+0000","last_peered":"2023-11-28T17:09:01.268602+0000","last_clean":"2023-11-28T17:09:01.268602+0000","last_became_active":"2023-11-28T17:09:00.262930+0000","last_became_peered":"2023-11-28T17:09:00.262930+0000","last_unstale":"2023-11-28T17:09:01.268602+0000","last_undegraded":"2023-11-28T17:09:01.268602+0000","last_fullsized":"2023-11-28T17:09:01.268602+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:06.582262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223461+0000","last_change":"2023-11-28T17:09:01.014772+0000","last_active":"2023-11-28T17:09:02.223461+0000","last_peered":"2023-11-28T17:09:02.223461+0000","last_clean":"2023-11-28T17:09:02.223461+0000","last_became_active":"2023-11-28T17:09:01.013249+0000","last_became_peered":"2023-11-28T17:09:01.013249+0000","last_unstale":"2023-11-28T17:09:02.223461+0000","last_undegraded":"2023-11-28T17:09:02.223461+0000","last_fullsized":"2023-11-28T17:09:02.223461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:34.895213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273335+0000","last_change":"2023-11-28T17:09:00.270822+0000","last_active":"2023-11-28T17:09:01.273335+0000","last_peered":"2023-11-28T17:09:01.273335+0000","last_clean":"2023-11-28T17:09:01.273335+0000","last_became_active":"2023-11-28T17:09:00.268995+0000","last_became_peered":"2023-11-28T17:09:00.268995+0000","last_unstale":"2023-11-28T17:09:01.273335+0000","last_undegraded":"2023-11-28T17:09:01.273335+0000","last_fullsized":"2023-11-28T17:09:01.273335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:24.313840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223456+0000","last_change":"2023-11-28T17:09:01.012994+0000","last_active":"2023-11-28T17:09:02.223456+0000","last_peered":"2023-11-28T17:09:02.223456+0000","last_clean":"2023-11-28T17:09:02.223456+0000","last_became_active":"2023-11-28T17:09:01.012496+0000","last_became_peered":"2023-11-28T17:09:01.012496+0000","last_unstale":"2023-11-28T17:09:02.223456+0000","last_undegraded":"2023-11-28T17:09:02.223456+0000","last_fullsized":"2023-11-28T17:09:02.223456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:59.985963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276064+0000","last_change":"2023-11-28T17:09:00.265200+0000","last_active":"2023-11-28T17:09:01.276064+0000","last_peered":"2023-11-28T17:09:01.276064+0000","last_clean":"2023-11-28T17:09:01.276064+0000","last_became_active":"2023-11-28T17:09:00.264913+0000","last_became_peered":"2023-11-28T17:09:00.264913+0000","last_unstale":"2023-11-28T17:09:01.276064+0000","last_undegraded":"2023-11-28T17:09:01.276064+0000","last_fullsized":"2023-11-28T17:09:01.276064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:24.313840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.446}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.875}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2190000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3069999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5740000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.97}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.696}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":806912,"data_stored":661370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.774}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.595}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2869999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2320000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.032}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":73194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4390000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1679999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.075}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.49}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.222}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4570000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.484}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7390000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8660000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5940000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.78}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.665}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.788}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8839999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8600000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":70966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.425}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T17:09:06.247 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T17:09:06.247 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-28T17:09:06.248 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-28T17:09:06.248 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-28T17:09:06.248 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-28T17:09:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.495+0000 7f1582000640 1 Processor -- start 2023-11-28T17:09:06.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.495+0000 7f1582000640 1 -- start start 2023-11-28T17:09:06.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.496+0000 7f1582000640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f157c13cee0 0x7f157c06c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:06.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.496+0000 7f1582000640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f157c140200 0x7f157c06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:06.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.497+0000 7f1582000640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f157c15a490 0x7f157c15a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:06.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.497+0000 7f1582000640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f157c0aea20 con 0x7f157c13cee0 2023-11-28T17:09:06.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.497+0000 7f1582000640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f157c13d2c0 con 0x7f157c15a490 2023-11-28T17:09:06.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.497+0000 7f1582000640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157c15b3a0 con 0x7f157c140200 2023-11-28T17:09:06.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.497+0000 7f157b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f157c13cee0 0x7f157c06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:06.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.497+0000 7f157b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f157c13cee0 0x7f157c06c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43686/0 (socket says 172.21.15.116:43686) 2023-11-28T17:09:06.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.497+0000 7f157b7fe640 1 -- 172.21.15.116:0/3877714163 learned_addr learned my addr 172.21.15.116:0/3877714163 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:06.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.497+0000 7f157b7fe640 1 -- 172.21.15.116:0/3877714163 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f157c140200 msgr2=0x7f157c06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:06.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.497+0000 7f1572ffd640 1 --2- 172.21.15.116:0/3877714163 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f157c140200 0x7f157c06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:06.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.497+0000 7f157b7fe640 1 --2- 172.21.15.116:0/3877714163 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f157c140200 0x7f157c06c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.497+0000 7f157b7fe640 1 -- 172.21.15.116:0/3877714163 >> v2:172.21.15.156:3300/0 conn(0x7f157c15a490 msgr2=0x7f157c15a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:06.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.497+0000 7f157b7fe640 1 --2- 172.21.15.116:0/3877714163 >> v2:172.21.15.156:3300/0 conn(0x7f157c15a490 0x7f157c15a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.497+0000 7f157b7fe640 1 -- 172.21.15.116:0/3877714163 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c15ae60 con 0x7f157c13cee0 2023-11-28T17:09:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.498+0000 7f157b7fe640 1 --2- 172.21.15.116:0/3877714163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c13cee0 0x7f157c06c250 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f157c0b2230 tx=0x7f156c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0e5efc14c68240 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.499+0000 7f157affd640 1 -- 172.21.15.116:0/3877714163 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c0317e0 con 0x7f157c13cee0 2023-11-28T17:09:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.499+0000 7f157affd640 1 -- 172.21.15.116:0/3877714163 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156c031940 con 0x7f157c13cee0 2023-11-28T17:09:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.499+0000 7f157affd640 1 -- 172.21.15.116:0/3877714163 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c031cd0 con 0x7f157c13cee0 2023-11-28T17:09:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.499+0000 7f1582000640 1 -- 172.21.15.116:0/3877714163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c13cee0 msgr2=0x7f157c06c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.499+0000 7f1582000640 1 --2- 172.21.15.116:0/3877714163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c13cee0 0x7f157c06c250 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f157c0b2230 tx=0x7f156c02b880 comp rx=0 tx=0).stop 2023-11-28T17:09:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.500+0000 7f1582000640 1 -- 172.21.15.116:0/3877714163 shutdown_connections 2023-11-28T17:09:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.500+0000 7f1582000640 1 --2- 172.21.15.116:0/3877714163 >> v2:172.21.15.156:3300/0 conn(0x7f157c15a490 0x7f157c15a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.500+0000 7f1582000640 1 --2- 172.21.15.116:0/3877714163 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f157c140200 0x7f157c06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.500+0000 7f1582000640 1 --2- 172.21.15.116:0/3877714163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c13cee0 0x7f157c06c250 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.500+0000 7f1582000640 1 -- 172.21.15.116:0/3877714163 >> 172.21.15.116:0/3877714163 conn(0x7f157c06a4c0 msgr2=0x7f157c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.500+0000 7f1582000640 1 -- 172.21.15.116:0/3877714163 shutdown_connections 2023-11-28T17:09:06.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.500+0000 7f1582000640 1 -- 172.21.15.116:0/3877714163 wait complete. 2023-11-28T17:09:06.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.502+0000 7f1582000640 1 Processor -- start 2023-11-28T17:09:06.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.502+0000 7f1582000640 1 -- start start 2023-11-28T17:09:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.503+0000 7f1582000640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f157c13cee0 0x7f157c1644e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.503+0000 7f1582000640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f157c140200 0x7f157c13e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.503+0000 7f1582000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c15a490 0x7f157c158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.503+0000 7f1582000640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157c164a20 con 0x7f157c15a490 2023-11-28T17:09:06.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.503+0000 7f1582000640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157c13d400 con 0x7f157c13cee0 2023-11-28T17:09:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.503+0000 7f1582000640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157c13dca0 con 0x7f157c140200 2023-11-28T17:09:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.503+0000 7f157bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c15a490 0x7f157c158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.504+0000 7f157bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c15a490 0x7f157c158020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43692/0 (socket says 172.21.15.116:43692) 2023-11-28T17:09:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.504+0000 7f157bfff640 1 -- 172.21.15.116:0/3202837233 learned_addr learned my addr 172.21.15.116:0/3202837233 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.504+0000 7f157b7fe640 1 --2- 172.21.15.116:0/3202837233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f157c13cee0 0x7f157c1644e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.504+0000 7f1572ffd640 1 --2- 172.21.15.116:0/3202837233 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f157c140200 0x7f157c13e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:06.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.504+0000 7f157bfff640 1 -- 172.21.15.116:0/3202837233 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f157c140200 msgr2=0x7f157c13e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.504+0000 7f157bfff640 1 --2- 172.21.15.116:0/3202837233 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f157c140200 0x7f157c13e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.504+0000 7f157bfff640 1 -- 172.21.15.116:0/3202837233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f157c13cee0 msgr2=0x7f157c1644e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.504+0000 7f157bfff640 1 --2- 172.21.15.116:0/3202837233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f157c13cee0 0x7f157c1644e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.504+0000 7f157bfff640 1 -- 172.21.15.116:0/3202837233 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156c0094d0 con 0x7f157c15a490 2023-11-28T17:09:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.504+0000 7f157bfff640 1 --2- 172.21.15.116:0/3202837233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c15a490 0x7f157c158020 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f157400b450 tx=0x7f157400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.505+0000 7f1578ff9640 1 -- 172.21.15.116:0/3202837233 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574011830 con 0x7f157c15a490 2023-11-28T17:09:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.505+0000 7f1578ff9640 1 -- 172.21.15.116:0/3202837233 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1574011990 con 0x7f157c15a490 2023-11-28T17:09:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.505+0000 7f1582000640 1 -- 172.21.15.116:0/3202837233 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157c13f1f0 con 0x7f157c15a490 2023-11-28T17:09:06.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.505+0000 7f1578ff9640 1 -- 172.21.15.116:0/3202837233 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574011d20 con 0x7f157c15a490 2023-11-28T17:09:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.505+0000 7f1582000640 1 -- 172.21.15.116:0/3202837233 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157c13d930 con 0x7f157c15a490 2023-11-28T17:09:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.506+0000 7f1582000640 1 -- 172.21.15.116:0/3202837233 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f157c0693d0 con 0x7f157c15a490 2023-11-28T17:09:06.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.507+0000 7f1578ff9640 1 -- 172.21.15.116:0/3202837233 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f157401a400 con 0x7f157c15a490 2023-11-28T17:09:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.508+0000 7f1578ff9640 1 --2- 172.21.15.116:0/3202837233 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f155c06b2a0 0x7f155c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.508+0000 7f1578ff9640 1 -- 172.21.15.116:0/3202837233 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1574089d30 con 0x7f157c15a490 2023-11-28T17:09:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.508+0000 7f157b7fe640 1 --2- 172.21.15.116:0/3202837233 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f155c06b2a0 0x7f155c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:06.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.509+0000 7f157b7fe640 1 --2- 172.21.15.116:0/3202837233 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f155c06b2a0 0x7f155c06d760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f156c036000 tx=0x7f156c03d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:06.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.512+0000 7f1578ff9640 1 -- 172.21.15.116:0/3202837233 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1574054c30 con 0x7f157c15a490 2023-11-28T17:09:06.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.752+0000 7f1582000640 1 -- 172.21.15.116:0/3202837233 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f157c13dde0 con 0x7f157c15a490 2023-11-28T17:09:06.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.758+0000 7f1578ff9640 1 -- 172.21.15.116:0/3202837233 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171577 (secure 0 0 0) 0x7f1574015020 con 0x7f157c15a490 2023-11-28T17:09:06.762 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:06.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.762+0000 7f1582000640 1 -- 172.21.15.116:0/3202837233 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f155c06b2a0 msgr2=0x7f155c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.762+0000 7f1582000640 1 --2- 172.21.15.116:0/3202837233 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f155c06b2a0 0x7f155c06d760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f156c036000 tx=0x7f156c03d040 comp rx=0 tx=0).stop 2023-11-28T17:09:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.762+0000 7f1582000640 1 -- 172.21.15.116:0/3202837233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c15a490 msgr2=0x7f157c158020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.762+0000 7f1582000640 1 --2- 172.21.15.116:0/3202837233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c15a490 0x7f157c158020 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f157400b450 tx=0x7f157400b910 comp rx=0 tx=0).stop 2023-11-28T17:09:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.762+0000 7f1582000640 1 -- 172.21.15.116:0/3202837233 shutdown_connections 2023-11-28T17:09:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.762+0000 7f1582000640 1 --2- 172.21.15.116:0/3202837233 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f155c06b2a0 0x7f155c06d760 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.762+0000 7f1582000640 1 --2- 172.21.15.116:0/3202837233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f157c15a490 0x7f157c158020 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.762+0000 7f1582000640 1 --2- 172.21.15.116:0/3202837233 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f157c140200 0x7f157c13e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.762+0000 7f1582000640 1 --2- 172.21.15.116:0/3202837233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f157c13cee0 0x7f157c1644e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.762+0000 7f1582000640 1 -- 172.21.15.116:0/3202837233 >> 172.21.15.116:0/3202837233 conn(0x7f157c06a4c0 msgr2=0x7f157c0af1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.762+0000 7f1582000640 1 -- 172.21.15.116:0/3202837233 shutdown_connections 2023-11-28T17:09:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.762+0000 7f1582000640 1 -- 172.21.15.116:0/3202837233 wait complete. 2023-11-28T17:09:06.783 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"active_gid":4098,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1077400178},{"type":"v1","addr":"172.21.15.116:6833","nonce":1077400178}]},"active_addr":"172.21.15.116:6833/1077400178","active_change":"2023-11-28T17:08:41.017630+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4108,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":4017451007}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":1464721272}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":2144983900}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":1058904188}]}]} 2023-11-28T17:09:06.786 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-28T17:09:06.786 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-28T17:09:06.787 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:06.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.992+0000 7fbf498cf640 1 Processor -- start 2023-11-28T17:09:06.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.993+0000 7fbf498cf640 1 -- start start 2023-11-28T17:09:06.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.994+0000 7fbf498cf640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf4413cee0 0x7fbf4406c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:06.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.995+0000 7fbf498cf640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf44140200 0x7fbf4406c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:06.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.995+0000 7fbf498cf640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf440aea20 con 0x7fbf4415a490 2023-11-28T17:09:06.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.995+0000 7fbf498cf640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4413d2c0 con 0x7fbf4413cee0 2023-11-28T17:09:06.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.995+0000 7fbf498cf640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4415b3a0 con 0x7fbf44140200 2023-11-28T17:09:06.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.995+0000 7fbf42ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf4413cee0 0x7fbf4406c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:06.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.995+0000 7fbf42ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf4413cee0 0x7fbf4406c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43704/0 (socket says 172.21.15.116:43704) 2023-11-28T17:09:06.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.995+0000 7fbf42ffd640 1 -- 172.21.15.116:0/1611223129 learned_addr learned my addr 172.21.15.116:0/1611223129 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:06.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.995+0000 7fbf42ffd640 1 -- 172.21.15.116:0/1611223129 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf44140200 msgr2=0x7fbf4406c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:06.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.995+0000 7fbf3a7fc640 1 --2- 172.21.15.116:0/1611223129 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf44140200 0x7fbf4406c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:06.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.995+0000 7fbf42ffd640 1 --2- 172.21.15.116:0/1611223129 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf44140200 0x7fbf4406c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.995+0000 7fbf42ffd640 1 -- 172.21.15.116:0/1611223129 >> v1:172.21.15.156:6789/0 conn(0x7fbf4415a490 legacy=0x7fbf44071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:07.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.995+0000 7fbf42ffd640 1 -- 172.21.15.116:0/1611223129 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4415b0c0 con 0x7fbf4413cee0 2023-11-28T17:09:07.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.996+0000 7fbf42ffd640 1 --2- 172.21.15.116:0/1611223129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4413cee0 0x7fbf4406c250 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009870 tx=0x7fbf3402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837cb68dd3797fd1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:07.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.997+0000 7fbf427fc640 1 -- 172.21.15.116:0/1611223129 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf340317c0 con 0x7fbf4413cee0 2023-11-28T17:09:07.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.997+0000 7fbf427fc640 1 -- 172.21.15.116:0/1611223129 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf34031920 con 0x7fbf4413cee0 2023-11-28T17:09:07.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.997+0000 7fbf427fc640 1 -- 172.21.15.116:0/1611223129 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34031cb0 con 0x7fbf4413cee0 2023-11-28T17:09:07.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.997+0000 7fbf498cf640 1 -- 172.21.15.116:0/1611223129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4413cee0 msgr2=0x7fbf4406c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.997+0000 7fbf498cf640 1 --2- 172.21.15.116:0/1611223129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4413cee0 0x7fbf4406c250 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009870 tx=0x7fbf3402b860 comp rx=0 tx=0).stop 2023-11-28T17:09:07.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.998+0000 7fbf498cf640 1 -- 172.21.15.116:0/1611223129 shutdown_connections 2023-11-28T17:09:07.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.998+0000 7fbf498cf640 1 --2- 172.21.15.116:0/1611223129 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf44140200 0x7fbf4406c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.998+0000 7fbf498cf640 1 --2- 172.21.15.116:0/1611223129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4413cee0 0x7fbf4406c250 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.998+0000 7fbf498cf640 1 -- 172.21.15.116:0/1611223129 >> 172.21.15.116:0/1611223129 conn(0x7fbf4406a4c0 msgr2=0x7fbf4406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:07.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.998+0000 7fbf498cf640 1 -- 172.21.15.116:0/1611223129 shutdown_connections 2023-11-28T17:09:07.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:06.998+0000 7fbf498cf640 1 -- 172.21.15.116:0/1611223129 wait complete. 2023-11-28T17:09:07.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.000+0000 7fbf498cf640 1 Processor -- start 2023-11-28T17:09:07.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.000+0000 7fbf498cf640 1 -- start start 2023-11-28T17:09:07.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.001+0000 7fbf498cf640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf4413cee0 0x7fbf4413bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.001+0000 7fbf498cf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf44140200 0x7fbf441502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.001+0000 7fbf498cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4415a490 0x7fbf4413ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.001+0000 7fbf498cf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4413c130 con 0x7fbf4415a490 2023-11-28T17:09:07.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.001+0000 7fbf498cf640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440703f0 con 0x7fbf44140200 2023-11-28T17:09:07.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.001+0000 7fbf498cf640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf441f3fc0 con 0x7fbf4413cee0 2023-11-28T17:09:07.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.001+0000 7fbf437fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4415a490 0x7fbf4413ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.001+0000 7fbf437fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4415a490 0x7fbf4413ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43712/0 (socket says 172.21.15.116:43712) 2023-11-28T17:09:07.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.001+0000 7fbf437fe640 1 -- 172.21.15.116:0/535407664 learned_addr learned my addr 172.21.15.116:0/535407664 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:07.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.002+0000 7fbf42ffd640 1 --2- 172.21.15.116:0/535407664 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf4413cee0 0x7fbf4413bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.002+0000 7fbf3a7fc640 1 --2- 172.21.15.116:0/535407664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf44140200 0x7fbf441502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.002+0000 7fbf437fe640 1 -- 172.21.15.116:0/535407664 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf4413cee0 msgr2=0x7fbf4413bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.002+0000 7fbf437fe640 1 --2- 172.21.15.116:0/535407664 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf4413cee0 0x7fbf4413bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.002+0000 7fbf437fe640 1 -- 172.21.15.116:0/535407664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf44140200 msgr2=0x7fbf441502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.002+0000 7fbf437fe640 1 --2- 172.21.15.116:0/535407664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf44140200 0x7fbf441502e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.002+0000 7fbf437fe640 1 -- 172.21.15.116:0/535407664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf340094d0 con 0x7fbf4415a490 2023-11-28T17:09:07.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.002+0000 7fbf437fe640 1 --2- 172.21.15.116:0/535407664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4415a490 0x7fbf4413ea50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00b9a0 tx=0x7fbf3c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:07.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.002+0000 7fbf40ff9640 1 -- 172.21.15.116:0/535407664 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c00cba0 con 0x7fbf4415a490 2023-11-28T17:09:07.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.002+0000 7fbf40ff9640 1 -- 172.21.15.116:0/535407664 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3c00cd00 con 0x7fbf4415a490 2023-11-28T17:09:07.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.002+0000 7fbf40ff9640 1 -- 172.21.15.116:0/535407664 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c014680 con 0x7fbf4415a490 2023-11-28T17:09:07.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.002+0000 7fbf498cf640 1 -- 172.21.15.116:0/535407664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf44150820 con 0x7fbf4415a490 2023-11-28T17:09:07.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.003+0000 7fbf498cf640 1 -- 172.21.15.116:0/535407664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf441f33b0 con 0x7fbf4415a490 2023-11-28T17:09:07.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.004+0000 7fbf498cf640 1 -- 172.21.15.116:0/535407664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf440adb80 con 0x7fbf4415a490 2023-11-28T17:09:07.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.005+0000 7fbf40ff9640 1 -- 172.21.15.116:0/535407664 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fbf3c01d3b0 con 0x7fbf4415a490 2023-11-28T17:09:07.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.006+0000 7fbf40ff9640 1 --2- 172.21.15.116:0/535407664 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf2806b2a0 0x7fbf2806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.006+0000 7fbf40ff9640 1 -- 172.21.15.116:0/535407664 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf3c0899f0 con 0x7fbf4415a490 2023-11-28T17:09:07.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.006+0000 7fbf42ffd640 1 --2- 172.21.15.116:0/535407664 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf2806b2a0 0x7fbf2806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.007+0000 7fbf42ffd640 1 --2- 172.21.15.116:0/535407664 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf2806b2a0 0x7fbf2806d760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf34036000 tx=0x7fbf3403d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:07.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.010+0000 7fbf40ff9640 1 -- 172.21.15.116:0/535407664 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf3c054850 con 0x7fbf4415a490 2023-11-28T17:09:07.203 INFO:tasks.ceph.mds.b.smithi156.stderr:2023-11-28T17:09:07.201+0000 7f29f4a64640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T17:09:07.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.202+0000 7fbf498cf640 1 -- 172.21.15.116:0/535407664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf440693d0 con 0x7fbf4415a490 2023-11-28T17:09:07.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.206+0000 7fbf40ff9640 1 -- 172.21.15.116:0/535407664 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16521 (secure 0 0 0) 0x7fbf3c057e70 con 0x7fbf4415a490 2023-11-28T17:09:07.208 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:07.209 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:01.970957+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:07.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.209+0000 7fbf498cf640 1 -- 172.21.15.116:0/535407664 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf2806b2a0 msgr2=0x7fbf2806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.209+0000 7fbf498cf640 1 --2- 172.21.15.116:0/535407664 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf2806b2a0 0x7fbf2806d760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf34036000 tx=0x7fbf3403d040 comp rx=0 tx=0).stop 2023-11-28T17:09:07.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.209+0000 7fbf498cf640 1 -- 172.21.15.116:0/535407664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4415a490 msgr2=0x7fbf4413ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.209+0000 7fbf498cf640 1 --2- 172.21.15.116:0/535407664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4415a490 0x7fbf4413ea50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00b9a0 tx=0x7fbf3c00be60 comp rx=0 tx=0).stop 2023-11-28T17:09:07.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.209+0000 7fbf498cf640 1 -- 172.21.15.116:0/535407664 shutdown_connections 2023-11-28T17:09:07.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.209+0000 7fbf498cf640 1 --2- 172.21.15.116:0/535407664 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf2806b2a0 0x7fbf2806d760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.209+0000 7fbf498cf640 1 --2- 172.21.15.116:0/535407664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf4415a490 0x7fbf4413ea50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.209+0000 7fbf498cf640 1 --2- 172.21.15.116:0/535407664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf44140200 0x7fbf441502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.209+0000 7fbf498cf640 1 --2- 172.21.15.116:0/535407664 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf4413cee0 0x7fbf4413bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.209+0000 7fbf498cf640 1 -- 172.21.15.116:0/535407664 >> 172.21.15.116:0/535407664 conn(0x7fbf4406a4c0 msgr2=0x7fbf440b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:07.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.209+0000 7fbf498cf640 1 -- 172.21.15.116:0/535407664 shutdown_connections 2023-11-28T17:09:07.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.210+0000 7fbf498cf640 1 -- 172.21.15.116:0/535407664 wait complete. 2023-11-28T17:09:07.230 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-28T17:09:07.230 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:07.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.446+0000 7fc728e56640 1 Processor -- start 2023-11-28T17:09:07.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.446+0000 7fc728e56640 1 -- start start 2023-11-28T17:09:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.448+0000 7fc728e56640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc724144430 0x7fc72415f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.448+0000 7fc728e56640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc72406c250 0x7fc72406c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.448+0000 7fc728e56640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc724070bd0 con 0x7fc7240703f0 2023-11-28T17:09:07.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.448+0000 7fc728e56640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc724144810 con 0x7fc724144430 2023-11-28T17:09:07.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.448+0000 7fc728e56640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72406cc20 con 0x7fc72406c250 2023-11-28T17:09:07.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.448+0000 7fc72259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc724144430 0x7fc72415f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.448+0000 7fc72259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc724144430 0x7fc72415f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43722/0 (socket says 172.21.15.116:43722) 2023-11-28T17:09:07.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.448+0000 7fc72259c640 1 -- 172.21.15.116:0/1131768662 learned_addr learned my addr 172.21.15.116:0/1131768662 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:07.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.448+0000 7fc72259c640 1 -- 172.21.15.116:0/1131768662 >> v2:172.21.15.156:3300/0 conn(0x7fc72406c250 msgr2=0x7fc72406c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:07.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.448+0000 7fc72259c640 1 --2- 172.21.15.116:0/1131768662 >> v2:172.21.15.156:3300/0 conn(0x7fc72406c250 0x7fc72406c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.448+0000 7fc72259c640 1 -- 172.21.15.116:0/1131768662 >> v1:172.21.15.156:6789/0 conn(0x7fc7240703f0 legacy=0x7fc724071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:07.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.448+0000 7fc72259c640 1 -- 172.21.15.116:0/1131768662 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc724070eb0 con 0x7fc724144430 2023-11-28T17:09:07.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.449+0000 7fc72259c640 1 --2- 172.21.15.116:0/1131768662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc724144430 0x7fc72415f7b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc72406ce10 tx=0x7fc71402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e58ebdc6667bcac server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:07.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.450+0000 7fc72159a640 1 -- 172.21.15.116:0/1131768662 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71402ca80 con 0x7fc724144430 2023-11-28T17:09:07.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.450+0000 7fc72159a640 1 -- 172.21.15.116:0/1131768662 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71402cbe0 con 0x7fc724144430 2023-11-28T17:09:07.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.450+0000 7fc72159a640 1 -- 172.21.15.116:0/1131768662 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7140343f0 con 0x7fc724144430 2023-11-28T17:09:07.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.450+0000 7fc728e56640 1 -- 172.21.15.116:0/1131768662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc724144430 msgr2=0x7fc72415f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.450+0000 7fc728e56640 1 --2- 172.21.15.116:0/1131768662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc724144430 0x7fc72415f7b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc72406ce10 tx=0x7fc71402bee0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.451+0000 7fc728e56640 1 -- 172.21.15.116:0/1131768662 shutdown_connections 2023-11-28T17:09:07.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.451+0000 7fc728e56640 1 --2- 172.21.15.116:0/1131768662 >> v2:172.21.15.156:3300/0 conn(0x7fc72406c250 0x7fc72406c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.451+0000 7fc728e56640 1 --2- 172.21.15.116:0/1131768662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc724144430 0x7fc72415f7b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.451+0000 7fc728e56640 1 -- 172.21.15.116:0/1131768662 >> 172.21.15.116:0/1131768662 conn(0x7fc72406a4c0 msgr2=0x7fc72406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:07.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.451+0000 7fc728e56640 1 -- 172.21.15.116:0/1131768662 shutdown_connections 2023-11-28T17:09:07.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.451+0000 7fc728e56640 1 -- 172.21.15.116:0/1131768662 wait complete. 2023-11-28T17:09:07.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.452+0000 7fc728e56640 1 Processor -- start 2023-11-28T17:09:07.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.452+0000 7fc728e56640 1 -- start start 2023-11-28T17:09:07.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.453+0000 7fc728e56640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72406c250 0x7fc724145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.453+0000 7fc728e56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7240703f0 0x7fc724142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.453+0000 7fc728e56640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc724144430 0x7fc724145850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.453+0000 7fc728e56640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7240b03a0 con 0x7fc7240703f0 2023-11-28T17:09:07.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.453+0000 7fc728e56640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7240ac350 con 0x7fc724144430 2023-11-28T17:09:07.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.453+0000 7fc728e56640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7240ac630 con 0x7fc72406c250 2023-11-28T17:09:07.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.453+0000 7fc721d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7240703f0 0x7fc724142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.454+0000 7fc721d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7240703f0 0x7fc724142fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43728/0 (socket says 172.21.15.116:43728) 2023-11-28T17:09:07.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.454+0000 7fc721d9b640 1 -- 172.21.15.116:0/1111178940 learned_addr learned my addr 172.21.15.116:0/1111178940 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:07.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.454+0000 7fc72259c640 1 --2- 172.21.15.116:0/1111178940 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72406c250 0x7fc724145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.454+0000 7fc721d9b640 1 -- 172.21.15.116:0/1111178940 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72406c250 msgr2=0x7fc724145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.454+0000 7fc721d9b640 1 --2- 172.21.15.116:0/1111178940 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72406c250 0x7fc724145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.454+0000 7fc721d9b640 1 -- 172.21.15.116:0/1111178940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc724144430 msgr2=0x7fc724145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.454+0000 7fc721d9b640 1 --2- 172.21.15.116:0/1111178940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc724144430 0x7fc724145850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.454+0000 7fc721d9b640 1 -- 172.21.15.116:0/1111178940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc714009b30 con 0x7fc7240703f0 2023-11-28T17:09:07.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.454+0000 7fc721d9b640 1 --2- 172.21.15.116:0/1111178940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7240703f0 0x7fc724142fd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc71800b3e0 tx=0x7fc71800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:07.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.454+0000 7fc712ffd640 1 -- 172.21.15.116:0/1111178940 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718011830 con 0x7fc7240703f0 2023-11-28T17:09:07.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.455+0000 7fc712ffd640 1 -- 172.21.15.116:0/1111178940 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7180119b0 con 0x7fc7240703f0 2023-11-28T17:09:07.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.455+0000 7fc712ffd640 1 -- 172.21.15.116:0/1111178940 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718011d60 con 0x7fc7240703f0 2023-11-28T17:09:07.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.455+0000 7fc728e56640 1 -- 172.21.15.116:0/1111178940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7240ac910 con 0x7fc7240703f0 2023-11-28T17:09:07.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.455+0000 7fc728e56640 1 -- 172.21.15.116:0/1111178940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc724143510 con 0x7fc7240703f0 2023-11-28T17:09:07.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.456+0000 7fc728e56640 1 -- 172.21.15.116:0/1111178940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7240ace30 con 0x7fc7240703f0 2023-11-28T17:09:07.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.457+0000 7fc712ffd640 1 -- 172.21.15.116:0/1111178940 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fc71801a400 con 0x7fc7240703f0 2023-11-28T17:09:07.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.458+0000 7fc712ffd640 1 --2- 172.21.15.116:0/1111178940 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc70406b2a0 0x7fc70406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.458+0000 7fc712ffd640 1 -- 172.21.15.116:0/1111178940 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc718089cc0 con 0x7fc7240703f0 2023-11-28T17:09:07.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.458+0000 7fc72259c640 1 --2- 172.21.15.116:0/1111178940 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc70406b2a0 0x7fc70406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.459+0000 7fc72259c640 1 --2- 172.21.15.116:0/1111178940 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc70406b2a0 0x7fc70406d760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc714035000 tx=0x7fc714030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:07.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.462+0000 7fc712ffd640 1 -- 172.21.15.116:0/1111178940 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7240ace30 con 0x7fc7240703f0 2023-11-28T17:09:07.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.652+0000 7fc728e56640 1 -- 172.21.15.116:0/1111178940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7240ace30 con 0x7fc7240703f0 2023-11-28T17:09:07.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.654+0000 7fc712ffd640 1 -- 172.21.15.116:0/1111178940 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16521 (secure 0 0 0) 0x7fc7240ace30 con 0x7fc7240703f0 2023-11-28T17:09:07.655 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:07.656 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:01.970957+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:07.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.656+0000 7fc728e56640 1 -- 172.21.15.116:0/1111178940 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc70406b2a0 msgr2=0x7fc70406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.656+0000 7fc728e56640 1 --2- 172.21.15.116:0/1111178940 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc70406b2a0 0x7fc70406d760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc714035000 tx=0x7fc714030000 comp rx=0 tx=0).stop 2023-11-28T17:09:07.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.656+0000 7fc728e56640 1 -- 172.21.15.116:0/1111178940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7240703f0 msgr2=0x7fc724142fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.656+0000 7fc728e56640 1 --2- 172.21.15.116:0/1111178940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7240703f0 0x7fc724142fd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc71800b3e0 tx=0x7fc71800b8a0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.656+0000 7fc728e56640 1 -- 172.21.15.116:0/1111178940 shutdown_connections 2023-11-28T17:09:07.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.656+0000 7fc728e56640 1 --2- 172.21.15.116:0/1111178940 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc70406b2a0 0x7fc70406d760 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.656+0000 7fc728e56640 1 --2- 172.21.15.116:0/1111178940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc724144430 0x7fc724145850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.656+0000 7fc728e56640 1 --2- 172.21.15.116:0/1111178940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7240703f0 0x7fc724142fd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.656+0000 7fc728e56640 1 --2- 172.21.15.116:0/1111178940 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc72406c250 0x7fc724145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.656+0000 7fc728e56640 1 -- 172.21.15.116:0/1111178940 >> 172.21.15.116:0/1111178940 conn(0x7fc72406a4c0 msgr2=0x7fc7240ca640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:07.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.657+0000 7fc728e56640 1 -- 172.21.15.116:0/1111178940 shutdown_connections 2023-11-28T17:09:07.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.657+0000 7fc728e56640 1 -- 172.21.15.116:0/1111178940 wait complete. 2023-11-28T17:09:07.676 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-28T17:09:07.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-28T17:09:07.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-28T17:09:07.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-28T17:09:07.678 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-11-28T17:09:07.678 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-11-28T17:09:07.678 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-11-28T17:09:07.679 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-11-28T17:09:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.975+0000 7f77a1505640 1 Processor -- start 2023-11-28T17:09:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.975+0000 7f59a5cda640 1 Processor -- start 2023-11-28T17:09:07.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.976+0000 7f77a1505640 1 -- start start 2023-11-28T17:09:07.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.976+0000 7f59a5cda640 1 -- start start 2023-11-28T17:09:07.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.977+0000 7f98754c6640 1 Processor -- start 2023-11-28T17:09:07.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.977+0000 7f98754c6640 1 -- start start 2023-11-28T17:09:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.978+0000 7f77a1505640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7794039890 0x7f779404fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.978+0000 7f77a1505640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7794039d90 con 0x7f77940dfe80 2023-11-28T17:09:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.978+0000 7f77a1505640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779403a070 con 0x7f7794054310 2023-11-28T17:09:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.978+0000 7f77a1505640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f779404f040 con 0x7f7794039890 2023-11-28T17:09:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.978+0000 7f59a5cda640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59a015d310 0x7f59a00b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.978+0000 7f59a5cda640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f59a00afea0 0x7f59a00b3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.978+0000 7f59a5cda640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a00b2130 con 0x7f59a0150310 2023-11-28T17:09:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.978+0000 7f59a5cda640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a00b3da0 con 0x7f59a00afea0 2023-11-28T17:09:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.978+0000 7f59a5cda640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a00b4080 con 0x7f59a015d310 2023-11-28T17:09:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.979+0000 7f779affd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f77940dfe80 0x7f77940525c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54554/0 (socket says 172.21.15.116:54554) 2023-11-28T17:09:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.979+0000 7f779b7fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7794054310 0x7f77940506b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.116:50608/0 (socket says 172.21.15.116:50608) 2023-11-28T17:09:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.979+0000 7f779b7fe640 1 -- 172.21.15.116:0/1492243834 learned_addr learned my addr 172.21.15.116:0/1492243834 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:07.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.978+0000 7f599ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f59a00afea0 0x7f59a00b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.979+0000 7f599f7fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f59a0150310 0x7f59a015cc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.116:50594/0 (socket says 172.21.15.116:50594) 2023-11-28T17:09:07.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.979+0000 7f599f7fe640 1 -- 172.21.15.116:0/3520468444 learned_addr learned my addr 172.21.15.116:0/3520468444 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:07.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.979+0000 7f779a7fc640 1 --2- 172.21.15.116:0/1492243834 >> v2:172.21.15.156:3300/0 conn(0x7f7794039890 0x7f779404fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.979+0000 7f599effd640 1 --2- 172.21.15.116:0/3520468444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59a015d310 0x7f59a00b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.980+0000 7f599effd640 1 -- 172.21.15.116:0/3520468444 >> v2:172.21.15.116:3300/0 conn(0x7f59a00afea0 msgr2=0x7f59a00b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:07.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.980+0000 7f599effd640 1 --2- 172.21.15.116:0/3520468444 >> v2:172.21.15.116:3300/0 conn(0x7f59a00afea0 0x7f59a00b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.980+0000 7f599effd640 1 -- 172.21.15.116:0/3520468444 >> v1:172.21.15.156:6789/0 conn(0x7f59a0150310 legacy=0x7f59a015cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.980+0000 7f599effd640 1 -- 172.21.15.116:0/3520468444 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a00b4360 con 0x7f59a015d310 2023-11-28T17:09:07.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.980+0000 7f779a7fc640 1 -- 172.21.15.116:0/1492243834 >> v1:172.21.15.156:6789/0 conn(0x7f7794054310 legacy=0x7f77940506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.980+0000 7f779a7fc640 1 -- 172.21.15.116:0/1492243834 >> v1:172.21.15.116:6789/0 conn(0x7f77940dfe80 legacy=0x7f77940525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.981+0000 7f98754c6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9870143820 0x7f9870160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.981+0000 7f98754c6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f98700ac870 0x7f98701611e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.981+0000 7f98754c6640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f98700b3b60 0x7f98700b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.981+0000 7f98754c6640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9870143c00 con 0x7f9870143820 2023-11-28T17:09:07.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.981+0000 7f98754c6640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98700b4900 con 0x7f98700b3b60 2023-11-28T17:09:07.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.981+0000 7f98754c6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98700b4be0 con 0x7f98700ac870 2023-11-28T17:09:07.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.981+0000 7f986effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9870143820 0x7f9870160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.982+0000 7f986effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9870143820 0x7f9870160ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51750/0 (socket says 172.21.15.116:51750) 2023-11-28T17:09:07.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.982+0000 7f986effd640 1 -- 172.21.15.116:0/298704927 learned_addr learned my addr 172.21.15.116:0/298704927 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:07.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.982+0000 7f986effd640 1 -- 172.21.15.116:0/298704927 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f98700ac870 msgr2=0x7f98701611e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:07.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.981+0000 7f599effd640 1 --2- 172.21.15.116:0/3520468444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59a015d310 0x7f59a00b31c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f599800be30 tx=0x7f59980092f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b52603f651070fc2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:07.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.981+0000 7f779a7fc640 1 -- 172.21.15.116:0/1492243834 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779404f320 con 0x7f7794039890 2023-11-28T17:09:07.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.983+0000 7f986effd640 1 --2- 172.21.15.116:0/298704927 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f98700ac870 0x7f98701611e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.983+0000 7f986f7fe640 1 --2- 172.21.15.116:0/298704927 >> v2:172.21.15.156:3300/0 conn(0x7f98700b3b60 0x7f98700b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.983+0000 7f986effd640 1 -- 172.21.15.116:0/298704927 >> v2:172.21.15.156:3300/0 conn(0x7f98700b3b60 msgr2=0x7f98700b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:07.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.983+0000 7f779a7fc640 1 --2- 172.21.15.116:0/1492243834 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7794039890 0x7f779404fb20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f77900097a0 tx=0x7f779002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=306fcf50b768d3c0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:07.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.983+0000 7f986effd640 1 --2- 172.21.15.116:0/298704927 >> v2:172.21.15.156:3300/0 conn(0x7f98700b3b60 0x7f98700b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.983+0000 7f986effd640 1 -- 172.21.15.116:0/298704927 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98700b4ec0 con 0x7f9870143820 2023-11-28T17:09:07.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.983+0000 7f986f7fe640 1 --2- 172.21.15.116:0/298704927 >> v2:172.21.15.156:3300/0 conn(0x7f98700b3b60 0x7f98700b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:07.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.985+0000 7f986effd640 1 --2- 172.21.15.116:0/298704927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9870143820 0x7f9870160ca0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9870072170 tx=0x7f985802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7108017fc516ab30 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.986+0000 7f986dffb640 1 -- 172.21.15.116:0/298704927 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9858031870 con 0x7f9870143820 2023-11-28T17:09:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.986+0000 7f986dffb640 1 -- 172.21.15.116:0/298704927 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98580319d0 con 0x7f9870143820 2023-11-28T17:09:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.986+0000 7f986dffb640 1 -- 172.21.15.116:0/298704927 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9858031d60 con 0x7f9870143820 2023-11-28T17:09:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.986+0000 7f599e7fc640 1 -- 172.21.15.116:0/3520468444 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599802c880 con 0x7f59a015d310 2023-11-28T17:09:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.986+0000 7f599e7fc640 1 -- 172.21.15.116:0/3520468444 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599802c9e0 con 0x7f59a015d310 2023-11-28T17:09:07.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.986+0000 7f599e7fc640 1 -- 172.21.15.116:0/3520468444 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599802c880 con 0x7f59a015d310 2023-11-28T17:09:07.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.987+0000 7f7799ffb640 1 -- 172.21.15.116:0/1492243834 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7790031640 con 0x7f7794039890 2023-11-28T17:09:07.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.987+0000 7f59a5cda640 1 -- 172.21.15.116:0/3520468444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59a015d310 msgr2=0x7f59a00b31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.987+0000 7f59a5cda640 1 --2- 172.21.15.116:0/3520468444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59a015d310 0x7f59a00b31c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f599800be30 tx=0x7f59980092f0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.987+0000 7f7799ffb640 1 -- 172.21.15.116:0/1492243834 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77900317a0 con 0x7f7794039890 2023-11-28T17:09:07.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.987+0000 7f7799ffb640 1 -- 172.21.15.116:0/1492243834 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7790031b70 con 0x7f7794039890 2023-11-28T17:09:07.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.987+0000 7f98754c6640 1 -- 172.21.15.116:0/298704927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9870143820 msgr2=0x7f9870160ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.987+0000 7f98754c6640 1 --2- 172.21.15.116:0/298704927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9870143820 0x7f9870160ca0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9870072170 tx=0x7f985802b910 comp rx=0 tx=0).stop 2023-11-28T17:09:07.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.988+0000 7f59a5cda640 1 -- 172.21.15.116:0/3520468444 shutdown_connections 2023-11-28T17:09:07.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.988+0000 7f59a5cda640 1 --2- 172.21.15.116:0/3520468444 >> v2:172.21.15.116:3300/0 conn(0x7f59a00afea0 0x7f59a00b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.988+0000 7f59a5cda640 1 --2- 172.21.15.116:0/3520468444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59a015d310 0x7f59a00b31c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.988+0000 7f59a5cda640 1 -- 172.21.15.116:0/3520468444 >> 172.21.15.116:0/3520468444 conn(0x7f59a00aea80 msgr2=0x7f59a0072250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:07.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.988+0000 7f59a5cda640 1 -- 172.21.15.116:0/3520468444 shutdown_connections 2023-11-28T17:09:07.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.988+0000 7f59a5cda640 1 -- 172.21.15.116:0/3520468444 wait complete. 2023-11-28T17:09:07.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.989+0000 7f77a1505640 1 -- 172.21.15.116:0/1492243834 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7794039890 msgr2=0x7f779404fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:07.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.989+0000 7f98754c6640 1 -- 172.21.15.116:0/298704927 shutdown_connections 2023-11-28T17:09:07.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.989+0000 7f98754c6640 1 --2- 172.21.15.116:0/298704927 >> v2:172.21.15.156:3300/0 conn(0x7f98700b3b60 0x7f98700b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.989+0000 7f98754c6640 1 --2- 172.21.15.116:0/298704927 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f98700ac870 0x7f98701611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.989+0000 7f98754c6640 1 --2- 172.21.15.116:0/298704927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9870143820 0x7f9870160ca0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.989+0000 7f98754c6640 1 -- 172.21.15.116:0/298704927 >> 172.21.15.116:0/298704927 conn(0x7f987006c4b0 msgr2=0x7f987006c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:07.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.989+0000 7f77a1505640 1 --2- 172.21.15.116:0/1492243834 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7794039890 0x7f779404fb20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f77900097a0 tx=0x7f779002b8f0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.989+0000 7f98754c6640 1 -- 172.21.15.116:0/298704927 shutdown_connections 2023-11-28T17:09:07.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.989+0000 7f98754c6640 1 -- 172.21.15.116:0/298704927 wait complete. 2023-11-28T17:09:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.990+0000 7f77a1505640 1 -- 172.21.15.116:0/1492243834 shutdown_connections 2023-11-28T17:09:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.990+0000 7f77a1505640 1 --2- 172.21.15.116:0/1492243834 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7794039890 0x7f779404fb20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.990+0000 7f77a1505640 1 -- 172.21.15.116:0/1492243834 >> 172.21.15.116:0/1492243834 conn(0x7f7794009f70 msgr2=0x7f779400a350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.990+0000 7f77a1505640 1 -- 172.21.15.116:0/1492243834 shutdown_connections 2023-11-28T17:09:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.990+0000 7f77a1505640 1 -- 172.21.15.116:0/1492243834 wait complete. 2023-11-28T17:09:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.992+0000 7f98754c6640 1 Processor -- start 2023-11-28T17:09:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.992+0000 7f59a5cda640 1 Processor -- start 2023-11-28T17:09:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.993+0000 7f98754c6640 1 -- start start 2023-11-28T17:09:07.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.993+0000 7fabc97c4640 1 Processor -- start 2023-11-28T17:09:07.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.993+0000 7f59a5cda640 1 -- start start 2023-11-28T17:09:07.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.993+0000 7fabc97c4640 1 -- start start 2023-11-28T17:09:07.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.993+0000 7f77a1505640 1 Processor -- start 2023-11-28T17:09:07.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.994+0000 7f77a1505640 1 -- start start 2023-11-28T17:09:07.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.995+0000 7f98754c6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98700ac870 0x7f9870133f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.994+0000 7f59a5cda640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59a00afea0 0x7f59a0150730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.994+0000 7f59a5cda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59a0150310 0x7f59a013f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.994+0000 7f59a5cda640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59a015d310 0x7f59a0151db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.995+0000 7f98754c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98700b3b60 0x7f987012e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.995+0000 7f98754c6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9870143820 0x7f98701353e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:07.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.995+0000 7f599f7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59a00afea0 0x7f59a0150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f599f7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59a00afea0 0x7f59a0150730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49366/0 (socket says 172.21.15.116:49366) 2023-11-28T17:09:07.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f599effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59a0150310 0x7f59a013f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:07.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f599effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59a0150310 0x7f59a013f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51766/0 (socket says 172.21.15.116:51766) 2023-11-28T17:09:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f77a1505640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7794039890 0x7f7794185820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f98754c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9870134710 con 0x7f98700b3b60 2023-11-28T17:09:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f59a5cda640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a0150d10 con 0x7f59a0150310 2023-11-28T17:09:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f59a5cda640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a0150ff0 con 0x7f59a015d310 2023-11-28T17:09:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f59a5cda640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a01512d0 con 0x7f59a00afea0 2023-11-28T17:09:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f599f7fe640 1 -- 172.21.15.116:0/1228333717 learned_addr learned my addr 172.21.15.116:0/1228333717 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f77a1505640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7794054310 0x7f77940e1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f98754c6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987012fec0 con 0x7f98700ac870 2023-11-28T17:09:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f98754c6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987012fac0 con 0x7f9870143820 2023-11-28T17:09:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f986effd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98700ac870 0x7f9870133f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f986f7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9870143820 0x7f98701353e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f986f7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9870143820 0x7f98701353e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49370/0 (socket says 172.21.15.116:49370) 2023-11-28T17:09:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f986f7fe640 1 -- 172.21.15.116:0/1732226841 learned_addr learned my addr 172.21.15.116:0/1732226841 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7f986effd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98700ac870 0x7f9870133f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:58856/0 (socket says 172.21.15.116:58856) 2023-11-28T17:09:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f986e7fc640 1 --2- 172.21.15.116:0/1732226841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98700b3b60 0x7f987012e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7fabc97c4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fabc40b20a0 0x7fabc40d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7fabc97c4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fabc40b2ad0 0x7fabc40b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.996+0000 7fabc37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fabc40b2ad0 0x7fabc40b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7fabc97c4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc40cdcb0 con 0x7fabc40d1780 2023-11-28T17:09:08.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7fabc97c4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc40d39e0 con 0x7fabc40b2ad0 2023-11-28T17:09:08.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7fabc97c4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc40d3cc0 con 0x7fabc40b20a0 2023-11-28T17:09:08.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7fabc37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fabc40b2ad0 0x7fabc40b2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51782/0 (socket says 172.21.15.116:51782) 2023-11-28T17:09:08.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7fabc37fe640 1 -- 172.21.15.116:0/1021077240 learned_addr learned my addr 172.21.15.116:0/1021077240 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f77a1505640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77940dfe80 0x7f7794049bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f779affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7794039890 0x7f7794185820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f599ffff640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59a015d310 0x7f59a0151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f779affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7794039890 0x7f7794185820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:58880/0 (socket says 172.21.15.116:58880) 2023-11-28T17:09:08.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f986f7fe640 1 -- 172.21.15.116:0/1732226841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98700ac870 msgr2=0x7f9870133f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.998+0000 7f779a7fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7794054310 0x7f77940e1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.998+0000 7f779a7fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7794054310 0x7f77940e1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49374/0 (socket says 172.21.15.116:49374) 2023-11-28T17:09:08.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.998+0000 7f779affd640 1 -- 172.21.15.116:0/344139607 learned_addr learned my addr 172.21.15.116:0/344139607 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f599effd640 1 -- 172.21.15.116:0/1228333717 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59a00afea0 msgr2=0x7f59a0150730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f599effd640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59a00afea0 0x7f59a0150730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f599effd640 1 -- 172.21.15.116:0/1228333717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59a015d310 msgr2=0x7f59a0151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f599effd640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59a015d310 0x7f59a0151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f599effd640 1 -- 172.21.15.116:0/1228333717 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599800ba90 con 0x7f59a0150310 2023-11-28T17:09:08.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.998+0000 7f599f7fe640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f59a00afea0 0x7f59a0150730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:09:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.998+0000 7f599ffff640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59a015d310 0x7f59a0151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7fabc37fe640 1 -- 172.21.15.116:0/1021077240 >> v2:172.21.15.156:3300/0 conn(0x7fabc40b20a0 msgr2=0x7fabc40d01e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T17:09:08.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7fabc37fe640 1 --2- 172.21.15.116:0/1021077240 >> v2:172.21.15.156:3300/0 conn(0x7fabc40b20a0 0x7fabc40d01e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.998+0000 7fabc37fe640 1 -- 172.21.15.116:0/1021077240 >> v1:172.21.15.156:6789/0 conn(0x7fabc40d1780 legacy=0x7fabc40d1b60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-28T17:09:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.998+0000 7fabc37fe640 1 -- 172.21.15.116:0/1021077240 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc40cdf50 con 0x7fabc40b2ad0 2023-11-28T17:09:08.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f986f7fe640 1 --2- 172.21.15.116:0/1732226841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98700ac870 0x7f9870133f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f986f7fe640 1 -- 172.21.15.116:0/1732226841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98700b3b60 msgr2=0x7f987012e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f986f7fe640 1 --2- 172.21.15.116:0/1732226841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98700b3b60 0x7f987012e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.997+0000 7f986f7fe640 1 -- 172.21.15.116:0/1732226841 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98580094d0 con 0x7f9870143820 2023-11-28T17:09:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.998+0000 7f986e7fc640 1 --2- 172.21.15.116:0/1732226841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98700b3b60 0x7f987012e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:08.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.998+0000 7f986f7fe640 1 --2- 172.21.15.116:0/1732226841 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9870143820 0x7f98701353e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f986800ce40 tx=0x7f986800a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.998+0000 7f77a1505640 1 -- 172.21.15.116:0/344139607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77940d6970 con 0x7f77940dfe80 2023-11-28T17:09:08.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.998+0000 7f779b7fe640 1 --2- 172.21.15.116:0/344139607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77940dfe80 0x7f7794049bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.999+0000 7f77a1505640 1 -- 172.21.15.116:0/344139607 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77940e03e0 con 0x7f7794039890 2023-11-28T17:09:08.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.999+0000 7f77a1505640 1 -- 172.21.15.116:0/344139607 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77940e1240 con 0x7f7794054310 2023-11-28T17:09:08.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.999+0000 7f599effd640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59a0150310 0x7f59a013f520 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5998032000 tx=0x7f5998004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.999+0000 7f98677fe640 1 -- 172.21.15.116:0/1732226841 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986800d840 con 0x7f9870143820 2023-11-28T17:09:08.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.999+0000 7f98754c6640 1 -- 172.21.15.116:0/1732226841 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98701430b0 con 0x7f9870143820 2023-11-28T17:09:08.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.999+0000 7f98754c6640 1 -- 172.21.15.116:0/1732226841 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987012f630 con 0x7f9870143820 2023-11-28T17:09:08.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.999+0000 7f98677fe640 1 -- 172.21.15.116:0/1732226841 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986800d9a0 con 0x7f9870143820 2023-11-28T17:09:08.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.000+0000 7f779a7fc640 1 -- 172.21.15.116:0/344139607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7794039890 msgr2=0x7f7794185820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.000+0000 7f779a7fc640 1 --2- 172.21.15.116:0/344139607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7794039890 0x7f7794185820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.000+0000 7f5987fff640 1 -- 172.21.15.116:0/1228333717 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998004240 con 0x7f59a0150310 2023-11-28T17:09:08.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.000+0000 7f5987fff640 1 -- 172.21.15.116:0/1228333717 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59980043a0 con 0x7f59a0150310 2023-11-28T17:09:08.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.000+0000 7f5987fff640 1 -- 172.21.15.116:0/1228333717 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998004750 con 0x7f59a0150310 2023-11-28T17:09:08.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.000+0000 7f779a7fc640 1 -- 172.21.15.116:0/344139607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77940dfe80 msgr2=0x7f7794049bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.000+0000 7f59a5cda640 1 -- 172.21.15.116:0/1228333717 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a01515b0 con 0x7f59a0150310 2023-11-28T17:09:08.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.001+0000 7f59a5cda640 1 -- 172.21.15.116:0/1228333717 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a014caa0 con 0x7f59a0150310 2023-11-28T17:09:08.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.000+0000 7f779a7fc640 1 --2- 172.21.15.116:0/344139607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77940dfe80 0x7f7794049bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.000+0000 7f779a7fc640 1 -- 172.21.15.116:0/344139607 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7790009400 con 0x7f7794054310 2023-11-28T17:09:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:07.999+0000 7f98677fe640 1 -- 172.21.15.116:0/1732226841 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986800dd30 con 0x7f9870143820 2023-11-28T17:09:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.001+0000 7f779a7fc640 1 --2- 172.21.15.116:0/344139607 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7794054310 0x7f77940e1750 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7790033000 tx=0x7f77900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.001+0000 7fabc37fe640 1 --2- 172.21.15.116:0/1021077240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc40b2ad0 0x7fabc40b2f00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fabc4162190 tx=0x7fabb402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf6212d282154fe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.001+0000 7fa06dd53640 1 Processor -- start 2023-11-28T17:09:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.002+0000 7f77837fe640 1 -- 172.21.15.116:0/344139607 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779003bbf0 con 0x7f7794054310 2023-11-28T17:09:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.002+0000 7f77837fe640 1 -- 172.21.15.116:0/344139607 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779003bd70 con 0x7f7794054310 2023-11-28T17:09:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.002+0000 7f77a1505640 1 -- 172.21.15.116:0/344139607 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7794052e50 con 0x7f7794054310 2023-11-28T17:09:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.002+0000 7f59a5cda640 1 -- 172.21.15.116:0/1228333717 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f59a00b4920 con 0x7f59a0150310 2023-11-28T17:09:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.002+0000 7f62a5436640 1 Processor -- start 2023-11-28T17:09:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.000+0000 7f98754c6640 1 -- 172.21.15.116:0/1732226841 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9854000fc0 con 0x7f9870143820 2023-11-28T17:09:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.002+0000 7f77837fe640 1 -- 172.21.15.116:0/344139607 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7790045630 con 0x7f7794054310 2023-11-28T17:09:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.002+0000 7f77a1505640 1 -- 172.21.15.116:0/344139607 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77940e0630 con 0x7f7794054310 2023-11-28T17:09:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.002+0000 7fabc1ffb640 1 -- 172.21.15.116:0/1021077240 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4031800 con 0x7fabc40b2ad0 2023-11-28T17:09:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.002+0000 7fabc1ffb640 1 -- 172.21.15.116:0/1021077240 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb4031960 con 0x7fabc40b2ad0 2023-11-28T17:09:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.002+0000 7fabc1ffb640 1 -- 172.21.15.116:0/1021077240 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4031cf0 con 0x7fabc40b2ad0 2023-11-28T17:09:08.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.003+0000 7f62a5436640 1 -- start start 2023-11-28T17:09:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.003+0000 7fabc97c4640 1 -- 172.21.15.116:0/1021077240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc40b2ad0 msgr2=0x7fabc40b2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.003+0000 7fabc97c4640 1 --2- 172.21.15.116:0/1021077240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc40b2ad0 0x7fabc40b2f00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fabc4162190 tx=0x7fabb402b8a0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.003+0000 7fa06dd53640 1 -- start start 2023-11-28T17:09:08.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.004+0000 7f77a1505640 1 -- 172.21.15.116:0/344139607 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7794047f00 con 0x7f7794054310 2023-11-28T17:09:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.004+0000 7efcbaa90640 1 Processor -- start 2023-11-28T17:09:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.004+0000 7fabc97c4640 1 -- 172.21.15.116:0/1021077240 shutdown_connections 2023-11-28T17:09:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.004+0000 7fabc97c4640 1 --2- 172.21.15.116:0/1021077240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc40b2ad0 0x7fabc40b2f00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.004+0000 7fabc97c4640 1 --2- 172.21.15.116:0/1021077240 >> v2:172.21.15.156:3300/0 conn(0x7fabc40b20a0 0x7fabc40d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.004+0000 7fabc97c4640 1 -- 172.21.15.116:0/1021077240 >> 172.21.15.116:0/1021077240 conn(0x7fabc409c4e0 msgr2=0x7fabc40d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.004+0000 7fabc97c4640 1 -- 172.21.15.116:0/1021077240 shutdown_connections 2023-11-28T17:09:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.005+0000 7efcbaa90640 1 -- start start 2023-11-28T17:09:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.006+0000 7fa06dd53640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa068146a50 0x7fa068161a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.006+0000 7fa06dd53640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa0680cd970 0x7fa0680cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.006+0000 7fa06dd53640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0680cadf0 con 0x7fa0680b2ba0 2023-11-28T17:09:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.006+0000 7fa06dd53640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0680ce7c0 con 0x7fa0680cd970 2023-11-28T17:09:08.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.006+0000 7fa06dd53640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0680ce4e0 con 0x7fa068146a50 2023-11-28T17:09:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.005+0000 7fabc97c4640 1 -- 172.21.15.116:0/1021077240 wait complete. 2023-11-28T17:09:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.006+0000 7f62a5436640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f62a00b20a0 0x7f62a00d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.005+0000 7f98677fe640 1 -- 172.21.15.116:0/1732226841 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f9868015440 con 0x7f9870143820 2023-11-28T17:09:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.006+0000 7f62a5436640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62a00b2ad0 0x7f62a00ce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.006+0000 7f62a5436640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a00cdcb0 con 0x7f62a00d1780 2023-11-28T17:09:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.006+0000 7f62a5436640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a00d39e0 con 0x7f62a00b20a0 2023-11-28T17:09:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.006+0000 7f62a5436640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a00d3cc0 con 0x7f62a00b2ad0 2023-11-28T17:09:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.006+0000 7f77837fe640 1 -- 172.21.15.116:0/344139607 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f779002f050 con 0x7f7794054310 2023-11-28T17:09:08.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.005+0000 7f5987fff640 1 -- 172.21.15.116:0/1228333717 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f5998040470 con 0x7f59a0150310 2023-11-28T17:09:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.006+0000 7f5987fff640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f598c06b2f0 0x7f598c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.006+0000 7f5987fff640 1 -- 172.21.15.116:0/1228333717 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59980aca20 con 0x7f59a0150310 2023-11-28T17:09:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.006+0000 7f5987fff640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.116:6808/1361195238,v1:172.21.15.116:6809/1361195238] conn(0x7f598c073510 0x7f598c075930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7f5987fff640 1 -- 172.21.15.116:0/1228333717 --> [v2:172.21.15.116:6808/1361195238,v1:172.21.15.116:6809/1361195238] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f598c075fe0 con 0x7f598c073510 2023-11-28T17:09:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7f5987fff640 1 -- 172.21.15.116:0/1228333717 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59980b12a0 con 0x7f59a0150310 2023-11-28T17:09:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7f629e7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f62a00b20a0 0x7f62a00d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7f629e7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f62a00b20a0 0x7f62a00d01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51802/0 (socket says 172.21.15.116:51802) 2023-11-28T17:09:08.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7f629e7fc640 1 -- 172.21.15.116:0/2591839991 learned_addr learned my addr 172.21.15.116:0/2591839991 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7f629f7fe640 1 --2- 172.21.15.116:0/2591839991 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62a00b2ad0 0x7f62a00ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7fa0677fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa068146a50 0x7fa068161a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7fa0677fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa068146a50 0x7fa068161a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49382/0 (socket says 172.21.15.116:49382) 2023-11-28T17:09:08.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7fa0677fe640 1 -- 172.21.15.116:0/3587900449 learned_addr learned my addr 172.21.15.116:0/3587900449 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7f629e7fc640 1 -- 172.21.15.116:0/2591839991 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62a00b2ad0 msgr2=0x7f62a00ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7f629e7fc640 1 --2- 172.21.15.116:0/2591839991 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62a00b2ad0 0x7f62a00ce750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7f629e7fc640 1 -- 172.21.15.116:0/2591839991 >> v1:172.21.15.116:6789/0 conn(0x7f62a00d1780 legacy=0x7f62a00d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7f629e7fc640 1 -- 172.21.15.116:0/2591839991 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a00cdf50 con 0x7f62a00b20a0 2023-11-28T17:09:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7f629f7fe640 1 --2- 172.21.15.116:0/2591839991 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62a00b2ad0 0x7f62a00ce750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:09:08.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.008+0000 7fabc97c4640 1 Processor -- start 2023-11-28T17:09:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.008+0000 7efcbaa90640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7efcb40b3850 0x7efcb40cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.008+0000 7fabc97c4640 1 -- start start 2023-11-28T17:09:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.008+0000 7efcbaa90640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcb40cb650 0x7efcb40703f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.008+0000 7efcbaa90640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb40b2770 con 0x7efcb41594f0 2023-11-28T17:09:08.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.008+0000 7efcbaa90640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb40ce090 con 0x7efcb40b3850 2023-11-28T17:09:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.008+0000 7efcbaa90640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40b3490 con 0x7efcb40cb650 2023-11-28T17:09:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.008+0000 7efcabfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efcb41594f0 0x7efcb40b3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54582/0 (socket says 172.21.15.116:54582) 2023-11-28T17:09:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.008+0000 7efcabfff640 1 -- 172.21.15.116:0/456489220 learned_addr learned my addr 172.21.15.116:0/456489220 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.009+0000 7efcb902d640 1 --2- 172.21.15.116:0/456489220 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcb40cb650 0x7efcb40703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.009+0000 7efcb882c640 1 --2- 172.21.15.116:0/456489220 >> v2:172.21.15.156:3300/0 conn(0x7efcb40b3850 0x7efcb40cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.009+0000 7fa0677fe640 1 -- 172.21.15.116:0/3587900449 >> v2:172.21.15.116:3300/0 conn(0x7fa0680cd970 msgr2=0x7fa0680cddd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:09:08.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.009+0000 7f629e7fc640 1 --2- 172.21.15.116:0/2591839991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62a00b20a0 0x7f62a00d01e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6294009870 tx=0x7f629402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770d98fddf39f5c9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.009+0000 7efcb902d640 1 -- 172.21.15.116:0/456489220 >> v2:172.21.15.156:3300/0 conn(0x7efcb40b3850 msgr2=0x7efcb40cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.009+0000 7efcb902d640 1 --2- 172.21.15.116:0/456489220 >> v2:172.21.15.156:3300/0 conn(0x7efcb40b3850 0x7efcb40cd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.009+0000 7efcb902d640 1 -- 172.21.15.116:0/456489220 >> v1:172.21.15.116:6789/0 conn(0x7efcb41594f0 legacy=0x7efcb40b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.009+0000 7efcb902d640 1 -- 172.21.15.116:0/456489220 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb40cddb0 con 0x7efcb40cb650 2023-11-28T17:09:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7fabc97c4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fabc40b20a0 0x7fabc4158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7fabc97c4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc40b2ad0 0x7fabc41573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7fabc97c4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabc40d1780 0x7fabc4157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7fabc97c4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc40ccc30 con 0x7fabc40b2ad0 2023-11-28T17:09:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7fabc97c4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc40cc0b0 con 0x7fabc40b20a0 2023-11-28T17:09:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7fabc97c4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc40cc390 con 0x7fabc40d1780 2023-11-28T17:09:08.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.009+0000 7fa066ffd640 1 --2- 172.21.15.116:0/3587900449 >> v2:172.21.15.116:3300/0 conn(0x7fa0680cd970 0x7fa0680cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7f629dffb640 1 -- 172.21.15.116:0/2591839991 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62940317c0 con 0x7f62a00b20a0 2023-11-28T17:09:08.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7f629dffb640 1 -- 172.21.15.116:0/2591839991 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6294031920 con 0x7f62a00b20a0 2023-11-28T17:09:08.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7f629dffb640 1 -- 172.21.15.116:0/2591839991 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62940317c0 con 0x7f62a00b20a0 2023-11-28T17:09:08.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7fabc37fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabc40d1780 0x7fabc4157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.011+0000 7fabc37fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabc40d1780 0x7fabc4157900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49410/0 (socket says 172.21.15.116:49410) 2023-11-28T17:09:08.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.011+0000 7f62a5436640 1 -- 172.21.15.116:0/2591839991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62a00b20a0 msgr2=0x7f62a00d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.011+0000 7f62a5436640 1 --2- 172.21.15.116:0/2591839991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62a00b20a0 0x7f62a00d01e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6294009870 tx=0x7f629402b860 comp rx=0 tx=0).stop 2023-11-28T17:09:08.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.011+0000 7fabc37fe640 1 -- 172.21.15.116:0/3017224462 learned_addr learned my addr 172.21.15.116:0/3017224462 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.011+0000 7f62a5436640 1 -- 172.21.15.116:0/2591839991 shutdown_connections 2023-11-28T17:09:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.011+0000 7f62a5436640 1 --2- 172.21.15.116:0/2591839991 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62a00b2ad0 0x7f62a00ce750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.011+0000 7f62a5436640 1 --2- 172.21.15.116:0/2591839991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62a00b20a0 0x7f62a00d01e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.011+0000 7f62a5436640 1 -- 172.21.15.116:0/2591839991 >> 172.21.15.116:0/2591839991 conn(0x7f62a009c4e0 msgr2=0x7f62a00d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.011+0000 7f62a5436640 1 -- 172.21.15.116:0/2591839991 shutdown_connections 2023-11-28T17:09:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.008+0000 7f98677fe640 1 --2- 172.21.15.116:0/1732226841 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f985006b2a0 0x7f985006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.009+0000 7f98677fe640 1 -- 172.21.15.116:0/1732226841 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f986808b070 con 0x7f9870143820 2023-11-28T17:09:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.009+0000 7f98677fe640 1 --2- 172.21.15.116:0/1732226841 >> [v2:172.21.15.116:6824/335787082,v1:172.21.15.116:6825/335787082] conn(0x7f98500734c0 0x7f98500758e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.009+0000 7f98677fe640 1 -- 172.21.15.116:0/1732226841 --> [v2:172.21.15.116:6824/335787082,v1:172.21.15.116:6825/335787082] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9850075f90 con 0x7f98500734c0 2023-11-28T17:09:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.009+0000 7f98677fe640 1 -- 172.21.15.116:0/1732226841 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f986808b420 con 0x7f9870143820 2023-11-28T17:09:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7f986e7fc640 1 --2- 172.21.15.116:0/1732226841 >> [v2:172.21.15.116:6824/335787082,v1:172.21.15.116:6825/335787082] conn(0x7f98500734c0 0x7f98500758e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.011+0000 7f986e7fc640 1 --2- 172.21.15.116:0/1732226841 >> [v2:172.21.15.116:6824/335787082,v1:172.21.15.116:6825/335787082] conn(0x7f98500734c0 0x7f98500758e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.012+0000 7f986effd640 1 --2- 172.21.15.116:0/1732226841 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f985006b2a0 0x7f985006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.007+0000 7f599ffff640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.116:6808/1361195238,v1:172.21.15.116:6809/1361195238] conn(0x7f598c073510 0x7f598c075930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.009+0000 7f599ffff640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.116:6808/1361195238,v1:172.21.15.116:6809/1361195238] conn(0x7f598c073510 0x7f598c075930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7f5987fff640 1 -- 172.21.15.116:0/1228333717 <== osd.2 v2:172.21.15.116:6808/1361195238 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f598c075fe0 con 0x7f598c073510 2023-11-28T17:09:08.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.011+0000 7fabc2ffd640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fabc40b20a0 0x7fabc4158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.012+0000 7f599f7fe640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f598c06b2f0 0x7f598c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7fa0677fe640 1 --2- 172.21.15.116:0/3587900449 >> v2:172.21.15.116:3300/0 conn(0x7fa0680cd970 0x7fa0680cddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7fa0677fe640 1 -- 172.21.15.116:0/3587900449 >> v1:172.21.15.116:6789/0 conn(0x7fa0680b2ba0 legacy=0x7fa06806be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7fa0677fe640 1 -- 172.21.15.116:0/3587900449 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa068160fa0 con 0x7fa068146a50 2023-11-28T17:09:08.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.012+0000 7fabc27fc640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc40b2ad0 0x7fabc41573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.013+0000 7fabc2ffd640 1 -- 172.21.15.116:0/3017224462 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabc40d1780 msgr2=0x7fabc4157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.013+0000 7fabc2ffd640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabc40d1780 0x7fabc4157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.013+0000 7fabc2ffd640 1 -- 172.21.15.116:0/3017224462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc40b2ad0 msgr2=0x7fabc41573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.013+0000 7fabc2ffd640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc40b2ad0 0x7fabc41573c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.013+0000 7fabc2ffd640 1 -- 172.21.15.116:0/3017224462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb40094d0 con 0x7fabc40b20a0 2023-11-28T17:09:08.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.013+0000 7fabc27fc640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc40b2ad0 0x7fabc41573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:09:08.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7f77837fe640 1 --2- 172.21.15.116:0/344139607 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f778806b2a0 0x7f778806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7f77837fe640 1 -- 172.21.15.116:0/344139607 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77900ac070 con 0x7f7794054310 2023-11-28T17:09:08.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7f77837fe640 1 --2- 172.21.15.116:0/344139607 >> [v2:172.21.15.116:6800/671929056,v1:172.21.15.116:6801/671929056] conn(0x7f7788073440 0x7f7788075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7f77837fe640 1 -- 172.21.15.116:0/344139607 --> [v2:172.21.15.116:6800/671929056,v1:172.21.15.116:6801/671929056] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7788075f10 con 0x7f7788073440 2023-11-28T17:09:08.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.010+0000 7f77837fe640 1 -- 172.21.15.116:0/344139607 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77900ac3e0 con 0x7f7794054310 2023-11-28T17:09:08.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.011+0000 7f779b7fe640 1 --2- 172.21.15.116:0/344139607 >> [v2:172.21.15.116:6800/671929056,v1:172.21.15.116:6801/671929056] conn(0x7f7788073440 0x7f7788075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.011+0000 7f779affd640 1 --2- 172.21.15.116:0/344139607 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f778806b2a0 0x7f778806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.012+0000 7f779b7fe640 1 --2- 172.21.15.116:0/344139607 >> [v2:172.21.15.116:6800/671929056,v1:172.21.15.116:6801/671929056] conn(0x7f7788073440 0x7f7788075860 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.012+0000 7f779affd640 1 --2- 172.21.15.116:0/344139607 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f778806b2a0 0x7f778806d760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f778c000f60 tx=0x7f778c007f00 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.013+0000 7f77837fe640 1 -- 172.21.15.116:0/344139607 <== osd.0 v2:172.21.15.116:6800/671929056 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7788075f10 con 0x7f7788073440 2023-11-28T17:09:08.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.013+0000 7fabc2ffd640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fabc40b20a0 0x7fabc4158540 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00be20 tx=0x7fabbc009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.013+0000 7efcb902d640 1 --2- 172.21.15.116:0/456489220 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcb40cb650 0x7efcb40703f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efcac0097a0 tx=0x7efcac02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6559d7d8c56dde49 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.014+0000 7f986effd640 1 --2- 172.21.15.116:0/1732226841 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f985006b2a0 0x7f985006d760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9858036000 tx=0x7f985803c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.014+0000 7f98677fe640 1 -- 172.21.15.116:0/1732226841 <== osd.1 v2:172.21.15.116:6824/335787082 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9850075f90 con 0x7f98500734c0 2023-11-28T17:09:08.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.011+0000 7f62a5436640 1 -- 172.21.15.116:0/2591839991 wait complete. 2023-11-28T17:09:08.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.014+0000 7f599f7fe640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f598c06b2f0 0x7f598c06d7b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5990005880 tx=0x7f5990009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.014+0000 7efcb3fff640 1 -- 172.21.15.116:0/456489220 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac0316f0 con 0x7efcb40cb650 2023-11-28T17:09:08.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.014+0000 7efcb3fff640 1 -- 172.21.15.116:0/456489220 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac031850 con 0x7efcb40cb650 2023-11-28T17:09:08.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.014+0000 7efcb3fff640 1 -- 172.21.15.116:0/456489220 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac031be0 con 0x7efcb40cb650 2023-11-28T17:09:08.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.014+0000 7fabab7fe640 1 -- 172.21.15.116:0/3017224462 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc014440 con 0x7fabc40b20a0 2023-11-28T17:09:08.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.014+0000 7fabab7fe640 1 -- 172.21.15.116:0/3017224462 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabbc0145a0 con 0x7fabc40b20a0 2023-11-28T17:09:08.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.014+0000 7fabab7fe640 1 -- 172.21.15.116:0/3017224462 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc014990 con 0x7fabc40b20a0 2023-11-28T17:09:08.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.014+0000 7fabc97c4640 1 -- 172.21.15.116:0/3017224462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc40cc670 con 0x7fabc40b20a0 2023-11-28T17:09:08.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.014+0000 7fabc97c4640 1 -- 172.21.15.116:0/3017224462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc40cbcd0 con 0x7fabc40b20a0 2023-11-28T17:09:08.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.014+0000 7f62a5436640 1 Processor -- start 2023-11-28T17:09:08.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.015+0000 7efcbaa90640 1 -- 172.21.15.116:0/456489220 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcb40cb650 msgr2=0x7efcb40703f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.015+0000 7efcbaa90640 1 --2- 172.21.15.116:0/456489220 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcb40cb650 0x7efcb40703f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efcac0097a0 tx=0x7efcac02b790 comp rx=0 tx=0).stop 2023-11-28T17:09:08.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.014+0000 7f62a5436640 1 -- start start 2023-11-28T17:09:08.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.015+0000 7fabc97c4640 1 -- 172.21.15.116:0/3017224462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fabc415d900 con 0x7fabc40b20a0 2023-11-28T17:09:08.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.016+0000 7efcbaa90640 1 -- 172.21.15.116:0/456489220 shutdown_connections 2023-11-28T17:09:08.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.016+0000 7efcbaa90640 1 --2- 172.21.15.116:0/456489220 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcb40cb650 0x7efcb40703f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.016+0000 7efcbaa90640 1 --2- 172.21.15.116:0/456489220 >> v2:172.21.15.156:3300/0 conn(0x7efcb40b3850 0x7efcb40cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.016+0000 7fa0677fe640 1 --2- 172.21.15.116:0/3587900449 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa068146a50 0x7fa068161a80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa060009870 tx=0x7fa06002b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4629f957c2e5786 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.016+0000 7efcbaa90640 1 -- 172.21.15.116:0/456489220 >> 172.21.15.116:0/456489220 conn(0x7efcb406c250 msgr2=0x7efcb406c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.016+0000 7f62a5436640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62a00b20a0 0x7f62a00c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.016+0000 7f62a5436640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62a00b2ad0 0x7f62a00b8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.016+0000 7f62a5436640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a00d1780 0x7f62a00c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.016+0000 7f62a5436640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a00cb070 con 0x7f62a00b20a0 2023-11-28T17:09:08.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.016+0000 7f62a5436640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a00b73d0 con 0x7f62a00d1780 2023-11-28T17:09:08.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.016+0000 7f62a5436640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a00b76b0 con 0x7f62a00b2ad0 2023-11-28T17:09:08.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.016+0000 7efcbaa90640 1 -- 172.21.15.116:0/456489220 shutdown_connections 2023-11-28T17:09:08.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.016+0000 7efcbaa90640 1 -- 172.21.15.116:0/456489220 wait complete. 2023-11-28T17:09:08.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.017+0000 7f629e7fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62a00b2ad0 0x7f62a00b8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.017+0000 7f629effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62a00b20a0 0x7f62a00c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.017+0000 7f629e7fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62a00b2ad0 0x7f62a00b8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49418/0 (socket says 172.21.15.116:49418) 2023-11-28T17:09:08.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.017+0000 7f629e7fc640 1 -- 172.21.15.116:0/193447209 learned_addr learned my addr 172.21.15.116:0/193447209 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.017+0000 7f629effd640 1 -- 172.21.15.116:0/193447209 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62a00b2ad0 msgr2=0x7f62a00b8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.018+0000 7f629f7fe640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a00d1780 0x7f62a00c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.018+0000 7f629effd640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62a00b2ad0 0x7f62a00b8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.018+0000 7f629effd640 1 -- 172.21.15.116:0/193447209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a00d1780 msgr2=0x7f62a00c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.018+0000 7f629effd640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a00d1780 0x7f62a00c6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.018+0000 7f629effd640 1 -- 172.21.15.116:0/193447209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62940094d0 con 0x7f62a00b20a0 2023-11-28T17:09:08.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.018+0000 7f629e7fc640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62a00b2ad0 0x7f62a00b8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:09:08.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.018+0000 7f629f7fe640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62a00d1780 0x7f62a00c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:08.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.018+0000 7efcbaa90640 1 Processor -- start 2023-11-28T17:09:08.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.019+0000 7fabab7fe640 1 -- 172.21.15.116:0/3017224462 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fabbc014af0 con 0x7fabc40b20a0 2023-11-28T17:09:08.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.019+0000 7efcbaa90640 1 -- start start 2023-11-28T17:09:08.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.019+0000 7f629effd640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62a00b20a0 0x7f62a00c8e20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f629000b3e0 tx=0x7f629000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.019+0000 7f62877fe640 1 -- 172.21.15.116:0/193447209 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290011830 con 0x7f62a00b20a0 2023-11-28T17:09:08.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.019+0000 7efcbaa90640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcb40b3850 0x7efcb41510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7efcbaa90640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcb40cb650 0x7efcb40afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7fabab7fe640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fabb006b2a0 0x7fabb006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.021+0000 7fabab7fe640 1 -- 172.21.15.116:0/3017224462 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fabbc059080 con 0x7fabc40b20a0 2023-11-28T17:09:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7efcbaa90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb40b03c0 0x7efcb413e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7efcbaa90640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb4151980 con 0x7efcb40b03c0 2023-11-28T17:09:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7efcbaa90640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb40b0850 con 0x7efcb40cb650 2023-11-28T17:09:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7efcbaa90640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb414e700 con 0x7efcb40b3850 2023-11-28T17:09:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7efcabfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcb40cb650 0x7efcb40afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7efcabfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcb40cb650 0x7efcb40afe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:58924/0 (socket says 172.21.15.116:58924) 2023-11-28T17:09:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7efcabfff640 1 -- 172.21.15.116:0/1241642622 learned_addr learned my addr 172.21.15.116:0/1241642622 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7efcb882c640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcb40b3850 0x7efcb41510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7f62a5436640 1 -- 172.21.15.116:0/193447209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a00b7990 con 0x7f62a00b20a0 2023-11-28T17:09:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7f62a5436640 1 -- 172.21.15.116:0/193447209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a00b7f50 con 0x7f62a00b20a0 2023-11-28T17:09:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7fa0667fc640 1 -- 172.21.15.116:0/3587900449 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0600317c0 con 0x7fa068146a50 2023-11-28T17:09:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7fa0667fc640 1 -- 172.21.15.116:0/3587900449 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa060031920 con 0x7fa068146a50 2023-11-28T17:09:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7fa0667fc640 1 -- 172.21.15.116:0/3587900449 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060031d10 con 0x7fa068146a50 2023-11-28T17:09:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.021+0000 7fa06dd53640 1 -- 172.21.15.116:0/3587900449 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa068146a50 msgr2=0x7fa068161a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.021+0000 7fa06dd53640 1 --2- 172.21.15.116:0/3587900449 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa068146a50 0x7fa068161a80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa060009870 tx=0x7fa06002b880 comp rx=0 tx=0).stop 2023-11-28T17:09:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.021+0000 7efcb902d640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb40b03c0 0x7efcb413e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.021+0000 7efcb882c640 1 -- 172.21.15.116:0/1241642622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcb40cb650 msgr2=0x7efcb40afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.021+0000 7efcb882c640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcb40cb650 0x7efcb40afe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.021+0000 7efcb882c640 1 -- 172.21.15.116:0/1241642622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb40b03c0 msgr2=0x7efcb413e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.021+0000 7fabab7fe640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.116:6811/3907008760,v1:172.21.15.116:6814/3907008760] conn(0x7fabb0074120 0x7fabb00765c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.021+0000 7fabab7fe640 1 -- 172.21.15.116:0/3017224462 --> [v2:172.21.15.116:6811/3907008760,v1:172.21.15.116:6814/3907008760] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabb0076c90 con 0x7fabb0074120 2023-11-28T17:09:08.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.021+0000 7fabab7fe640 1 -- 172.21.15.116:0/3017224462 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabbc009890 con 0x7fabc40b20a0 2023-11-28T17:09:08.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.021+0000 7fabc27fc640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fabb006b2a0 0x7fabb006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7f62877fe640 1 -- 172.21.15.116:0/193447209 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6290011990 con 0x7f62a00b20a0 2023-11-28T17:09:08.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.020+0000 7f62877fe640 1 -- 172.21.15.116:0/193447209 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290011830 con 0x7f62a00b20a0 2023-11-28T17:09:08.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.021+0000 7fa06dd53640 1 -- 172.21.15.116:0/3587900449 shutdown_connections 2023-11-28T17:09:08.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.022+0000 7fa06dd53640 1 --2- 172.21.15.116:0/3587900449 >> v2:172.21.15.116:3300/0 conn(0x7fa0680cd970 0x7fa0680cddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.022+0000 7fa06dd53640 1 --2- 172.21.15.116:0/3587900449 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa068146a50 0x7fa068161a80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.022+0000 7fa06dd53640 1 -- 172.21.15.116:0/3587900449 >> 172.21.15.116:0/3587900449 conn(0x7fa06806a4c0 msgr2=0x7fa06806a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.022+0000 7fa06dd53640 1 -- 172.21.15.116:0/3587900449 shutdown_connections 2023-11-28T17:09:08.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.022+0000 7fabc27fc640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fabb006b2a0 0x7fabb006d760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fabb8006fd0 tx=0x7fabb8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.022+0000 7fa06dd53640 1 -- 172.21.15.116:0/3587900449 wait complete. 2023-11-28T17:09:08.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.022+0000 7f30adec6640 1 Processor -- start 2023-11-28T17:09:08.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.023+0000 7f30adec6640 1 -- start start 2023-11-28T17:09:08.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.023+0000 7fabc37fe640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.116:6811/3907008760,v1:172.21.15.116:6814/3907008760] conn(0x7fabb0074120 0x7fabb00765c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.022+0000 7efcb882c640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb40b03c0 0x7efcb413e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.022+0000 7efcb882c640 1 -- 172.21.15.116:0/1241642622 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac009400 con 0x7efcb40b3850 2023-11-28T17:09:08.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.023+0000 7efcb902d640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcb40b03c0 0x7efcb413e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:08.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.023+0000 7fabc37fe640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.116:6811/3907008760,v1:172.21.15.116:6814/3907008760] conn(0x7fabb0074120 0x7fabb00765c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.024+0000 7efcb882c640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcb40b3850 0x7efcb41510c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efca400efc0 tx=0x7efca400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.024+0000 7efcb1ffb640 1 -- 172.21.15.116:0/1241642622 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4009e80 con 0x7efcb40b3850 2023-11-28T17:09:08.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.024+0000 7efcb1ffb640 1 -- 172.21.15.116:0/1241642622 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca400f040 con 0x7efcb40b3850 2023-11-28T17:09:08.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.024+0000 7efcb1ffb640 1 -- 172.21.15.116:0/1241642622 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4014680 con 0x7efcb40b3850 2023-11-28T17:09:08.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.024+0000 7fabab7fe640 1 -- 172.21.15.116:0/3017224462 <== osd.3 v2:172.21.15.116:6811/3907008760 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fabb0076c90 con 0x7fabb0074120 2023-11-28T17:09:08.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.025+0000 7efcbaa90640 1 -- 172.21.15.116:0/1241642622 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb41f74f0 con 0x7efcb40b3850 2023-11-28T17:09:08.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.025+0000 7fa06dd53640 1 Processor -- start 2023-11-28T17:09:08.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.026+0000 7fa06dd53640 1 -- start start 2023-11-28T17:09:08.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.026+0000 7f30adec6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f30a81594f0 0x7f30a80cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.026+0000 7f30adec6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a80cb650 0x7f30a80703f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.025+0000 7efcbaa90640 1 -- 172.21.15.116:0/1241642622 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb413c280 con 0x7efcb40b3850 2023-11-28T17:09:08.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.026+0000 7efcbaa90640 1 -- 172.21.15.116:0/1241642622 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efcb40cec50 con 0x7efcb40b3850 2023-11-28T17:09:08.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.026+0000 7f62a5436640 1 -- 172.21.15.116:0/193447209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6280000fc0 con 0x7f62a00b20a0 2023-11-28T17:09:08.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.026+0000 7f30adec6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a80b2770 con 0x7f30a80b3850 2023-11-28T17:09:08.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.026+0000 7f30adec6640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a80ce090 con 0x7f30a81594f0 2023-11-28T17:09:08.064 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.026+0000 7f30adec6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a80cddb0 con 0x7f30a80cb650 2023-11-28T17:09:08.064 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.027+0000 7f30a7fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a80cb650 0x7f30a80703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.064 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.027+0000 7f30a7fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a80cb650 0x7f30a80703f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49440/0 (socket says 172.21.15.116:49440) 2023-11-28T17:09:08.064 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.027+0000 7f30a7fff640 1 -- 172.21.15.116:0/767561753 learned_addr learned my addr 172.21.15.116:0/767561753 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.064 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.028+0000 7fa06dd53640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0680b2ba0 0x7fa06806bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.027+0000 7f30a6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f30a81594f0 0x7f30a80cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.028+0000 7f30a7fff640 1 -- 172.21.15.116:0/767561753 >> v2:172.21.15.116:3300/0 conn(0x7f30a81594f0 msgr2=0x7f30a80cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.028+0000 7f30a7fff640 1 --2- 172.21.15.116:0/767561753 >> v2:172.21.15.116:3300/0 conn(0x7f30a81594f0 0x7f30a80cd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.028+0000 7f30a7fff640 1 -- 172.21.15.116:0/767561753 >> v1:172.21.15.116:6789/0 conn(0x7f30a80b3850 legacy=0x7f30a80b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.028+0000 7f30a7fff640 1 -- 172.21.15.116:0/767561753 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a80b2120 con 0x7f30a80cb650 2023-11-28T17:09:08.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.028+0000 7f30a6ffd640 1 --2- 172.21.15.116:0/767561753 >> v2:172.21.15.116:3300/0 conn(0x7f30a81594f0 0x7f30a80cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:08.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.029+0000 7fa06dd53640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0680cd970 0x7fa06806c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.029+0000 7fa06dd53640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa068146a50 0x7fa068203690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.029+0000 7fa0677fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0680b2ba0 0x7fa06806bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.029+0000 7fa0677fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0680b2ba0 0x7fa06806bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49448/0 (socket says 172.21.15.116:49448) 2023-11-28T17:09:08.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.030+0000 7f30a7fff640 1 --2- 172.21.15.116:0/767561753 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a80cb650 0x7f30a80703f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f30a000d370 tx=0x7f30a002e250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13951d51771b83cc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.029+0000 7f62877fe640 1 -- 172.21.15.116:0/193447209 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6290011b00 con 0x7f62a00b20a0 2023-11-28T17:09:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.030+0000 7fa066ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0680cd970 0x7fa06806c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.030+0000 7fa066ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0680cd970 0x7fa06806c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:58934/0 (socket says 172.21.15.116:58934) 2023-11-28T17:09:08.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.030+0000 7f30a67fc640 1 -- 172.21.15.116:0/767561753 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a0038400 con 0x7f30a80cb650 2023-11-28T17:09:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.030+0000 7fa06dd53640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa068204400 con 0x7fa068146a50 2023-11-28T17:09:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.030+0000 7fa06dd53640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0682009b0 con 0x7fa0680cd970 2023-11-28T17:09:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.030+0000 7fa06dd53640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0681ffe20 con 0x7fa0680b2ba0 2023-11-28T17:09:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.030+0000 7fa0677fe640 1 -- 172.21.15.116:0/1558014008 learned_addr learned my addr 172.21.15.116:0/1558014008 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.031+0000 7fa0677fe640 1 -- 172.21.15.116:0/1558014008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0680cd970 msgr2=0x7fa06806c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.031+0000 7fa0677fe640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0680cd970 0x7fa06806c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.031+0000 7fa0677fe640 1 -- 172.21.15.116:0/1558014008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa068146a50 msgr2=0x7fa068203690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.031+0000 7fa067fff640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa068146a50 0x7fa068203690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.031+0000 7fa0677fe640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa068146a50 0x7fa068203690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.031+0000 7fa0677fe640 1 -- 172.21.15.116:0/1558014008 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0600094d0 con 0x7fa0680b2ba0 2023-11-28T17:09:08.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.031+0000 7fa066ffd640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0680cd970 0x7fa06806c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2023-11-28T17:09:08.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.031+0000 7fa067fff640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa068146a50 0x7fa068203690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:08.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.030+0000 7f30a67fc640 1 -- 172.21.15.116:0/767561753 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a0032070 con 0x7f30a80cb650 2023-11-28T17:09:08.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.030+0000 7f30a67fc640 1 -- 172.21.15.116:0/767561753 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a0038780 con 0x7f30a80cb650 2023-11-28T17:09:08.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.031+0000 7f30adec6640 1 -- 172.21.15.116:0/767561753 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a80cb650 msgr2=0x7f30a80703f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.031+0000 7f30adec6640 1 --2- 172.21.15.116:0/767561753 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a80cb650 0x7f30a80703f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f30a000d370 tx=0x7f30a002e250 comp rx=0 tx=0).stop 2023-11-28T17:09:08.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.031+0000 7f30adec6640 1 -- 172.21.15.116:0/767561753 shutdown_connections 2023-11-28T17:09:08.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.031+0000 7f30adec6640 1 --2- 172.21.15.116:0/767561753 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a80cb650 0x7f30a80703f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.031+0000 7f30adec6640 1 --2- 172.21.15.116:0/767561753 >> v2:172.21.15.116:3300/0 conn(0x7f30a81594f0 0x7f30a80cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.031+0000 7f30adec6640 1 -- 172.21.15.116:0/767561753 >> 172.21.15.116:0/767561753 conn(0x7f30a806c250 msgr2=0x7f30a806c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.032+0000 7fa0677fe640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0680b2ba0 0x7fa06806bdc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa06002b8c0 tx=0x7fa0600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.032+0000 7f30adec6640 1 -- 172.21.15.116:0/767561753 shutdown_connections 2023-11-28T17:09:08.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.032+0000 7f30adec6640 1 -- 172.21.15.116:0/767561753 wait complete. 2023-11-28T17:09:08.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.032+0000 7fa05ffff640 1 -- 172.21.15.116:0/1558014008 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060004450 con 0x7fa0680b2ba0 2023-11-28T17:09:08.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.032+0000 7fa05ffff640 1 -- 172.21.15.116:0/1558014008 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06003f070 con 0x7fa0680b2ba0 2023-11-28T17:09:08.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.032+0000 7fa05ffff640 1 -- 172.21.15.116:0/1558014008 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060003850 con 0x7fa0680b2ba0 2023-11-28T17:09:08.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.032+0000 7fa06dd53640 1 -- 172.21.15.116:0/1558014008 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa068200e40 con 0x7fa0680b2ba0 2023-11-28T17:09:08.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.032+0000 7fa06dd53640 1 -- 172.21.15.116:0/1558014008 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa068200360 con 0x7fa0680b2ba0 2023-11-28T17:09:08.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.033+0000 7efcb1ffb640 1 -- 172.21.15.116:0/1241642622 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7efca40147e0 con 0x7efcb40b3850 2023-11-28T17:09:08.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.033+0000 7f62877fe640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f628c06b300 0x7f628c06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.034+0000 7fa06dd53640 1 -- 172.21.15.116:0/1558014008 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa068146e30 con 0x7fa0680b2ba0 2023-11-28T17:09:08.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.034+0000 7f62877fe640 1 -- 172.21.15.116:0/193447209 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6290089590 con 0x7f62a00b20a0 2023-11-28T17:09:08.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.034+0000 7f629e7fc640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f628c06b300 0x7f628c06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.034+0000 7f62877fe640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.156:6808/526051277,v1:172.21.15.156:6809/526051277] conn(0x7f628c073520 0x7f628c075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.034+0000 7f62877fe640 1 -- 172.21.15.116:0/193447209 --> [v2:172.21.15.156:6808/526051277,v1:172.21.15.156:6809/526051277] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f628c075ff0 con 0x7f628c073520 2023-11-28T17:09:08.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.034+0000 7f62877fe640 1 -- 172.21.15.116:0/193447209 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f629008bbb0 con 0x7f62a00b20a0 2023-11-28T17:09:08.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.034+0000 7efcb1ffb640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7efc9406b2f0 0x7efc9406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.035+0000 7efcb1ffb640 1 -- 172.21.15.116:0/1241642622 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efca408a960 con 0x7efcb40b3850 2023-11-28T17:09:08.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.035+0000 7efcabfff640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7efc9406b2f0 0x7efc9406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.035+0000 7efcb1ffb640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.156:6800/467190845,v1:172.21.15.156:6801/467190845] conn(0x7efc94073510 0x7efc94075930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.035+0000 7efcb1ffb640 1 -- 172.21.15.116:0/1241642622 --> [v2:172.21.15.156:6800/467190845,v1:172.21.15.156:6801/467190845] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc94075fe0 con 0x7efc94073510 2023-11-28T17:09:08.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.035+0000 7efcb1ffb640 1 -- 172.21.15.116:0/1241642622 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7efca408f2c0 con 0x7efcb40b3850 2023-11-28T17:09:08.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.035+0000 7f629e7fc640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f628c06b300 0x7f628c06d7c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6294009870 tx=0x7f629403d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.035+0000 7f629f7fe640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.156:6808/526051277,v1:172.21.15.156:6809/526051277] conn(0x7f628c073520 0x7f628c075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.036+0000 7f30adec6640 1 Processor -- start 2023-11-28T17:09:08.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.036+0000 7efcb902d640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.156:6800/467190845,v1:172.21.15.156:6801/467190845] conn(0x7efc94073510 0x7efc94075930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.036+0000 7efcabfff640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7efc9406b2f0 0x7efc9406d7b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009370 tx=0x7efc9c009300 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.036+0000 7f629f7fe640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.156:6808/526051277,v1:172.21.15.156:6809/526051277] conn(0x7f628c073520 0x7f628c075940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.036+0000 7fa05ffff640 1 -- 172.21.15.116:0/1558014008 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa0600039b0 con 0x7fa0680b2ba0 2023-11-28T17:09:08.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.037+0000 7f30adec6640 1 -- start start 2023-11-28T17:09:08.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.037+0000 7efcb902d640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.156:6800/467190845,v1:172.21.15.156:6801/467190845] conn(0x7efc94073510 0x7efc94075930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.037+0000 7fa05ffff640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa04806b2a0 0x7fa04806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.037+0000 7fa05ffff640 1 -- 172.21.15.116:0/1558014008 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0600ab960 con 0x7fa0680b2ba0 2023-11-28T17:09:08.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.037+0000 7fa05ffff640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.156:6824/4082913363,v1:172.21.15.156:6825/4082913363] conn(0x7fa048073440 0x7fa048075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.037+0000 7fa05ffff640 1 -- 172.21.15.116:0/1558014008 --> [v2:172.21.15.156:6824/4082913363,v1:172.21.15.156:6825/4082913363] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa048075f10 con 0x7fa048073440 2023-11-28T17:09:08.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.037+0000 7fa05ffff640 1 -- 172.21.15.116:0/1558014008 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0600abdb0 con 0x7fa0680b2ba0 2023-11-28T17:09:08.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.038+0000 7fa066ffd640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa04806b2a0 0x7fa04806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.038+0000 7fa067fff640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.156:6824/4082913363,v1:172.21.15.156:6825/4082913363] conn(0x7fa048073440 0x7fa048075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.038+0000 7fa066ffd640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa04806b2a0 0x7fa04806d760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa06813e820 tx=0x7fa050008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.039+0000 7f30adec6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a80b3850 0x7f30a813cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.039+0000 7f30adec6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30a80cb650 0x7f30a80b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.039+0000 7f30adec6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30a80b3040 0x7f30a815acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.039+0000 7f30adec6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a80b34f0 con 0x7f30a80cb650 2023-11-28T17:09:08.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.039+0000 7f30adec6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a815ca90 con 0x7f30a80b3040 2023-11-28T17:09:08.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.039+0000 7f30adec6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a815b9f0 con 0x7f30a80b3850 2023-11-28T17:09:08.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.039+0000 7efcb1ffb640 1 -- 172.21.15.116:0/1241642622 <== osd.4 v2:172.21.15.156:6800/467190845 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7efc94075fe0 con 0x7efc94073510 2023-11-28T17:09:08.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.039+0000 7f62877fe640 1 -- 172.21.15.116:0/193447209 <== osd.5 v2:172.21.15.156:6808/526051277 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f628c075ff0 con 0x7f628c073520 2023-11-28T17:09:08.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.039+0000 7f30a77fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a80b3850 0x7f30a813cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.040+0000 7fa067fff640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.156:6824/4082913363,v1:172.21.15.156:6825/4082913363] conn(0x7fa048073440 0x7fa048075860 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.039+0000 7f30a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30a80cb650 0x7f30a80b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.040+0000 7f30a7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30a80b3040 0x7f30a815acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.040+0000 7f30a7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30a80b3040 0x7f30a815acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:58938/0 (socket says 172.21.15.116:58938) 2023-11-28T17:09:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.040+0000 7f30a7fff640 1 -- 172.21.15.116:0/2365829064 learned_addr learned my addr 172.21.15.116:0/2365829064 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.040+0000 7f30a7fff640 1 -- 172.21.15.116:0/2365829064 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a80b3850 msgr2=0x7f30a813cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.040+0000 7f30a7fff640 1 --2- 172.21.15.116:0/2365829064 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30a80b3850 0x7f30a813cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.040+0000 7f30a7fff640 1 -- 172.21.15.116:0/2365829064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30a80cb650 msgr2=0x7f30a80b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.040+0000 7f30a7fff640 1 --2- 172.21.15.116:0/2365829064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30a80cb650 0x7f30a80b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.040+0000 7f30a7fff640 1 -- 172.21.15.116:0/2365829064 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a000d050 con 0x7f30a80b3040 2023-11-28T17:09:08.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.040+0000 7f30a6ffd640 1 --2- 172.21.15.116:0/2365829064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30a80cb650 0x7f30a80b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:09:08.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.041+0000 7f30a7fff640 1 --2- 172.21.15.116:0/2365829064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30a80b3040 0x7f30a815acc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f30a0008b40 tx=0x7f30a0004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.041+0000 7f59a5cda640 1 -- 172.21.15.116:0/1228333717 --> [v2:172.21.15.116:6808/1361195238,v1:172.21.15.116:6809/1361195238] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f59a0151ad0 con 0x7f598c073510 2023-11-28T17:09:08.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.042+0000 7fa05ffff640 1 -- 172.21.15.116:0/1558014008 <== osd.6 v2:172.21.15.156:6824/4082913363 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa048075f10 con 0x7fa048073440 2023-11-28T17:09:08.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.042+0000 7f5987fff640 1 -- 172.21.15.116:0/1228333717 <== osd.2 v2:172.21.15.116:6808/1361195238 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f59a0151ad0 con 0x7f598c073510 2023-11-28T17:09:08.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.044+0000 7f59a5cda640 1 -- 172.21.15.116:0/1228333717 >> [v2:172.21.15.116:6808/1361195238,v1:172.21.15.116:6809/1361195238] conn(0x7f598c073510 msgr2=0x7f598c075930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.044+0000 7f59a5cda640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.116:6808/1361195238,v1:172.21.15.116:6809/1361195238] conn(0x7f598c073510 0x7f598c075930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.044+0000 7f59a5cda640 1 -- 172.21.15.116:0/1228333717 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f598c06b2f0 msgr2=0x7f598c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.044+0000 7f59a5cda640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f598c06b2f0 0x7f598c06d7b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5990005880 tx=0x7f5990009040 comp rx=0 tx=0).stop 2023-11-28T17:09:08.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.044+0000 7f59a5cda640 1 -- 172.21.15.116:0/1228333717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59a0150310 msgr2=0x7f59a013f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.044+0000 7f59a5cda640 1 --2- 172.21.15.116:0/1228333717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59a0150310 0x7f59a013f520 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5998032000 tx=0x7f5998004a40 comp rx=0 tx=0).stop 2023-11-28T17:09:08.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.041+0000 7f308ffff640 1 -- 172.21.15.116:0/2365829064 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a0046070 con 0x7f30a80b3040 2023-11-28T17:09:08.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.042+0000 7f308ffff640 1 -- 172.21.15.116:0/2365829064 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a000be90 con 0x7f30a80b3040 2023-11-28T17:09:08.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.042+0000 7f308ffff640 1 -- 172.21.15.116:0/2365829064 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a0041660 con 0x7f30a80b3040 2023-11-28T17:09:08.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.042+0000 7f30adec6640 1 -- 172.21.15.116:0/2365829064 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a815bcd0 con 0x7f30a80b3040 2023-11-28T17:09:08.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.042+0000 7f30adec6640 1 -- 172.21.15.116:0/2365829064 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a815c290 con 0x7f30a80b3040 2023-11-28T17:09:08.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.044+0000 7f599ffff640 1 -- 172.21.15.116:0/1228333717 reap_dead start 2023-11-28T17:09:08.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.045+0000 7f30adec6640 1 -- 172.21.15.116:0/2365829064 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f30a80cec50 con 0x7f30a80b3040 2023-11-28T17:09:08.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.045+0000 7f59a5cda640 1 -- 172.21.15.116:0/1228333717 shutdown_connections 2023-11-28T17:09:08.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.045+0000 7f59a5cda640 1 -- 172.21.15.116:0/1228333717 >> 172.21.15.116:0/1228333717 conn(0x7f59a00aea80 msgr2=0x7f59a006d480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.045+0000 7f59a5cda640 1 -- 172.21.15.116:0/1228333717 shutdown_connections 2023-11-28T17:09:08.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.045+0000 7f59a5cda640 1 -- 172.21.15.116:0/1228333717 wait complete. 2023-11-28T17:09:08.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.047+0000 7f98754c6640 1 -- 172.21.15.116:0/1732226841 --> [v2:172.21.15.116:6824/335787082,v1:172.21.15.116:6825/335787082] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9854002cb0 con 0x7f98500734c0 2023-11-28T17:09:08.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.049+0000 7f308ffff640 1 -- 172.21.15.116:0/2365829064 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f30a00417c0 con 0x7f30a80b3040 2023-11-28T17:09:08.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.049+0000 7f98677fe640 1 -- 172.21.15.116:0/1732226841 <== osd.1 v2:172.21.15.116:6824/335787082 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9854002cb0 con 0x7f98500734c0 2023-11-28T17:09:08.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.050+0000 7f98754c6640 1 -- 172.21.15.116:0/1732226841 >> [v2:172.21.15.116:6824/335787082,v1:172.21.15.116:6825/335787082] conn(0x7f98500734c0 msgr2=0x7f98500758e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.050+0000 7f98754c6640 1 --2- 172.21.15.116:0/1732226841 >> [v2:172.21.15.116:6824/335787082,v1:172.21.15.116:6825/335787082] conn(0x7f98500734c0 0x7f98500758e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.050+0000 7f77a1505640 1 -- 172.21.15.116:0/344139607 --> [v2:172.21.15.116:6800/671929056,v1:172.21.15.116:6801/671929056] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7794052cd0 con 0x7f7788073440 2023-11-28T17:09:08.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.050+0000 7f98754c6640 1 -- 172.21.15.116:0/1732226841 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f985006b2a0 msgr2=0x7f985006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.050+0000 7f98754c6640 1 --2- 172.21.15.116:0/1732226841 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f985006b2a0 0x7f985006d760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9858036000 tx=0x7f985803c040 comp rx=0 tx=0).stop 2023-11-28T17:09:08.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.050+0000 7f98754c6640 1 -- 172.21.15.116:0/1732226841 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9870143820 msgr2=0x7f98701353e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.050+0000 7f98754c6640 1 --2- 172.21.15.116:0/1732226841 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9870143820 0x7f98701353e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f986800ce40 tx=0x7f986800a710 comp rx=0 tx=0).stop 2023-11-28T17:09:08.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.050+0000 7f986f7fe640 1 -- 172.21.15.116:0/1732226841 reap_dead start 2023-11-28T17:09:08.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.050+0000 7f308ffff640 1 --2- 172.21.15.116:0/2365829064 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f309406b2f0 0x7f309406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.051+0000 7f308ffff640 1 -- 172.21.15.116:0/2365829064 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30a00ac910 con 0x7f30a80b3040 2023-11-28T17:09:08.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.051+0000 7f308ffff640 1 --2- 172.21.15.116:0/2365829064 >> [v2:172.21.15.156:6815/1619141236,v1:172.21.15.156:6817/1619141236] conn(0x7f3094073510 0x7f3094075930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.051+0000 7f308ffff640 1 -- 172.21.15.116:0/2365829064 --> [v2:172.21.15.156:6815/1619141236,v1:172.21.15.156:6817/1619141236] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3094075fe0 con 0x7f3094073510 2023-11-28T17:09:08.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.051+0000 7f308ffff640 1 -- 172.21.15.116:0/2365829064 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30a00acc80 con 0x7f30a80b3040 2023-11-28T17:09:08.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.051+0000 7f98754c6640 1 -- 172.21.15.116:0/1732226841 shutdown_connections 2023-11-28T17:09:08.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.051+0000 7f98754c6640 1 -- 172.21.15.116:0/1732226841 >> 172.21.15.116:0/1732226841 conn(0x7f987006c4b0 msgr2=0x7f987013a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.051+0000 7f77837fe640 1 -- 172.21.15.116:0/344139607 <== osd.0 v2:172.21.15.116:6800/671929056 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f779c04f1b0 con 0x7f7788073440 2023-11-28T17:09:08.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.051+0000 7f30a77fe640 1 --2- 172.21.15.116:0/2365829064 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f309406b2f0 0x7f309406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.052+0000 7f30a6ffd640 1 --2- 172.21.15.116:0/2365829064 >> [v2:172.21.15.156:6815/1619141236,v1:172.21.15.156:6817/1619141236] conn(0x7f3094073510 0x7f3094075930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.053+0000 7f98754c6640 1 -- 172.21.15.116:0/1732226841 shutdown_connections 2023-11-28T17:09:08.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.053+0000 7f98754c6640 1 -- 172.21.15.116:0/1732226841 wait complete. 2023-11-28T17:09:08.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.053+0000 7f30a6ffd640 1 --2- 172.21.15.116:0/2365829064 >> [v2:172.21.15.156:6815/1619141236,v1:172.21.15.156:6817/1619141236] conn(0x7f3094073510 0x7f3094075930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.053+0000 7f77a1505640 1 -- 172.21.15.116:0/344139607 >> [v2:172.21.15.116:6800/671929056,v1:172.21.15.116:6801/671929056] conn(0x7f7788073440 msgr2=0x7f7788075860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.053+0000 7f77a1505640 1 --2- 172.21.15.116:0/344139607 >> [v2:172.21.15.116:6800/671929056,v1:172.21.15.116:6801/671929056] conn(0x7f7788073440 0x7f7788075860 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.053+0000 7f30a77fe640 1 --2- 172.21.15.116:0/2365829064 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f309406b2f0 0x7f309406d7b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f309c000f60 tx=0x7f309c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.054+0000 7f77a1505640 1 -- 172.21.15.116:0/344139607 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f778806b2a0 msgr2=0x7f778806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.054+0000 7f77a1505640 1 --2- 172.21.15.116:0/344139607 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f778806b2a0 0x7f778806d760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f778c000f60 tx=0x7f778c007f00 comp rx=0 tx=0).stop 2023-11-28T17:09:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.054+0000 7f77a1505640 1 -- 172.21.15.116:0/344139607 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7794054310 msgr2=0x7f77940e1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.054+0000 7f77a1505640 1 --2- 172.21.15.116:0/344139607 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7794054310 0x7f77940e1750 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7790033000 tx=0x7f77900046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.054+0000 7f779b7fe640 1 -- 172.21.15.116:0/344139607 reap_dead start 2023-11-28T17:09:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.054+0000 7f77a1505640 1 -- 172.21.15.116:0/344139607 shutdown_connections 2023-11-28T17:09:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.054+0000 7f77a1505640 1 -- 172.21.15.116:0/344139607 >> 172.21.15.116:0/344139607 conn(0x7f7794009f70 msgr2=0x7f77940506b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.056+0000 7f77a1505640 1 -- 172.21.15.116:0/344139607 shutdown_connections 2023-11-28T17:09:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.056+0000 7f77a1505640 1 -- 172.21.15.116:0/344139607 wait complete. 2023-11-28T17:09:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.055+0000 7f308ffff640 1 -- 172.21.15.116:0/2365829064 <== osd.7 v2:172.21.15.156:6815/1619141236 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f3094075fe0 con 0x7f3094073510 2023-11-28T17:09:08.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.060+0000 7f62a5436640 1 -- 172.21.15.116:0/193447209 --> [v2:172.21.15.156:6808/526051277,v1:172.21.15.156:6809/526051277] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6280002cb0 con 0x7f628c073520 2023-11-28T17:09:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.064+0000 7f62877fe640 1 -- 172.21.15.116:0/193447209 <== osd.5 v2:172.21.15.156:6808/526051277 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6280002cb0 con 0x7f628c073520 2023-11-28T17:09:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.064+0000 7f62a5436640 1 -- 172.21.15.116:0/193447209 >> [v2:172.21.15.156:6808/526051277,v1:172.21.15.156:6809/526051277] conn(0x7f628c073520 msgr2=0x7f628c075940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.064+0000 7f62a5436640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.156:6808/526051277,v1:172.21.15.156:6809/526051277] conn(0x7f628c073520 0x7f628c075940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.065+0000 7f62a5436640 1 -- 172.21.15.116:0/193447209 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f628c06b300 msgr2=0x7f628c06d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.065+0000 7f62a5436640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f628c06b300 0x7f628c06d7c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6294009870 tx=0x7f629403d040 comp rx=0 tx=0).stop 2023-11-28T17:09:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.065+0000 7f62a5436640 1 -- 172.21.15.116:0/193447209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62a00b20a0 msgr2=0x7f62a00c8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.065+0000 7f62a5436640 1 --2- 172.21.15.116:0/193447209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62a00b20a0 0x7f62a00c8e20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f629000b3e0 tx=0x7f629000b8a0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.065+0000 7f629f7fe640 1 -- 172.21.15.116:0/193447209 reap_dead start 2023-11-28T17:09:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.066+0000 7f62a5436640 1 -- 172.21.15.116:0/193447209 shutdown_connections 2023-11-28T17:09:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.066+0000 7f62a5436640 1 -- 172.21.15.116:0/193447209 >> 172.21.15.116:0/193447209 conn(0x7f62a009c4e0 msgr2=0x7f62a00d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.066+0000 7f62a5436640 1 -- 172.21.15.116:0/193447209 shutdown_connections 2023-11-28T17:09:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.066+0000 7f62a5436640 1 -- 172.21.15.116:0/193447209 wait complete. 2023-11-28T17:09:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.067+0000 7fabc97c4640 1 -- 172.21.15.116:0/3017224462 --> [v2:172.21.15.116:6811/3907008760,v1:172.21.15.116:6814/3907008760] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fabc40d1240 con 0x7fabb0074120 2023-11-28T17:09:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.068+0000 7fabab7fe640 1 -- 172.21.15.116:0/3017224462 <== osd.3 v2:172.21.15.116:6811/3907008760 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fabb4031b50 con 0x7fabb0074120 2023-11-28T17:09:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.069+0000 7fabc97c4640 1 -- 172.21.15.116:0/3017224462 >> [v2:172.21.15.116:6811/3907008760,v1:172.21.15.116:6814/3907008760] conn(0x7fabb0074120 msgr2=0x7fabb00765c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.069+0000 7fabc97c4640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.116:6811/3907008760,v1:172.21.15.116:6814/3907008760] conn(0x7fabb0074120 0x7fabb00765c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.069+0000 7fabc97c4640 1 -- 172.21.15.116:0/3017224462 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fabb006b2a0 msgr2=0x7fabb006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.070+0000 7fabc97c4640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fabb006b2a0 0x7fabb006d760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fabb8006fd0 tx=0x7fabb8008040 comp rx=0 tx=0).stop 2023-11-28T17:09:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.070+0000 7fabc97c4640 1 -- 172.21.15.116:0/3017224462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fabc40b20a0 msgr2=0x7fabc4158540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.070+0000 7fabc97c4640 1 --2- 172.21.15.116:0/3017224462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fabc40b20a0 0x7fabc4158540 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00be20 tx=0x7fabbc009f70 comp rx=0 tx=0).stop 2023-11-28T17:09:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.070+0000 7fabc37fe640 1 -- 172.21.15.116:0/3017224462 reap_dead start 2023-11-28T17:09:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.070+0000 7fabc97c4640 1 -- 172.21.15.116:0/3017224462 shutdown_connections 2023-11-28T17:09:08.098 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-28T17:09:08.099 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-28T17:09:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.070+0000 7fabc97c4640 1 -- 172.21.15.116:0/3017224462 >> 172.21.15.116:0/3017224462 conn(0x7fabc409c4e0 msgr2=0x7fabc40cfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.071+0000 7fabc97c4640 1 -- 172.21.15.116:0/3017224462 shutdown_connections 2023-11-28T17:09:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.071+0000 7fabc97c4640 1 -- 172.21.15.116:0/3017224462 wait complete. 2023-11-28T17:09:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.072+0000 7fa06dd53640 1 -- 172.21.15.116:0/1558014008 --> [v2:172.21.15.156:6824/4082913363,v1:172.21.15.156:6825/4082913363] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa068161560 con 0x7fa048073440 2023-11-28T17:09:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.073+0000 7efcbaa90640 1 -- 172.21.15.116:0/1241642622 --> [v2:172.21.15.156:6800/467190845,v1:172.21.15.156:6801/467190845] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efcb40b2400 con 0x7efc94073510 2023-11-28T17:09:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.073+0000 7fa05ffff640 1 -- 172.21.15.116:0/1558014008 <== osd.6 v2:172.21.15.156:6824/4082913363 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa0580019d0 con 0x7fa048073440 2023-11-28T17:09:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.074+0000 7fa06dd53640 1 -- 172.21.15.116:0/1558014008 >> [v2:172.21.15.156:6824/4082913363,v1:172.21.15.156:6825/4082913363] conn(0x7fa048073440 msgr2=0x7fa048075860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.074+0000 7fa06dd53640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.156:6824/4082913363,v1:172.21.15.156:6825/4082913363] conn(0x7fa048073440 0x7fa048075860 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.074+0000 7fa06dd53640 1 -- 172.21.15.116:0/1558014008 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa04806b2a0 msgr2=0x7fa04806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.074+0000 7fa06dd53640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa04806b2a0 0x7fa04806d760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa06813e820 tx=0x7fa050008040 comp rx=0 tx=0).stop 2023-11-28T17:09:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.074+0000 7fa06dd53640 1 -- 172.21.15.116:0/1558014008 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0680b2ba0 msgr2=0x7fa06806bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.074+0000 7fa06dd53640 1 --2- 172.21.15.116:0/1558014008 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0680b2ba0 0x7fa06806bdc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa06002b8c0 tx=0x7fa0600046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.075+0000 7efcb1ffb640 1 -- 172.21.15.116:0/1241642622 <== osd.4 v2:172.21.15.156:6800/467190845 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efcac0316d0 con 0x7efc94073510 2023-11-28T17:09:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.075+0000 7fa067fff640 1 -- 172.21.15.116:0/1558014008 reap_dead start 2023-11-28T17:09:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.075+0000 7fa06dd53640 1 -- 172.21.15.116:0/1558014008 shutdown_connections 2023-11-28T17:09:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.075+0000 7fa06dd53640 1 -- 172.21.15.116:0/1558014008 >> 172.21.15.116:0/1558014008 conn(0x7fa06806a4c0 msgr2=0x7fa0680cbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.075+0000 7fa06dd53640 1 -- 172.21.15.116:0/1558014008 shutdown_connections 2023-11-28T17:09:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.075+0000 7fa06dd53640 1 -- 172.21.15.116:0/1558014008 wait complete. 2023-11-28T17:09:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.076+0000 7efcbaa90640 1 -- 172.21.15.116:0/1241642622 >> [v2:172.21.15.156:6800/467190845,v1:172.21.15.156:6801/467190845] conn(0x7efc94073510 msgr2=0x7efc94075930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.076+0000 7efcbaa90640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.156:6800/467190845,v1:172.21.15.156:6801/467190845] conn(0x7efc94073510 0x7efc94075930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.076+0000 7efcbaa90640 1 -- 172.21.15.116:0/1241642622 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7efc9406b2f0 msgr2=0x7efc9406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.104 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-28T17:09:08.105 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-28T17:09:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.076+0000 7efcbaa90640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7efc9406b2f0 0x7efc9406d7b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009370 tx=0x7efc9c009300 comp rx=0 tx=0).stop 2023-11-28T17:09:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.076+0000 7efcbaa90640 1 -- 172.21.15.116:0/1241642622 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcb40b3850 msgr2=0x7efcb41510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.076+0000 7efcbaa90640 1 --2- 172.21.15.116:0/1241642622 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcb40b3850 0x7efcb41510c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efca400efc0 tx=0x7efca400c490 comp rx=0 tx=0).stop 2023-11-28T17:09:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.076+0000 7efcb902d640 1 -- 172.21.15.116:0/1241642622 reap_dead start 2023-11-28T17:09:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.077+0000 7efcbaa90640 1 -- 172.21.15.116:0/1241642622 shutdown_connections 2023-11-28T17:09:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.077+0000 7efcbaa90640 1 -- 172.21.15.116:0/1241642622 >> 172.21.15.116:0/1241642622 conn(0x7efcb406c250 msgr2=0x7efcb4164b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.077+0000 7efcbaa90640 1 -- 172.21.15.116:0/1241642622 shutdown_connections 2023-11-28T17:09:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.077+0000 7efcbaa90640 1 -- 172.21.15.116:0/1241642622 wait complete. 2023-11-28T17:09:08.107 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-28T17:09:08.107 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-28T17:09:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.087+0000 7f30adec6640 1 -- 172.21.15.116:0/2365829064 --> [v2:172.21.15.156:6815/1619141236,v1:172.21.15.156:6817/1619141236] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f30a80693d0 con 0x7f3094073510 2023-11-28T17:09:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.088+0000 7f308ffff640 1 -- 172.21.15.116:0/2365829064 <== osd.7 v2:172.21.15.156:6815/1619141236 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f30a80693d0 con 0x7f3094073510 2023-11-28T17:09:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.089+0000 7f30adec6640 1 -- 172.21.15.116:0/2365829064 >> [v2:172.21.15.156:6815/1619141236,v1:172.21.15.156:6817/1619141236] conn(0x7f3094073510 msgr2=0x7f3094075930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.089+0000 7f30adec6640 1 --2- 172.21.15.116:0/2365829064 >> [v2:172.21.15.156:6815/1619141236,v1:172.21.15.156:6817/1619141236] conn(0x7f3094073510 0x7f3094075930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.089+0000 7f30adec6640 1 -- 172.21.15.116:0/2365829064 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f309406b2f0 msgr2=0x7f309406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.089+0000 7f30adec6640 1 --2- 172.21.15.116:0/2365829064 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f309406b2f0 0x7f309406d7b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f309c000f60 tx=0x7f309c009040 comp rx=0 tx=0).stop 2023-11-28T17:09:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.089+0000 7f30adec6640 1 -- 172.21.15.116:0/2365829064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30a80b3040 msgr2=0x7f30a815acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.089+0000 7f30adec6640 1 --2- 172.21.15.116:0/2365829064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30a80b3040 0x7f30a815acc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f30a0008b40 tx=0x7f30a0004ab0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.089+0000 7f30a7fff640 1 -- 172.21.15.116:0/2365829064 reap_dead start 2023-11-28T17:09:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.089+0000 7f30adec6640 1 -- 172.21.15.116:0/2365829064 shutdown_connections 2023-11-28T17:09:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.089+0000 7f30adec6640 1 -- 172.21.15.116:0/2365829064 >> 172.21.15.116:0/2365829064 conn(0x7f30a806c250 msgr2=0x7f30a8150b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.089+0000 7f30adec6640 1 -- 172.21.15.116:0/2365829064 shutdown_connections 2023-11-28T17:09:08.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.089+0000 7f30adec6640 1 -- 172.21.15.116:0/2365829064 wait complete. 2023-11-28T17:09:08.110 INFO:teuthology.orchestra.run.smithi116.stdout:55834574855 2023-11-28T17:09:08.110 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-28T17:09:08.111 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-28T17:09:08.112 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-28T17:09:08.112 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-28T17:09:08.113 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-28T17:09:08.113 INFO:teuthology.orchestra.run.smithi116.stdout:55834574855 2023-11-28T17:09:08.114 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-28T17:09:08.114 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-28T17:09:08.115 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-28T17:09:08.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.443+0000 7fe925a09640 1 Processor -- start 2023-11-28T17:09:08.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.444+0000 7fe925a09640 1 -- start start 2023-11-28T17:09:08.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.445+0000 7fe925a09640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9180e77e0 0x7fe918038240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.445+0000 7fe925a09640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe9180eb2c0 0x7fe9180eb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.446+0000 7fe925a09640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9180e7bc0 con 0x7fe9180ebc80 2023-11-28T17:09:08.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.446+0000 7fe925a09640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe918057ee0 con 0x7fe9180eb2c0 2023-11-28T17:09:08.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.446+0000 7fe925a09640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9180581c0 con 0x7fe9180e77e0 2023-11-28T17:09:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.447+0000 7fe91f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe9180ebc80 0x7fe9180534e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54606/0 (socket says 172.21.15.116:54606) 2023-11-28T17:09:08.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.447+0000 7fe91f7fe640 1 -- 172.21.15.116:0/3647593720 learned_addr learned my addr 172.21.15.116:0/3647593720 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:08.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.447+0000 7fe91effd640 1 --2- 172.21.15.116:0/3647593720 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9180e77e0 0x7fe918038240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.448+0000 7fe91e7fc640 1 --2- 172.21.15.116:0/3647593720 >> v2:172.21.15.156:3300/0 conn(0x7fe9180eb2c0 0x7fe9180eb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.449+0000 7fe91dffb640 1 -- 172.21.15.116:0/3647593720 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500526532 0 0) 0x7fe9180e7bc0 con 0x7fe9180ebc80 2023-11-28T17:09:08.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.449+0000 7fe91dffb640 1 -- 172.21.15.116:0/3647593720 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe908003520 con 0x7fe9180ebc80 2023-11-28T17:09:08.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.450+0000 7fe91effd640 1 -- 172.21.15.116:0/3647593720 >> v2:172.21.15.156:3300/0 conn(0x7fe9180eb2c0 msgr2=0x7fe9180eb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.450+0000 7fe91effd640 1 --2- 172.21.15.116:0/3647593720 >> v2:172.21.15.156:3300/0 conn(0x7fe9180eb2c0 0x7fe9180eb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.450+0000 7fe91effd640 1 -- 172.21.15.116:0/3647593720 >> v1:172.21.15.116:6789/0 conn(0x7fe9180ebc80 legacy=0x7fe9180534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.450+0000 7fe91effd640 1 -- 172.21.15.116:0/3647593720 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9180584a0 con 0x7fe9180e77e0 2023-11-28T17:09:08.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.452+0000 7fe91e7fc640 1 --2- 172.21.15.116:0/3647593720 >> v2:172.21.15.156:3300/0 conn(0x7fe9180eb2c0 0x7fe9180eb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:09:08.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.452+0000 7fe91effd640 1 --2- 172.21.15.116:0/3647593720 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9180e77e0 0x7fe918038240 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe9200681e0 tx=0x7fe9200a5d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=423e117de8e59152 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.454+0000 7fe91dffb640 1 -- 172.21.15.116:0/3647593720 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9200a6bd0 con 0x7fe9180e77e0 2023-11-28T17:09:08.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.454+0000 7fe91dffb640 1 -- 172.21.15.116:0/3647593720 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9200a6d30 con 0x7fe9180e77e0 2023-11-28T17:09:08.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.455+0000 7fe91dffb640 1 -- 172.21.15.116:0/3647593720 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9200ae640 con 0x7fe9180e77e0 2023-11-28T17:09:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.456+0000 7fe925a09640 1 -- 172.21.15.116:0/3647593720 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9180e77e0 msgr2=0x7fe918038240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.456+0000 7fe925a09640 1 --2- 172.21.15.116:0/3647593720 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9180e77e0 0x7fe918038240 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe9200681e0 tx=0x7fe9200a5d60 comp rx=0 tx=0).stop 2023-11-28T17:09:08.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.456+0000 7f6cb078a640 1 Processor -- start 2023-11-28T17:09:08.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.456+0000 7fe925a09640 1 -- 172.21.15.116:0/3647593720 shutdown_connections 2023-11-28T17:09:08.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.456+0000 7fe925a09640 1 --2- 172.21.15.116:0/3647593720 >> v2:172.21.15.156:3300/0 conn(0x7fe9180eb2c0 0x7fe9180eb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.456+0000 7fe925a09640 1 --2- 172.21.15.116:0/3647593720 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9180e77e0 0x7fe918038240 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.456+0000 7fe925a09640 1 -- 172.21.15.116:0/3647593720 >> 172.21.15.116:0/3647593720 conn(0x7fe9180559a0 msgr2=0x7fe918055db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.456+0000 7f6cb078a640 1 -- start start 2023-11-28T17:09:08.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.456+0000 7fe925a09640 1 -- 172.21.15.116:0/3647593720 shutdown_connections 2023-11-28T17:09:08.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.456+0000 7fe925a09640 1 -- 172.21.15.116:0/3647593720 wait complete. 2023-11-28T17:09:08.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.457+0000 7fe11f59e640 1 Processor -- start 2023-11-28T17:09:08.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.457+0000 7fe11f59e640 1 -- start start 2023-11-28T17:09:08.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.458+0000 7f6cb078a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6ca00e3080 0x7f6ca0057d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.458+0000 7f6cb078a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca003e800 0x7f6ca0053bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.458+0000 7f6cb078a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6ca00540f0 0x7f6ca00e6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.458+0000 7f6cb078a640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca0054580 con 0x7f6ca00540f0 2023-11-28T17:09:08.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.458+0000 7f6cb078a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca0006160 con 0x7f6ca00e3080 2023-11-28T17:09:08.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.458+0000 7f6cb078a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca0058290 con 0x7f6ca003e800 2023-11-28T17:09:08.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7f6caed27640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6ca00540f0 0x7f6ca00e6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7f6caed27640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6ca00540f0 0x7f6ca00e6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51870/0 (socket says 172.21.15.116:51870) 2023-11-28T17:09:08.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7f6caed27640 1 -- 172.21.15.116:0/551782155 learned_addr learned my addr 172.21.15.116:0/551782155 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7fe925a09640 1 Processor -- start 2023-11-28T17:09:08.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7fe11f59e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe1180e76e0 0x7fe11804ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7fe11f59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe118053c10 0x7fe118006ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7fe11f59e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe118057d60 con 0x7fe118058990 2023-11-28T17:09:08.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7fe11f59e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe118006230 con 0x7fe118053c10 2023-11-28T17:09:08.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7fe11f59e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe118006510 con 0x7fe1180e76e0 2023-11-28T17:09:08.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7fe925a09640 1 -- start start 2023-11-28T17:09:08.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.460+0000 7f10c2708640 1 Processor -- start 2023-11-28T17:09:08.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.460+0000 7fe11dd9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe118058990 0x7fe118053500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54610/0 (socket says 172.21.15.116:54610) 2023-11-28T17:09:08.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.460+0000 7fe11dd9b640 1 -- 172.21.15.116:0/3511308425 learned_addr learned my addr 172.21.15.116:0/3511308425 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:08.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.461+0000 7fe925a09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9180e77e0 0x7fe918009c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.461+0000 7fe925a09640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9180eb2c0 0x7fe91800c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.461+0000 7fe925a09640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9180ebc80 0x7fe91800c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.461+0000 7fe11e59c640 1 --2- 172.21.15.116:0/3511308425 >> v2:172.21.15.156:3300/0 conn(0x7fe1180e76e0 0x7fe11804ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.461+0000 7fe925a09640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9180d2f10 con 0x7fe9180e77e0 2023-11-28T17:09:08.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.461+0000 7fe91effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9180e77e0 0x7fe918009c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.461+0000 7fe91e7fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9180eb2c0 0x7fe91800c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.462+0000 7fe925a09640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9180e3240 con 0x7fe9180ebc80 2023-11-28T17:09:08.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.462+0000 7fe925a09640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe918051fa0 con 0x7fe9180eb2c0 2023-11-28T17:09:08.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.462+0000 7fe91e7fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9180eb2c0 0x7fe91800c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49488/0 (socket says 172.21.15.116:49488) 2023-11-28T17:09:08.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.462+0000 7fe91e7fc640 1 -- 172.21.15.116:0/31693707 learned_addr learned my addr 172.21.15.116:0/31693707 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.462+0000 7fe11e59c640 1 -- 172.21.15.116:0/3511308425 >> v2:172.21.15.116:3300/0 conn(0x7fe118053c10 msgr2=0x7fe118006ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.462+0000 7fe11e59c640 1 --2- 172.21.15.116:0/3511308425 >> v2:172.21.15.116:3300/0 conn(0x7fe118053c10 0x7fe118006ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.462+0000 7fe11e59c640 1 -- 172.21.15.116:0/3511308425 >> v1:172.21.15.116:6789/0 conn(0x7fe118058990 legacy=0x7fe118053500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.462+0000 7fe11e59c640 1 -- 172.21.15.116:0/3511308425 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1180067f0 con 0x7fe1180e76e0 2023-11-28T17:09:08.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.462+0000 7fe91f7fe640 1 --2- 172.21.15.116:0/31693707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9180ebc80 0x7fe91800c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.461+0000 7f10c2708640 1 -- start start 2023-11-28T17:09:08.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.462+0000 7f10c2708640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f10bc0d1780 0x7f10bc0d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7f6cadd25640 1 --2- 172.21.15.116:0/551782155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca003e800 0x7f6ca0053bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7f6caed27640 1 -- 172.21.15.116:0/551782155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca003e800 msgr2=0x7f6ca0053bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7f6caed27640 1 --2- 172.21.15.116:0/551782155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca003e800 0x7f6ca0053bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7f6caed27640 1 -- 172.21.15.116:0/551782155 >> v2:172.21.15.156:3300/0 conn(0x7f6ca00e3080 msgr2=0x7f6ca0057d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7f6caed27640 1 --2- 172.21.15.116:0/551782155 >> v2:172.21.15.156:3300/0 conn(0x7f6ca00e3080 0x7f6ca0057d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.459+0000 7f6caed27640 1 -- 172.21.15.116:0/551782155 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca0058530 con 0x7f6ca00540f0 2023-11-28T17:09:08.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.462+0000 7f6caed27640 1 --2- 172.21.15.116:0/551782155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca00540f0 0x7f6ca00e6bf0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8068b70 tx=0x7f6ca80a5ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11eb15aa7a0ad6d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.463+0000 7f10c2708640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f10bc0b20a0 0x7f10bc0d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.463+0000 7fe91e7fc640 1 -- 172.21.15.116:0/31693707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9180ebc80 msgr2=0x7fe91800c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.463+0000 7fe91e7fc640 1 --2- 172.21.15.116:0/31693707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9180ebc80 0x7fe91800c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.463+0000 7fe91e7fc640 1 -- 172.21.15.116:0/31693707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9180e77e0 msgr2=0x7fe918009c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.463+0000 7f10c2708640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10bc0cdcb0 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.463+0000 7f10c2708640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10bc0d20a0 con 0x7f10bc0b20a0 2023-11-28T17:09:08.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.463+0000 7f10c2708640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0d39e0 con 0x7f10bc0d1780 2023-11-28T17:09:08.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.463+0000 7f10bb7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f10bc0b20a0 0x7f10bc0d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.463+0000 7f10bbfff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f10bc0d1780 0x7f10bc0d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.463+0000 7f10bbfff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f10bc0d1780 0x7f10bc0d01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49492/0 (socket says 172.21.15.116:49492) 2023-11-28T17:09:08.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.463+0000 7f10bbfff640 1 -- 172.21.15.116:0/1376858279 learned_addr learned my addr 172.21.15.116:0/1376858279 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.463+0000 7fe91e7fc640 1 --2- 172.21.15.116:0/31693707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9180e77e0 0x7fe918009c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.463+0000 7fe91e7fc640 1 -- 172.21.15.116:0/31693707 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe914009400 con 0x7fe9180eb2c0 2023-11-28T17:09:08.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.464+0000 7fe91effd640 1 --2- 172.21.15.116:0/31693707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9180e77e0 0x7fe918009c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:08.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.464+0000 7f10bb7fe640 1 -- 172.21.15.116:0/1376858279 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f10bc0d1780 msgr2=0x7f10bc0d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.464+0000 7f6cad524640 1 -- 172.21.15.116:0/551782155 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca80a6d50 con 0x7f6ca00540f0 2023-11-28T17:09:08.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.464+0000 7f6cad524640 1 -- 172.21.15.116:0/551782155 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca80ae440 con 0x7f6ca00540f0 2023-11-28T17:09:08.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.464+0000 7f6cad524640 1 -- 172.21.15.116:0/551782155 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca80ae870 con 0x7f6ca00540f0 2023-11-28T17:09:08.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.464+0000 7f10baffd640 1 -- 172.21.15.116:0/1376858279 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734340379 0 0) 0x7f10bc0cdcb0 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.464+0000 7f10bb7fe640 1 --2- 172.21.15.116:0/1376858279 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f10bc0d1780 0x7f10bc0d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.464+0000 7f10bb7fe640 1 -- 172.21.15.116:0/1376858279 >> v1:172.21.15.116:6789/0 conn(0x7f10bc0b2ad0 legacy=0x7f10bc0ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.464+0000 7f10bb7fe640 1 -- 172.21.15.116:0/1376858279 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10bc0d3cc0 con 0x7f10bc0b20a0 2023-11-28T17:09:08.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.464+0000 7fe11e59c640 1 --2- 172.21.15.116:0/3511308425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1180e76e0 0x7fe11804ca00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe11800f7c0 tx=0x7fe11402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f535a3f1d46a079d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.464+0000 7fe91e7fc640 1 --2- 172.21.15.116:0/31693707 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9180eb2c0 0x7fe91800c380 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe91400ebe0 tx=0x7fe91400c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.465+0000 7fe907fff640 1 -- 172.21.15.116:0/31693707 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914009760 con 0x7fe9180eb2c0 2023-11-28T17:09:08.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.465+0000 7fe925a09640 1 -- 172.21.15.116:0/31693707 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe920068870 con 0x7fe9180eb2c0 2023-11-28T17:09:08.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.465+0000 7fe925a09640 1 -- 172.21.15.116:0/31693707 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe918052560 con 0x7fe9180eb2c0 2023-11-28T17:09:08.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.465+0000 7fe11d59a640 1 -- 172.21.15.116:0/3511308425 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe114031640 con 0x7fe1180e76e0 2023-11-28T17:09:08.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.465+0000 7fe11d59a640 1 -- 172.21.15.116:0/3511308425 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1140317a0 con 0x7fe1180e76e0 2023-11-28T17:09:08.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.465+0000 7fe11d59a640 1 -- 172.21.15.116:0/3511308425 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe114031b30 con 0x7fe1180e76e0 2023-11-28T17:09:08.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7f6cb078a640 1 -- 172.21.15.116:0/551782155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca00540f0 msgr2=0x7f6ca00e6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7f6cb078a640 1 --2- 172.21.15.116:0/551782155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca00540f0 0x7f6ca00e6bf0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8068b70 tx=0x7f6ca80a5ee0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.465+0000 7f10bbfff640 1 --2- 172.21.15.116:0/1376858279 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f10bc0d1780 0x7f10bc0d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:09:08.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7f43e1f68640 1 Processor -- start 2023-11-28T17:09:08.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7f43e1f68640 1 -- start start 2023-11-28T17:09:08.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7f6cb078a640 1 -- 172.21.15.116:0/551782155 shutdown_connections 2023-11-28T17:09:08.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7f6cb078a640 1 --2- 172.21.15.116:0/551782155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca00540f0 0x7f6ca00e6bf0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7f6cb078a640 1 --2- 172.21.15.116:0/551782155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca003e800 0x7f6ca0053bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7f6cb078a640 1 --2- 172.21.15.116:0/551782155 >> v2:172.21.15.156:3300/0 conn(0x7f6ca00e3080 0x7f6ca0057d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7f6cb078a640 1 -- 172.21.15.116:0/551782155 >> 172.21.15.116:0/551782155 conn(0x7f6ca00559a0 msgr2=0x7f6ca0055db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7fe11f59e640 1 -- 172.21.15.116:0/3511308425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1180e76e0 msgr2=0x7fe11804ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7fe11f59e640 1 --2- 172.21.15.116:0/3511308425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1180e76e0 0x7fe11804ca00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe11800f7c0 tx=0x7fe11402b8f0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.465+0000 7fe907fff640 1 -- 172.21.15.116:0/31693707 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91400cc30 con 0x7fe9180eb2c0 2023-11-28T17:09:08.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.465+0000 7fe907fff640 1 -- 172.21.15.116:0/31693707 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91400eee0 con 0x7fe9180eb2c0 2023-11-28T17:09:08.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7fe11f59e640 1 -- 172.21.15.116:0/3511308425 shutdown_connections 2023-11-28T17:09:08.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7fe11f59e640 1 --2- 172.21.15.116:0/3511308425 >> v2:172.21.15.116:3300/0 conn(0x7fe118053c10 0x7fe118006ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7fe11f59e640 1 --2- 172.21.15.116:0/3511308425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1180e76e0 0x7fe11804ca00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.466+0000 7fe11f59e640 1 -- 172.21.15.116:0/3511308425 >> 172.21.15.116:0/3511308425 conn(0x7fe118058eb0 msgr2=0x7fe118059290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.467+0000 7f10bb7fe640 1 --2- 172.21.15.116:0/1376858279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10bc0b20a0 0x7f10bc0d1b60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f10b00097a0 tx=0x7f10b002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef01cf6e83af48bd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.467+0000 7fe11f59e640 1 -- 172.21.15.116:0/3511308425 shutdown_connections 2023-11-28T17:09:08.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.467+0000 7fe925a09640 1 -- 172.21.15.116:0/31693707 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe900005250 con 0x7fe9180eb2c0 2023-11-28T17:09:08.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.467+0000 7fe11f59e640 1 -- 172.21.15.116:0/3511308425 wait complete. 2023-11-28T17:09:08.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.467+0000 7f10baffd640 1 -- 172.21.15.116:0/1376858279 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b0031780 con 0x7f10bc0b20a0 2023-11-28T17:09:08.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.468+0000 7f10baffd640 1 -- 172.21.15.116:0/1376858279 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b00318e0 con 0x7f10bc0b20a0 2023-11-28T17:09:08.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.468+0000 7f43e1f68640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43dc0d1780 0x7f43dc0d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.468+0000 7f43e1f68640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f43dc0b2ad0 0x7f43dc0ce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.468+0000 7f43e1f68640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43dc0cdcb0 con 0x7f43dc0b20a0 2023-11-28T17:09:08.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.468+0000 7f43e1f68640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43dc0d39e0 con 0x7f43dc0b2ad0 2023-11-28T17:09:08.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.468+0000 7f43e1f68640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc0d3cc0 con 0x7f43dc0d1780 2023-11-28T17:09:08.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.468+0000 7f10baffd640 1 -- 172.21.15.116:0/1376858279 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b0031c70 con 0x7f10bc0b20a0 2023-11-28T17:09:08.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.468+0000 7f6cb078a640 1 -- 172.21.15.116:0/551782155 shutdown_connections 2023-11-28T17:09:08.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.468+0000 7fe907fff640 1 -- 172.21.15.116:0/31693707 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe914013470 con 0x7fe9180eb2c0 2023-11-28T17:09:08.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.468+0000 7f6cb078a640 1 -- 172.21.15.116:0/551782155 wait complete. 2023-11-28T17:09:08.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.468+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43dc0d1780 0x7f43dc0d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.469+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43dc0d1780 0x7f43dc0d01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49496/0 (socket says 172.21.15.116:49496) 2023-11-28T17:09:08.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.469+0000 7f43db7fe640 1 -- 172.21.15.116:0/2682364089 learned_addr learned my addr 172.21.15.116:0/2682364089 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.469+0000 7f43dbfff640 1 --2- 172.21.15.116:0/2682364089 >> v2:172.21.15.156:3300/0 conn(0x7f43dc0b2ad0 0x7f43dc0ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.469+0000 7f10c2708640 1 -- 172.21.15.116:0/1376858279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10bc0b20a0 msgr2=0x7f10bc0d1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.470+0000 7f10c2708640 1 --2- 172.21.15.116:0/1376858279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10bc0b20a0 0x7f10bc0d1b60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f10b00097a0 tx=0x7f10b002b820 comp rx=0 tx=0).stop 2023-11-28T17:09:08.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.470+0000 7fe11f59e640 1 Processor -- start 2023-11-28T17:09:08.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.470+0000 7f43dbfff640 1 -- 172.21.15.116:0/2682364089 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43dc0d1780 msgr2=0x7f43dc0d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.470+0000 7f43dbfff640 1 --2- 172.21.15.116:0/2682364089 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43dc0d1780 0x7f43dc0d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.470+0000 7f43dbfff640 1 -- 172.21.15.116:0/2682364089 >> v1:172.21.15.156:6789/0 conn(0x7f43dc0b20a0 legacy=0x7f43dc0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.470+0000 7f43dbfff640 1 -- 172.21.15.116:0/2682364089 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43dc0cdf50 con 0x7f43dc0b2ad0 2023-11-28T17:09:08.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.470+0000 7fe907fff640 1 --2- 172.21.15.116:0/31693707 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe90c06b2a0 0x7fe90c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.470+0000 7fe907fff640 1 -- 172.21.15.116:0/31693707 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe914089f30 con 0x7fe9180eb2c0 2023-11-28T17:09:08.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.471+0000 7f10c2708640 1 -- 172.21.15.116:0/1376858279 shutdown_connections 2023-11-28T17:09:08.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.471+0000 7f10c2708640 1 --2- 172.21.15.116:0/1376858279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10bc0b20a0 0x7f10bc0d1b60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.471+0000 7fe91effd640 1 --2- 172.21.15.116:0/31693707 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe90c06b2a0 0x7fe90c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.471+0000 7f6cb078a640 1 Processor -- start 2023-11-28T17:09:08.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.470+0000 7fe11f59e640 1 -- start start 2023-11-28T17:09:08.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.471+0000 7f10c2708640 1 --2- 172.21.15.116:0/1376858279 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f10bc0d1780 0x7f10bc0d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.471+0000 7f6cb078a640 1 -- start start 2023-11-28T17:09:08.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.471+0000 7f05422ec640 1 Processor -- start 2023-11-28T17:09:08.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.471+0000 7f10c2708640 1 -- 172.21.15.116:0/1376858279 >> 172.21.15.116:0/1376858279 conn(0x7f10bc09c4e0 msgr2=0x7f10bc0d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.472+0000 7fe91effd640 1 --2- 172.21.15.116:0/31693707 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe90c06b2a0 0x7fe90c06d760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe918058300 tx=0x7fe9200aa000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.472+0000 7f10c2708640 1 -- 172.21.15.116:0/1376858279 shutdown_connections 2023-11-28T17:09:08.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.472+0000 7f10c2708640 1 -- 172.21.15.116:0/1376858279 wait complete. 2023-11-28T17:09:08.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.471+0000 7f05422ec640 1 -- start start 2023-11-28T17:09:08.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.472+0000 7f43dbfff640 1 --2- 172.21.15.116:0/2682364089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43dc0b2ad0 0x7f43dc0ce750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f43c40097a0 tx=0x7f43c402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0c5a19e1ad84a5e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.472+0000 7f6cb078a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca003e800 0x7f6ca00d8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.472+0000 7f6cb078a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca00540f0 0x7f6ca0051e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.472+0000 7f6cb078a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca00e3080 0x7f6ca0052340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.472+0000 7f6cb078a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca00daa30 con 0x7f6ca003e800 2023-11-28T17:09:08.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.472+0000 7f6cb078a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca00dad10 con 0x7f6ca00e3080 2023-11-28T17:09:08.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.472+0000 7f6cb078a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca00daff0 con 0x7f6ca00540f0 2023-11-28T17:09:08.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7f6cae526640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca003e800 0x7f6ca00d8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7f6cae526640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca003e800 0x7f6ca00d8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51906/0 (socket says 172.21.15.116:51906) 2023-11-28T17:09:08.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7f6cae526640 1 -- 172.21.15.116:0/3122852559 learned_addr learned my addr 172.21.15.116:0/3122852559 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7f6cadd25640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca00540f0 0x7f6ca0051e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7fe11f59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe118053c10 0x7fe1180e14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7f43da7fc640 1 -- 172.21.15.116:0/2682364089 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c40316f0 con 0x7f43dc0b2ad0 2023-11-28T17:09:08.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7f43da7fc640 1 -- 172.21.15.116:0/2682364089 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c4031850 con 0x7f43dc0b2ad0 2023-11-28T17:09:08.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7f43da7fc640 1 -- 172.21.15.116:0/2682364089 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c4031be0 con 0x7f43dc0b2ad0 2023-11-28T17:09:08.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7fe11f59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe118058990 0x7fe11804e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7f05422ec640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f053c0d1710 0x7f053c0d0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7fe11f59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe1180e76e0 0x7fe11804ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7f6caed27640 1 --2- 172.21.15.116:0/3122852559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca00e3080 0x7f6ca0052340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7f05422ec640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f053c0ceba0 0x7f053c0afd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7f05422ec640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053c0ce710 con 0x7f053c0d3840 2023-11-28T17:09:08.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7f05422ec640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f053c0cf010 con 0x7f053c0d1710 2023-11-28T17:09:08.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7fe11f59e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1180e1a10 con 0x7fe118058990 2023-11-28T17:09:08.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7f6cae526640 1 -- 172.21.15.116:0/3122852559 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca00540f0 msgr2=0x7f6ca0051e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7f6cae526640 1 --2- 172.21.15.116:0/3122852559 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca00540f0 0x7f6ca0051e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7f6cae526640 1 -- 172.21.15.116:0/3122852559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca00e3080 msgr2=0x7f6ca0052340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7f6cae526640 1 --2- 172.21.15.116:0/3122852559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca00e3080 0x7f6ca0052340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7f6cae526640 1 -- 172.21.15.116:0/3122852559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca8068870 con 0x7f6ca003e800 2023-11-28T17:09:08.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7f05422ec640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f053c0d3e00 con 0x7f053c0ceba0 2023-11-28T17:09:08.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7f053bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f053c0d1710 0x7f053c0d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7f053bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f053c0d1710 0x7f053c0d0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51920/0 (socket says 172.21.15.116:51920) 2023-11-28T17:09:08.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7f053bfff640 1 -- 172.21.15.116:0/4214803197 learned_addr learned my addr 172.21.15.116:0/4214803197 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7fe11f59e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe118180720 con 0x7fe118053c10 2023-11-28T17:09:08.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.473+0000 7fe11f59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe118180970 con 0x7fe1180e76e0 2023-11-28T17:09:08.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7fe11ed9d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe1180e76e0 0x7fe11804ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7fe11ed9d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe1180e76e0 0x7fe11804ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49522/0 (socket says 172.21.15.116:49522) 2023-11-28T17:09:08.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7fe11ed9d640 1 -- 172.21.15.116:0/2008980462 learned_addr learned my addr 172.21.15.116:0/2008980462 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7fe11e59c640 1 --2- 172.21.15.116:0/2008980462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe118053c10 0x7fe1180e14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7fe11dd9b640 1 --2- 172.21.15.116:0/2008980462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe118058990 0x7fe11804e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7f43e1f68640 1 -- 172.21.15.116:0/2682364089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43dc0b2ad0 msgr2=0x7f43dc0ce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7f43e1f68640 1 --2- 172.21.15.116:0/2682364089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43dc0b2ad0 0x7f43dc0ce750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f43c40097a0 tx=0x7f43c402b790 comp rx=0 tx=0).stop 2023-11-28T17:09:08.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7f6cae526640 1 --2- 172.21.15.116:0/3122852559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca003e800 0x7f6ca00d8ee0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6ca400b410 tx=0x7f6ca400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7f43e1f68640 1 -- 172.21.15.116:0/2682364089 shutdown_connections 2023-11-28T17:09:08.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.474+0000 7f0540889640 1 --2- 172.21.15.116:0/4214803197 >> v2:172.21.15.156:3300/0 conn(0x7f053c0ceba0 0x7f053c0afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7f43e1f68640 1 --2- 172.21.15.116:0/2682364089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43dc0b2ad0 0x7f43dc0ce750 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7f43e1f68640 1 --2- 172.21.15.116:0/2682364089 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43dc0d1780 0x7f43dc0d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7f43e1f68640 1 -- 172.21.15.116:0/2682364089 >> 172.21.15.116:0/2682364089 conn(0x7f43dc09c4e0 msgr2=0x7f43dc0d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7fe11e59c640 1 -- 172.21.15.116:0/2008980462 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe1180e76e0 msgr2=0x7fe11804ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7fe11e59c640 1 --2- 172.21.15.116:0/2008980462 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe1180e76e0 0x7fe11804ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7fe11e59c640 1 -- 172.21.15.116:0/2008980462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe118058990 msgr2=0x7fe11804e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7fe11e59c640 1 --2- 172.21.15.116:0/2008980462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe118058990 0x7fe11804e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7fe11e59c640 1 -- 172.21.15.116:0/2008980462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114009400 con 0x7fe118053c10 2023-11-28T17:09:08.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7f053bfff640 1 -- 172.21.15.116:0/4214803197 >> v2:172.21.15.156:3300/0 conn(0x7f053c0ceba0 msgr2=0x7f053c0afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7f053bfff640 1 --2- 172.21.15.116:0/4214803197 >> v2:172.21.15.156:3300/0 conn(0x7f053c0ceba0 0x7f053c0afd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7f053bfff640 1 -- 172.21.15.116:0/4214803197 >> v1:172.21.15.156:6789/0 conn(0x7f053c0d3840 legacy=0x7f053c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7f053bfff640 1 -- 172.21.15.116:0/4214803197 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c0cdde0 con 0x7f053c0d1710 2023-11-28T17:09:08.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7f43e1f68640 1 -- 172.21.15.116:0/2682364089 shutdown_connections 2023-11-28T17:09:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7f6c9effd640 1 -- 172.21.15.116:0/3122852559 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4011830 con 0x7f6ca003e800 2023-11-28T17:09:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7fe11e59c640 1 --2- 172.21.15.116:0/2008980462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe118053c10 0x7fe1180e14d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe114033000 tx=0x7fe1140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7f6c9effd640 1 -- 172.21.15.116:0/3122852559 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca4011990 con 0x7f6ca003e800 2023-11-28T17:09:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7f6c9effd640 1 -- 172.21.15.116:0/3122852559 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4011d40 con 0x7f6ca003e800 2023-11-28T17:09:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.476+0000 7f10c2708640 1 Processor -- start 2023-11-28T17:09:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.476+0000 7f6cb078a640 1 -- 172.21.15.116:0/3122852559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca00db2d0 con 0x7f6ca003e800 2023-11-28T17:09:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.476+0000 7f6cb078a640 1 -- 172.21.15.116:0/3122852559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca00d9420 con 0x7f6ca003e800 2023-11-28T17:09:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.476+0000 7f10c2708640 1 -- start start 2023-11-28T17:09:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.477+0000 7fe10effd640 1 -- 172.21.15.116:0/2008980462 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11403ac60 con 0x7fe118053c10 2023-11-28T17:09:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.477+0000 7fe10effd640 1 -- 172.21.15.116:0/2008980462 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11403ade0 con 0x7fe118053c10 2023-11-28T17:09:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.477+0000 7fe10effd640 1 -- 172.21.15.116:0/2008980462 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe114044630 con 0x7fe118053c10 2023-11-28T17:09:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.475+0000 7f43e1f68640 1 -- 172.21.15.116:0/2682364089 wait complete. 2023-11-28T17:09:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.477+0000 7f34648da640 1 Processor -- start 2023-11-28T17:09:08.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.477+0000 7f053bfff640 1 --2- 172.21.15.116:0/4214803197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f053c0d1710 0x7f053c0d0170 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f053c08bfa0 tx=0x7f052c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f8b7df6bfc892f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.477+0000 7f34648da640 1 -- start start 2023-11-28T17:09:08.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.477+0000 7fe11f59e640 1 -- 172.21.15.116:0/2008980462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe118180c50 con 0x7fe118053c10 2023-11-28T17:09:08.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.477+0000 7fe11f59e640 1 -- 172.21.15.116:0/2008980462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe118181210 con 0x7fe118053c10 2023-11-28T17:09:08.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.477+0000 7f6cb078a640 1 -- 172.21.15.116:0/3122852559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c90005250 con 0x7f6ca003e800 2023-11-28T17:09:08.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.478+0000 7f053affd640 1 -- 172.21.15.116:0/4214803197 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c030860 con 0x7f053c0d1710 2023-11-28T17:09:08.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.478+0000 7f053affd640 1 -- 172.21.15.116:0/4214803197 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c0309c0 con 0x7f053c0d1710 2023-11-28T17:09:08.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.478+0000 7f053affd640 1 -- 172.21.15.116:0/4214803197 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c030860 con 0x7f053c0d1710 2023-11-28T17:09:08.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.478+0000 7f10c2708640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f10bc0b20a0 0x7f10bc160320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.478+0000 7f10c2708640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10bc0b2ad0 0x7f10bc0c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.478+0000 7f10c2708640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10bc0d1780 0x7f10bc0c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.478+0000 7f10c2708640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0c99d0 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.478+0000 7f10c2708640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0c96f0 con 0x7f10bc0d1780 2023-11-28T17:09:08.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.478+0000 7f10c2708640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0b6430 con 0x7f10bc0b20a0 2023-11-28T17:09:08.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.479+0000 7f10bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10bc0b2ad0 0x7f10bc0c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.479+0000 7f10bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10bc0b2ad0 0x7f10bc0c8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51928/0 (socket says 172.21.15.116:51928) 2023-11-28T17:09:08.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.479+0000 7f10bb7fe640 1 -- 172.21.15.116:0/3742349631 learned_addr learned my addr 172.21.15.116:0/3742349631 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.479+0000 7f10bbfff640 1 --2- 172.21.15.116:0/3742349631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f10bc0b20a0 0x7f10bc160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.479+0000 7f43e1f68640 1 Processor -- start 2023-11-28T17:09:08.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.479+0000 7fe11f59e640 1 -- 172.21.15.116:0/2008980462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1180534c0 con 0x7fe118053c10 2023-11-28T17:09:08.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.479+0000 7f34648da640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f34601594f0 0x7f34600cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.479+0000 7f34648da640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34600cb650 0x7f34600703f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.479+0000 7f34648da640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34600b2770 con 0x7f34600b3850 2023-11-28T17:09:08.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.479+0000 7f34648da640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34600ce090 con 0x7f34601594f0 2023-11-28T17:09:08.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.479+0000 7f34648da640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34600cddb0 con 0x7f34600cb650 2023-11-28T17:09:08.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.480+0000 7f43e1f68640 1 -- start start 2023-11-28T17:09:08.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.480+0000 7f10c0ca5640 1 --2- 172.21.15.116:0/3742349631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10bc0d1780 0x7f10bc0c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.479+0000 7f05422ec640 1 -- 172.21.15.116:0/4214803197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f053c0d1710 msgr2=0x7f053c0d0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.479+0000 7f05422ec640 1 --2- 172.21.15.116:0/4214803197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f053c0d1710 0x7f053c0d0170 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f053c08bfa0 tx=0x7f052c02b9a0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.480+0000 7f345dd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f34601594f0 0x7f34600cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.480+0000 7f10bb7fe640 1 -- 172.21.15.116:0/3742349631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f10bc0b20a0 msgr2=0x7f10bc160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.480+0000 7f10bb7fe640 1 --2- 172.21.15.116:0/3742349631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f10bc0b20a0 0x7f10bc160320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.480+0000 7f10bb7fe640 1 -- 172.21.15.116:0/3742349631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10bc0d1780 msgr2=0x7f10bc0c9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.480+0000 7f10bb7fe640 1 --2- 172.21.15.116:0/3742349631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10bc0d1780 0x7f10bc0c9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.480+0000 7f10bb7fe640 1 -- 172.21.15.116:0/3742349631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ac0094d0 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.480+0000 7f345dd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f34601594f0 0x7f34600cd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51940/0 (socket says 172.21.15.116:51940) 2023-11-28T17:09:08.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.480+0000 7f345dd9b640 1 -- 172.21.15.116:0/265983708 learned_addr learned my addr 172.21.15.116:0/265983708 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.480+0000 7f05422ec640 1 -- 172.21.15.116:0/4214803197 shutdown_connections 2023-11-28T17:09:08.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.480+0000 7f05422ec640 1 --2- 172.21.15.116:0/4214803197 >> v2:172.21.15.156:3300/0 conn(0x7f053c0ceba0 0x7f053c0afd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.480+0000 7f05422ec640 1 --2- 172.21.15.116:0/4214803197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f053c0d1710 0x7f053c0d0170 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.480+0000 7f05422ec640 1 -- 172.21.15.116:0/4214803197 >> 172.21.15.116:0/4214803197 conn(0x7f053c09c470 msgr2=0x7f053c0d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.481+0000 7f345dd9b640 1 -- 172.21.15.116:0/265983708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34600cb650 msgr2=0x7f34600703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.481+0000 7f345ed9d640 1 --2- 172.21.15.116:0/265983708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34600cb650 0x7f34600703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.481+0000 7f10bb7fe640 1 --2- 172.21.15.116:0/3742349631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10bc0b2ad0 0x7f10bc0c8b30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f10b0034040 tx=0x7f10b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.481+0000 7f345dd9b640 1 --2- 172.21.15.116:0/265983708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34600cb650 0x7f34600703f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.479+0000 7f6c9effd640 1 -- 172.21.15.116:0/3122852559 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6ca401a400 con 0x7f6ca003e800 2023-11-28T17:09:08.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.481+0000 7f43e1f68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43dc0b20a0 0x7f43dc0b1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.481+0000 7f43e1f68640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43dc0b2ad0 0x7f43dc14b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.481+0000 7f345dd9b640 1 -- 172.21.15.116:0/265983708 >> v1:172.21.15.156:6789/0 conn(0x7f34600b3850 legacy=0x7f34600b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.481+0000 7f345dd9b640 1 -- 172.21.15.116:0/265983708 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34600b2120 con 0x7f34601594f0 2023-11-28T17:09:08.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.481+0000 7f345ed9d640 1 --2- 172.21.15.116:0/265983708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34600cb650 0x7f34600703f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:08.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.481+0000 7f43e1f68640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43dc0d1780 0x7f43dc14b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.481+0000 7f43e1f68640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc0cbcd0 con 0x7f43dc0b20a0 2023-11-28T17:09:08.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.481+0000 7f43e1f68640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc0cb050 con 0x7f43dc0d1780 2023-11-28T17:09:08.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.481+0000 7f43e1f68640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc0cb330 con 0x7f43dc0b2ad0 2023-11-28T17:09:08.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.481+0000 7f6c9effd640 1 --2- 172.21.15.116:0/3122852559 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6c8c06b370 0x7f6c8c06d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7f6c9effd640 1 -- 172.21.15.116:0/3122852559 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ca4089ce0 con 0x7f6ca003e800 2023-11-28T17:09:08.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7fe10effd640 1 -- 172.21.15.116:0/2008980462 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe11402f050 con 0x7fe118053c10 2023-11-28T17:09:08.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43dc0b20a0 0x7f43dc0b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43dc0b20a0 0x7f43dc0b1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51946/0 (socket says 172.21.15.116:51946) 2023-11-28T17:09:08.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7f43db7fe640 1 -- 172.21.15.116:0/3058004972 learned_addr learned my addr 172.21.15.116:0/3058004972 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7fd793fff640 1 Processor -- start 2023-11-28T17:09:08.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7f43daffd640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43dc0b2ad0 0x7f43dc14b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7f43dbfff640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43dc0d1780 0x7f43dc14b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7f43db7fe640 1 -- 172.21.15.116:0/3058004972 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43dc0b2ad0 msgr2=0x7f43dc14b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7f43db7fe640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43dc0b2ad0 0x7f43dc14b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7f43db7fe640 1 -- 172.21.15.116:0/3058004972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43dc0d1780 msgr2=0x7f43dc14b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7f43db7fe640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43dc0d1780 0x7f43dc14b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7f43db7fe640 1 -- 172.21.15.116:0/3058004972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c4009400 con 0x7f43dc0b20a0 2023-11-28T17:09:08.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7f345dd9b640 1 --2- 172.21.15.116:0/265983708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34601594f0 0x7f34600cd7d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f34540091e0 tx=0x7f345402dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36530eefb4a7dc66 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.483+0000 7f05422ec640 1 -- 172.21.15.116:0/4214803197 shutdown_connections 2023-11-28T17:09:08.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.483+0000 7f05422ec640 1 -- 172.21.15.116:0/4214803197 wait complete. 2023-11-28T17:09:08.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7fd793fff640 1 -- start start 2023-11-28T17:09:08.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.482+0000 7f6cadd25640 1 --2- 172.21.15.116:0/3122852559 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6c8c06b370 0x7f6c8c06d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.483+0000 7fe10effd640 1 --2- 172.21.15.116:0/2008980462 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe10006b370 0x7fe10006d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.483+0000 7f43db7fe640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43dc0b20a0 0x7f43dc0b1090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f43d40070b0 tx=0x7f43d400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.483+0000 7f43dbfff640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43dc0d1780 0x7f43dc14b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:08.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.483+0000 7fe10effd640 1 -- 172.21.15.116:0/2008980462 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1140abd20 con 0x7fe118053c10 2023-11-28T17:09:08.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.483+0000 7f6cadd25640 1 --2- 172.21.15.116:0/3122852559 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6c8c06b370 0x7f6c8c06d830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6c98006fd0 tx=0x7f6c98008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.483+0000 7fe11dd9b640 1 --2- 172.21.15.116:0/2008980462 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe10006b370 0x7fe10006d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.484+0000 7fe11dd9b640 1 --2- 172.21.15.116:0/2008980462 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe10006b370 0x7fe10006d830 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe1180e33a0 tx=0x7fe110009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.484+0000 7fd793fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd7940d0030 0x7fd7940d0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.484+0000 7fd793fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd79415d500 0x7fd7940d3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.484+0000 7fd793fff640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7940cf060 con 0x7fd79415d120 2023-11-28T17:09:08.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.484+0000 7fd793fff640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7940cf620 con 0x7fd7940d0030 2023-11-28T17:09:08.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.484+0000 7fd793fff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7940b2250 con 0x7fd79415d500 2023-11-28T17:09:08.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.485+0000 7f05422ec640 1 Processor -- start 2023-11-28T17:09:08.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.485+0000 7fd7937fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd79415d500 0x7fd7940d3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.485+0000 7fd7937fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd79415d500 0x7fd7940d3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49550/0 (socket says 172.21.15.116:49550) 2023-11-28T17:09:08.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.485+0000 7fd7937fe640 1 -- 172.21.15.116:0/2486575479 learned_addr learned my addr 172.21.15.116:0/2486575479 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.485+0000 7fd7927fc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd7940d0030 0x7fd7940d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.485+0000 7f05422ec640 1 -- start start 2023-11-28T17:09:08.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.486+0000 7f05422ec640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f053c0ceba0 0x7f053c0b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.486+0000 7f05422ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f053c0d1710 0x7f053c159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.486+0000 7f05422ec640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f053c0d3840 0x7f053c157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.486+0000 7f05422ec640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c0cbcc0 con 0x7f053c0d1710 2023-11-28T17:09:08.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.486+0000 7f05422ec640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053c157d10 con 0x7f053c0ceba0 2023-11-28T17:09:08.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.486+0000 7fd7937fe640 1 -- 172.21.15.116:0/2486575479 >> v2:172.21.15.156:3300/0 conn(0x7fd7940d0030 msgr2=0x7fd7940d0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:08.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.486+0000 7fd7937fe640 1 --2- 172.21.15.116:0/2486575479 >> v2:172.21.15.156:3300/0 conn(0x7fd7940d0030 0x7fd7940d0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.486+0000 7fd7937fe640 1 -- 172.21.15.116:0/2486575479 >> v1:172.21.15.156:6789/0 conn(0x7fd79415d120 legacy=0x7fd7940cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.486+0000 7fd7937fe640 1 -- 172.21.15.116:0/2486575479 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7940cf340 con 0x7fd79415d500 2023-11-28T17:09:08.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.486+0000 7f05422ec640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053c0b6200 con 0x7f053c0d3840 2023-11-28T17:09:08.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.487+0000 7f053b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f053c0d1710 0x7f053c159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.487+0000 7f053bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f053c0ceba0 0x7f053c0b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.487+0000 7f053b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f053c0d1710 0x7f053c159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51950/0 (socket says 172.21.15.116:51950) 2023-11-28T17:09:08.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.487+0000 7f053b7fe640 1 -- 172.21.15.116:0/695272040 learned_addr learned my addr 172.21.15.116:0/695272040 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.487+0000 7f0540889640 1 --2- 172.21.15.116:0/695272040 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f053c0d3840 0x7f053c157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.487+0000 7f053b7fe640 1 -- 172.21.15.116:0/695272040 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f053c0d3840 msgr2=0x7f053c157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.487+0000 7f053b7fe640 1 --2- 172.21.15.116:0/695272040 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f053c0d3840 0x7f053c157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.487+0000 7f053b7fe640 1 -- 172.21.15.116:0/695272040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f053c0ceba0 msgr2=0x7f053c0b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.487+0000 7f053b7fe640 1 --2- 172.21.15.116:0/695272040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f053c0ceba0 0x7f053c0b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.487+0000 7f053b7fe640 1 -- 172.21.15.116:0/695272040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c0094d0 con 0x7f053c0d1710 2023-11-28T17:09:08.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.487+0000 7f0540889640 1 --2- 172.21.15.116:0/695272040 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f053c0d3840 0x7f053c157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:08.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.487+0000 7f10b8ff9640 1 -- 172.21.15.116:0/3742349631 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b0032820 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.487+0000 7fd7937fe640 1 --2- 172.21.15.116:0/2486575479 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd79415d500 0x7fd7940d3a30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd784009e50 tx=0x7fd78402bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74581ede39468767 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.487+0000 7f345d59a640 1 -- 172.21.15.116:0/265983708 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345402eae0 con 0x7f34601594f0 2023-11-28T17:09:08.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7f345d59a640 1 -- 172.21.15.116:0/265983708 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345402ec40 con 0x7f34601594f0 2023-11-28T17:09:08.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7f345d59a640 1 -- 172.21.15.116:0/265983708 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454036440 con 0x7f34601594f0 2023-11-28T17:09:08.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7f10b8ff9640 1 -- 172.21.15.116:0/3742349631 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b0032980 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7f10b8ff9640 1 -- 172.21.15.116:0/3742349631 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b0032820 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7f43d3fff640 1 -- 172.21.15.116:0/3058004972 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d400dd80 con 0x7f43dc0b20a0 2023-11-28T17:09:08.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7f43d3fff640 1 -- 172.21.15.116:0/3058004972 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d400f040 con 0x7f43dc0b20a0 2023-11-28T17:09:08.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7f43d3fff640 1 -- 172.21.15.116:0/3058004972 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d4015600 con 0x7f43dc0b20a0 2023-11-28T17:09:08.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7f43e1f68640 1 -- 172.21.15.116:0/3058004972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43dc0cb610 con 0x7f43dc0b20a0 2023-11-28T17:09:08.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7f6c9effd640 1 -- 172.21.15.116:0/3122852559 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ca4054bc0 con 0x7f6ca003e800 2023-11-28T17:09:08.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7f053b7fe640 1 --2- 172.21.15.116:0/695272040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f053c0d1710 0x7f053c159d40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f053000c480 tx=0x7f053000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7f43e1f68640 1 -- 172.21.15.116:0/3058004972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43dc0b15d0 con 0x7f43dc0b20a0 2023-11-28T17:09:08.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7fd791ffb640 1 -- 172.21.15.116:0/2486575479 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78402ca80 con 0x7fd79415d500 2023-11-28T17:09:08.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7fd791ffb640 1 -- 172.21.15.116:0/2486575479 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78402cbe0 con 0x7fd79415d500 2023-11-28T17:09:08.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7fd791ffb640 1 -- 172.21.15.116:0/2486575479 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7840343f0 con 0x7fd79415d500 2023-11-28T17:09:08.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7f34648da640 1 -- 172.21.15.116:0/265983708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34601594f0 msgr2=0x7f34600cd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.488+0000 7f34648da640 1 --2- 172.21.15.116:0/265983708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34601594f0 0x7f34600cd7d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f34540091e0 tx=0x7f345402dd30 comp rx=0 tx=0).stop 2023-11-28T17:09:08.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.489+0000 7f10c2708640 1 -- 172.21.15.116:0/3742349631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b0009400 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.489+0000 7f0538ff9640 1 -- 172.21.15.116:0/695272040 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05300052d0 con 0x7f053c0d1710 2023-11-28T17:09:08.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.489+0000 7f0538ff9640 1 -- 172.21.15.116:0/695272040 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0530005430 con 0x7f053c0d1710 2023-11-28T17:09:08.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.489+0000 7f0538ff9640 1 -- 172.21.15.116:0/695272040 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0530012870 con 0x7f053c0d1710 2023-11-28T17:09:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.489+0000 7f34648da640 1 -- 172.21.15.116:0/265983708 shutdown_connections 2023-11-28T17:09:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.489+0000 7f34648da640 1 --2- 172.21.15.116:0/265983708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34600cb650 0x7f34600703f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.489+0000 7f34648da640 1 --2- 172.21.15.116:0/265983708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34601594f0 0x7f34600cd7d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.489+0000 7f34648da640 1 -- 172.21.15.116:0/265983708 >> 172.21.15.116:0/265983708 conn(0x7f346006c250 msgr2=0x7f346006c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.489+0000 7fd793fff640 1 -- 172.21.15.116:0/2486575479 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd79415d500 msgr2=0x7fd7940d3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.489+0000 7fd793fff640 1 --2- 172.21.15.116:0/2486575479 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd79415d500 0x7fd7940d3a30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd784009e50 tx=0x7fd78402bee0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.490+0000 7fd793fff640 1 -- 172.21.15.116:0/2486575479 shutdown_connections 2023-11-28T17:09:08.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.490+0000 7fd793fff640 1 --2- 172.21.15.116:0/2486575479 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd79415d500 0x7fd7940d3a30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.490+0000 7fd793fff640 1 --2- 172.21.15.116:0/2486575479 >> v2:172.21.15.156:3300/0 conn(0x7fd7940d0030 0x7fd7940d0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.490+0000 7fd793fff640 1 -- 172.21.15.116:0/2486575479 >> 172.21.15.116:0/2486575479 conn(0x7fd7940d2f10 msgr2=0x7fd7940d3320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.490+0000 7f43e1f68640 1 -- 172.21.15.116:0/3058004972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43dc0b17c0 con 0x7f43dc0b20a0 2023-11-28T17:09:08.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.490+0000 7f05422ec640 1 -- 172.21.15.116:0/695272040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053c15a280 con 0x7f053c0d1710 2023-11-28T17:09:08.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.490+0000 7f05422ec640 1 -- 172.21.15.116:0/695272040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053c0b6710 con 0x7f053c0d1710 2023-11-28T17:09:08.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.490+0000 7f34648da640 1 -- 172.21.15.116:0/265983708 shutdown_connections 2023-11-28T17:09:08.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.490+0000 7fd793fff640 1 -- 172.21.15.116:0/2486575479 shutdown_connections 2023-11-28T17:09:08.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.489+0000 7f10c2708640 1 -- 172.21.15.116:0/3742349631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10bc0b69f0 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.491+0000 7f10c2708640 1 -- 172.21.15.116:0/3742349631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10bc0b6c10 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.491+0000 7fe907fff640 1 -- 172.21.15.116:0/31693707 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe914054d20 con 0x7fe9180eb2c0 2023-11-28T17:09:08.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.491+0000 7f05422ec640 1 -- 172.21.15.116:0/695272040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f053c0cbe00 con 0x7f053c0d1710 2023-11-28T17:09:08.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.490+0000 7f34648da640 1 -- 172.21.15.116:0/265983708 wait complete. 2023-11-28T17:09:08.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.490+0000 7fd793fff640 1 -- 172.21.15.116:0/2486575479 wait complete. 2023-11-28T17:09:08.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.492+0000 7fd793fff640 1 Processor -- start 2023-11-28T17:09:08.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.492+0000 7fd793fff640 1 -- start start 2023-11-28T17:09:08.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.493+0000 7fd793fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7940d0030 0x7fd7941fee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.493+0000 7fd793fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd79415d120 0x7fd79415f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.493+0000 7fd793fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd79415d500 0x7fd79406b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.493+0000 7fd793fff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79415fbe0 con 0x7fd7940d0030 2023-11-28T17:09:08.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.493+0000 7fd793fff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79415fec0 con 0x7fd79415d120 2023-11-28T17:09:08.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.493+0000 7fd793fff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7941601a0 con 0x7fd79415d500 2023-11-28T17:09:08.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.493+0000 7f34648da640 1 Processor -- start 2023-11-28T17:09:08.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.494+0000 7f34648da640 1 -- start start 2023-11-28T17:09:08.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.494+0000 7f43d3fff640 1 -- 172.21.15.116:0/3058004972 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f43d4015760 con 0x7f43dc0b20a0 2023-11-28T17:09:08.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.494+0000 7fd7937fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd79415d500 0x7fd79406b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.494+0000 7fd7937fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd79415d500 0x7fd79406b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49576/0 (socket says 172.21.15.116:49576) 2023-11-28T17:09:08.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.494+0000 7fd7937fe640 1 -- 172.21.15.116:0/1418210099 learned_addr learned my addr 172.21.15.116:0/1418210099 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.495+0000 7fd7937fe640 1 -- 172.21.15.116:0/1418210099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd79415d120 msgr2=0x7fd79415f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.495+0000 7fd7937fe640 1 --2- 172.21.15.116:0/1418210099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd79415d120 0x7fd79415f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.495+0000 7fd7937fe640 1 -- 172.21.15.116:0/1418210099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7940d0030 msgr2=0x7fd7941fee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.495+0000 7fd7937fe640 1 --2- 172.21.15.116:0/1418210099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7940d0030 0x7fd7941fee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.495+0000 7fd7937fe640 1 -- 172.21.15.116:0/1418210099 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd784009b30 con 0x7fd79415d500 2023-11-28T17:09:08.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.495+0000 7f43d3fff640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f43bc06b370 0x7f43bc06d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.495+0000 7f43d3fff640 1 -- 172.21.15.116:0/3058004972 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43d4089ad0 con 0x7f43dc0b20a0 2023-11-28T17:09:08.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.495+0000 7f43daffd640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f43bc06b370 0x7f43bc06d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.495+0000 7fd7937fe640 1 --2- 172.21.15.116:0/1418210099 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd79415d500 0x7fd79406b280 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd784009e50 tx=0x7fd784004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.496+0000 7fd77b7fe640 1 -- 172.21.15.116:0/1418210099 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78403e450 con 0x7fd79415d500 2023-11-28T17:09:08.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.496+0000 7fd793fff640 1 -- 172.21.15.116:0/1418210099 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd794160480 con 0x7fd79415d500 2023-11-28T17:09:08.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.496+0000 7fd77b7fe640 1 -- 172.21.15.116:0/1418210099 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78403e5b0 con 0x7fd79415d500 2023-11-28T17:09:08.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.496+0000 7fd77b7fe640 1 -- 172.21.15.116:0/1418210099 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78403e960 con 0x7fd79415d500 2023-11-28T17:09:08.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.496+0000 7f34648da640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34600b3850 0x7f346014f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.496+0000 7f34648da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34600cb650 0x7f34601559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.497+0000 7f345e59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34600b3850 0x7f346014f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.497+0000 7f345e59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34600b3850 0x7f346014f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49578/0 (socket says 172.21.15.116:49578) 2023-11-28T17:09:08.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.497+0000 7f345e59c640 1 -- 172.21.15.116:0/1773642002 learned_addr learned my addr 172.21.15.116:0/1773642002 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:08.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.497+0000 7f345dd9b640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34600cb650 0x7f34601559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.497+0000 7f34648da640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3460155ef0 0x7f34601f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.496+0000 7fd793fff640 1 -- 172.21.15.116:0/1418210099 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd79415ba90 con 0x7fd79415d500 2023-11-28T17:09:08.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.497+0000 7f34648da640 1 -- 172.21.15.116:0/1773642002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346014f7f0 con 0x7f34600cb650 2023-11-28T17:09:08.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.497+0000 7fd793fff640 1 -- 172.21.15.116:0/1418210099 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd794069360 con 0x7fd79415d500 2023-11-28T17:09:08.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.497+0000 7f34648da640 1 -- 172.21.15.116:0/1773642002 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34600a8460 con 0x7f3460155ef0 2023-11-28T17:09:08.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.497+0000 7f34648da640 1 -- 172.21.15.116:0/1773642002 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34601ee950 con 0x7f34600b3850 2023-11-28T17:09:08.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.499+0000 7f43daffd640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f43bc06b370 0x7f43bc06d830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0ce920 tx=0x7f43cc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.498+0000 7f345ed9d640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3460155ef0 0x7f34601f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.498+0000 7f345ed9d640 1 -- 172.21.15.116:0/1773642002 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34600b3850 msgr2=0x7f346014f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.498+0000 7f345ed9d640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34600b3850 0x7f346014f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.498+0000 7f345ed9d640 1 -- 172.21.15.116:0/1773642002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34600cb650 msgr2=0x7f34601559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.498+0000 7f345ed9d640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34600cb650 0x7f34601559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.498+0000 7f345ed9d640 1 -- 172.21.15.116:0/1773642002 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3454009d60 con 0x7f3460155ef0 2023-11-28T17:09:08.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.499+0000 7fd77b7fe640 1 -- 172.21.15.116:0/1418210099 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fd78402cd50 con 0x7fd79415d500 2023-11-28T17:09:08.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.499+0000 7f345ed9d640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3460155ef0 0x7f34601f2100 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3458010330 tx=0x7f34580107f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.500+0000 7f10b8ff9640 1 -- 172.21.15.116:0/3742349631 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f10b0032af0 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.500+0000 7fe10effd640 1 -- 172.21.15.116:0/2008980462 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe114076c20 con 0x7fe118053c10 2023-11-28T17:09:08.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.500+0000 7fd77b7fe640 1 --2- 172.21.15.116:0/1418210099 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd78006b2a0 0x7fd78006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.500+0000 7f0538ff9640 1 -- 172.21.15.116:0/695272040 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f05300129d0 con 0x7f053c0d1710 2023-11-28T17:09:08.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.501+0000 7fd77b7fe640 1 -- 172.21.15.116:0/1418210099 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7840aca30 con 0x7fd79415d500 2023-11-28T17:09:08.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.501+0000 7f344effd640 1 -- 172.21.15.116:0/1773642002 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345800e480 con 0x7f3460155ef0 2023-11-28T17:09:08.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.501+0000 7f344effd640 1 -- 172.21.15.116:0/1773642002 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345800e5e0 con 0x7f3460155ef0 2023-11-28T17:09:08.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.501+0000 7f344effd640 1 -- 172.21.15.116:0/1773642002 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345800a670 con 0x7f3460155ef0 2023-11-28T17:09:08.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.501+0000 7f34648da640 1 -- 172.21.15.116:0/1773642002 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3460156480 con 0x7f3460155ef0 2023-11-28T17:09:08.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.501+0000 7f34648da640 1 -- 172.21.15.116:0/1773642002 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3460156a40 con 0x7f3460155ef0 2023-11-28T17:09:08.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.502+0000 7f10b8ff9640 1 --2- 172.21.15.116:0/3742349631 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f10a806b300 0x7f10a806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.502+0000 7f0538ff9640 1 --2- 172.21.15.116:0/695272040 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f052806b370 0x7f052806d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.502+0000 7f0538ff9640 1 -- 172.21.15.116:0/695272040 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f053008a030 con 0x7f053c0d1710 2023-11-28T17:09:08.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.502+0000 7fd792ffd640 1 --2- 172.21.15.116:0/1418210099 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd78006b2a0 0x7fd78006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.503+0000 7f053bfff640 1 --2- 172.21.15.116:0/695272040 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f052806b370 0x7f052806d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.503+0000 7f34648da640 1 -- 172.21.15.116:0/1773642002 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34600693d0 con 0x7f3460155ef0 2023-11-28T17:09:08.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.503+0000 7f10b8ff9640 1 -- 172.21.15.116:0/3742349631 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10b00abc30 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.504+0000 7f053bfff640 1 --2- 172.21.15.116:0/695272040 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f052806b370 0x7f052806d830 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f052c036000 tx=0x7f052c03c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.505+0000 7f10bbfff640 1 --2- 172.21.15.116:0/3742349631 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f10a806b300 0x7f10a806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.505+0000 7f344effd640 1 -- 172.21.15.116:0/1773642002 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f345800a7d0 con 0x7f3460155ef0 2023-11-28T17:09:08.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.505+0000 7fd792ffd640 1 --2- 172.21.15.116:0/1418210099 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd78006b2a0 0x7fd78006d760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd78c000f60 tx=0x7fd78c0079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.505+0000 7f344effd640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f344006b2f0 0x7f344006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:08.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.506+0000 7f344effd640 1 -- 172.21.15.116:0/1773642002 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f345808a160 con 0x7f3460155ef0 2023-11-28T17:09:08.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.506+0000 7f10bbfff640 1 --2- 172.21.15.116:0/3742349631 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f10a806b300 0x7f10a806d7c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f10ac009c50 tx=0x7f10ac008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.506+0000 7f345e59c640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f344006b2f0 0x7f344006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:08.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.506+0000 7f345e59c640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f344006b2f0 0x7f344006d7b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3450007fd0 tx=0x7f3450016040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:08.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.508+0000 7f10b8ff9640 1 -- 172.21.15.116:0/3742349631 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10b00769f0 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.512+0000 7fd77b7fe640 1 -- 172.21.15.116:0/1418210099 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd784077b30 con 0x7fd79415d500 2023-11-28T17:09:08.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.525+0000 7f0538ff9640 1 -- 172.21.15.116:0/695272040 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0530054e80 con 0x7f053c0d1710 2023-11-28T17:09:08.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.525+0000 7f344effd640 1 -- 172.21.15.116:0/1773642002 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3458055160 con 0x7f3460155ef0 2023-11-28T17:09:08.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.526+0000 7f43d3fff640 1 -- 172.21.15.116:0/3058004972 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f43d40549d0 con 0x7f43dc0b20a0 2023-11-28T17:09:08.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.876+0000 7f6cb078a640 1 -- 172.21.15.116:0/3122852559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f6c90005500 con 0x7f6ca003e800 2023-11-28T17:09:08.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.877+0000 7f6c9effd640 1 -- 172.21.15.116:0/3122852559 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6ca40581e0 con 0x7f6ca003e800 2023-11-28T17:09:08.882 INFO:teuthology.orchestra.run.smithi116.stdout:55834574852 2023-11-28T17:09:08.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.881+0000 7f6c9cff9640 1 -- 172.21.15.116:0/3122852559 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6c8c06b370 msgr2=0x7f6c8c06d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.881+0000 7f6c9cff9640 1 --2- 172.21.15.116:0/3122852559 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6c8c06b370 0x7f6c8c06d830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6c98006fd0 tx=0x7f6c98008040 comp rx=0 tx=0).stop 2023-11-28T17:09:08.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.882+0000 7f6c9cff9640 1 -- 172.21.15.116:0/3122852559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca003e800 msgr2=0x7f6ca00d8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.882+0000 7f6c9cff9640 1 --2- 172.21.15.116:0/3122852559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca003e800 0x7f6ca00d8ee0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6ca400b410 tx=0x7f6ca400b8d0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.882+0000 7f6c9cff9640 1 -- 172.21.15.116:0/3122852559 shutdown_connections 2023-11-28T17:09:08.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.882+0000 7f6c9cff9640 1 --2- 172.21.15.116:0/3122852559 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6c8c06b370 0x7f6c8c06d830 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.882+0000 7f6c9cff9640 1 --2- 172.21.15.116:0/3122852559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca00e3080 0x7f6ca0052340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.882+0000 7f6c9cff9640 1 --2- 172.21.15.116:0/3122852559 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca00540f0 0x7f6ca0051e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.883+0000 7f6c9cff9640 1 --2- 172.21.15.116:0/3122852559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca003e800 0x7f6ca00d8ee0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.883+0000 7f6c9cff9640 1 -- 172.21.15.116:0/3122852559 >> 172.21.15.116:0/3122852559 conn(0x7f6ca00559a0 msgr2=0x7f6ca003a010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.883+0000 7f6c9cff9640 1 -- 172.21.15.116:0/3122852559 shutdown_connections 2023-11-28T17:09:08.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.883+0000 7f6c9cff9640 1 -- 172.21.15.116:0/3122852559 wait complete. 2023-11-28T17:09:08.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.884+0000 7fe925a09640 1 -- 172.21.15.116:0/31693707 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe900005500 con 0x7fe9180eb2c0 2023-11-28T17:09:08.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.885+0000 7fe907fff640 1 -- 172.21.15.116:0/31693707 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe914021090 con 0x7fe9180eb2c0 2023-11-28T17:09:08.886 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-28T17:09:08.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.888+0000 7fe925a09640 1 -- 172.21.15.116:0/31693707 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe90c06b2a0 msgr2=0x7fe90c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.888+0000 7fe925a09640 1 --2- 172.21.15.116:0/31693707 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe90c06b2a0 0x7fe90c06d760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe918058300 tx=0x7fe9200aa000 comp rx=0 tx=0).stop 2023-11-28T17:09:08.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.888+0000 7fe925a09640 1 -- 172.21.15.116:0/31693707 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9180eb2c0 msgr2=0x7fe91800c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.888+0000 7fe925a09640 1 --2- 172.21.15.116:0/31693707 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9180eb2c0 0x7fe91800c380 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe91400ebe0 tx=0x7fe91400c520 comp rx=0 tx=0).stop 2023-11-28T17:09:08.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.888+0000 7fe925a09640 1 -- 172.21.15.116:0/31693707 shutdown_connections 2023-11-28T17:09:08.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.888+0000 7fe925a09640 1 --2- 172.21.15.116:0/31693707 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe90c06b2a0 0x7fe90c06d760 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.888+0000 7fe925a09640 1 --2- 172.21.15.116:0/31693707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9180ebc80 0x7fe91800c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.888+0000 7fe925a09640 1 --2- 172.21.15.116:0/31693707 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9180eb2c0 0x7fe91800c380 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.888+0000 7fe925a09640 1 --2- 172.21.15.116:0/31693707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9180e77e0 0x7fe918009c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.888+0000 7fe925a09640 1 -- 172.21.15.116:0/31693707 >> 172.21.15.116:0/31693707 conn(0x7fe9180559a0 msgr2=0x7fe91803d070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.888+0000 7fe925a09640 1 -- 172.21.15.116:0/31693707 shutdown_connections 2023-11-28T17:09:08.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.888+0000 7fe925a09640 1 -- 172.21.15.116:0/31693707 wait complete. 2023-11-28T17:09:08.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.897+0000 7fe11f59e640 1 -- 172.21.15.116:0/2008980462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe118181400 con 0x7fe118053c10 2023-11-28T17:09:08.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.898+0000 7fe10effd640 1 -- 172.21.15.116:0/2008980462 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe114006e10 con 0x7fe118053c10 2023-11-28T17:09:08.902 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-28T17:09:08.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.902+0000 7fe10cff9640 1 -- 172.21.15.116:0/2008980462 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe10006b370 msgr2=0x7fe10006d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.902+0000 7fe10cff9640 1 --2- 172.21.15.116:0/2008980462 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe10006b370 0x7fe10006d830 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe1180e33a0 tx=0x7fe110009040 comp rx=0 tx=0).stop 2023-11-28T17:09:08.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.902+0000 7fe10cff9640 1 -- 172.21.15.116:0/2008980462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe118053c10 msgr2=0x7fe1180e14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.902+0000 7fe10cff9640 1 --2- 172.21.15.116:0/2008980462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe118053c10 0x7fe1180e14d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe114033000 tx=0x7fe1140046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.903+0000 7fe10cff9640 1 -- 172.21.15.116:0/2008980462 shutdown_connections 2023-11-28T17:09:08.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.903+0000 7fe10cff9640 1 --2- 172.21.15.116:0/2008980462 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe10006b370 0x7fe10006d830 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.903+0000 7fe10cff9640 1 --2- 172.21.15.116:0/2008980462 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe1180e76e0 0x7fe11804ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.903+0000 7fe10cff9640 1 --2- 172.21.15.116:0/2008980462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe118058990 0x7fe11804e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.903+0000 7fe10cff9640 1 --2- 172.21.15.116:0/2008980462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe118053c10 0x7fe1180e14d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.903+0000 7fe10cff9640 1 -- 172.21.15.116:0/2008980462 >> 172.21.15.116:0/2008980462 conn(0x7fe118058eb0 msgr2=0x7fe118039c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.904+0000 7fe10cff9640 1 -- 172.21.15.116:0/2008980462 shutdown_connections 2023-11-28T17:09:08.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.905+0000 7fe10cff9640 1 -- 172.21.15.116:0/2008980462 wait complete. 2023-11-28T17:09:08.916 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2023-11-28T17:09:08.920 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2023-11-28T17:09:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.922+0000 7f43e1f68640 1 -- 172.21.15.116:0/3058004972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f43dc0693d0 con 0x7f43dc0b20a0 2023-11-28T17:09:08.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.922+0000 7f43d3fff640 1 -- 172.21.15.116:0/3058004972 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f43d401a020 con 0x7f43dc0b20a0 2023-11-28T17:09:08.924 INFO:teuthology.orchestra.run.smithi116.stdout:55834574852 2023-11-28T17:09:08.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.923+0000 7f34648da640 1 -- 172.21.15.116:0/1773642002 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f346006cfb0 con 0x7f3460155ef0 2023-11-28T17:09:08.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.924+0000 7f344effd640 1 -- 172.21.15.116:0/1773642002 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3458058780 con 0x7f3460155ef0 2023-11-28T17:09:08.926 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-28T17:09:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.925+0000 7f43e1f68640 1 -- 172.21.15.116:0/3058004972 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f43bc06b370 msgr2=0x7f43bc06d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.925+0000 7f43e1f68640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f43bc06b370 0x7f43bc06d830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0ce920 tx=0x7f43cc009040 comp rx=0 tx=0).stop 2023-11-28T17:09:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.925+0000 7f43e1f68640 1 -- 172.21.15.116:0/3058004972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43dc0b20a0 msgr2=0x7f43dc0b1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.925+0000 7f43e1f68640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43dc0b20a0 0x7f43dc0b1090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f43d40070b0 tx=0x7f43d400d4a0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.925+0000 7f43e1f68640 1 -- 172.21.15.116:0/3058004972 shutdown_connections 2023-11-28T17:09:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.925+0000 7f43e1f68640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f43bc06b370 0x7f43bc06d830 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.925+0000 7f43e1f68640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43dc0d1780 0x7f43dc14b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.925+0000 7f43e1f68640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43dc0b2ad0 0x7f43dc14b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.925+0000 7f43e1f68640 1 --2- 172.21.15.116:0/3058004972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43dc0b20a0 0x7f43dc0b1090 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.925+0000 7f43e1f68640 1 -- 172.21.15.116:0/3058004972 >> 172.21.15.116:0/3058004972 conn(0x7f43dc09c4e0 msgr2=0x7f43dc0d26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.925+0000 7f43e1f68640 1 -- 172.21.15.116:0/3058004972 shutdown_connections 2023-11-28T17:09:08.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.925+0000 7f43e1f68640 1 -- 172.21.15.116:0/3058004972 wait complete. 2023-11-28T17:09:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.927+0000 7f34648da640 1 -- 172.21.15.116:0/1773642002 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f344006b2f0 msgr2=0x7f344006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.927+0000 7f34648da640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f344006b2f0 0x7f344006d7b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3450007fd0 tx=0x7f3450016040 comp rx=0 tx=0).stop 2023-11-28T17:09:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.927+0000 7f34648da640 1 -- 172.21.15.116:0/1773642002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3460155ef0 msgr2=0x7f34601f2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.927+0000 7f34648da640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3460155ef0 0x7f34601f2100 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3458010330 tx=0x7f34580107f0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.927+0000 7f34648da640 1 -- 172.21.15.116:0/1773642002 shutdown_connections 2023-11-28T17:09:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.927+0000 7f34648da640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f344006b2f0 0x7f344006d7b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.927+0000 7f34648da640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3460155ef0 0x7f34601f2100 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.927+0000 7f34648da640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34600cb650 0x7f34601559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.927+0000 7f34648da640 1 --2- 172.21.15.116:0/1773642002 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34600b3850 0x7f346014f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.927+0000 7f34648da640 1 -- 172.21.15.116:0/1773642002 >> 172.21.15.116:0/1773642002 conn(0x7f346006c250 msgr2=0x7f3460070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.928+0000 7f34648da640 1 -- 172.21.15.116:0/1773642002 shutdown_connections 2023-11-28T17:09:08.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.928+0000 7f34648da640 1 -- 172.21.15.116:0/1773642002 wait complete. 2023-11-28T17:09:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.928+0000 7fd793fff640 1 -- 172.21.15.116:0/1418210099 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd79415c950 con 0x7fd79415d500 2023-11-28T17:09:08.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.929+0000 7fd77b7fe640 1 -- 172.21.15.116:0/1418210099 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd784043020 con 0x7fd79415d500 2023-11-28T17:09:08.931 INFO:teuthology.orchestra.run.smithi116.stdout:55834574852 2023-11-28T17:09:08.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.932+0000 7f05422ec640 1 -- 172.21.15.116:0/695272040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f053c0cbe00 con 0x7f053c0d1710 2023-11-28T17:09:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.932+0000 7fd793fff640 1 -- 172.21.15.116:0/1418210099 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd78006b2a0 msgr2=0x7fd78006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.933+0000 7fd793fff640 1 --2- 172.21.15.116:0/1418210099 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd78006b2a0 0x7fd78006d760 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd78c000f60 tx=0x7fd78c0079b0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.933+0000 7fd793fff640 1 -- 172.21.15.116:0/1418210099 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd79415d500 msgr2=0x7fd79406b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.933+0000 7fd793fff640 1 --2- 172.21.15.116:0/1418210099 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd79415d500 0x7fd79406b280 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd784009e50 tx=0x7fd784004a40 comp rx=0 tx=0).stop 2023-11-28T17:09:08.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.933+0000 7f0538ff9640 1 -- 172.21.15.116:0/695272040 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f053c0cbe00 con 0x7f053c0d1710 2023-11-28T17:09:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.933+0000 7f10c2708640 1 -- 172.21.15.116:0/3742349631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f10bc0693d0 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.933+0000 7fd793fff640 1 -- 172.21.15.116:0/1418210099 shutdown_connections 2023-11-28T17:09:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.933+0000 7fd793fff640 1 --2- 172.21.15.116:0/1418210099 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd78006b2a0 0x7fd78006d760 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.933+0000 7fd793fff640 1 --2- 172.21.15.116:0/1418210099 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd79415d500 0x7fd79406b280 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.933+0000 7fd793fff640 1 --2- 172.21.15.116:0/1418210099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd79415d120 0x7fd79415f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.933+0000 7fd793fff640 1 --2- 172.21.15.116:0/1418210099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7940d0030 0x7fd7941fee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.933+0000 7fd793fff640 1 -- 172.21.15.116:0/1418210099 >> 172.21.15.116:0/1418210099 conn(0x7fd7940d2f10 msgr2=0x7fd7940d32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.936 INFO:teuthology.orchestra.run.smithi116.stdout:55834574852 2023-11-28T17:09:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.933+0000 7fd793fff640 1 -- 172.21.15.116:0/1418210099 shutdown_connections 2023-11-28T17:09:08.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.933+0000 7fd793fff640 1 -- 172.21.15.116:0/1418210099 wait complete. 2023-11-28T17:09:08.937 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2023-11-28T17:09:08.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.934+0000 7f10b8ff9640 1 -- 172.21.15.116:0/3742349631 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f10b007a010 con 0x7f10bc0b2ad0 2023-11-28T17:09:08.938 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-28T17:09:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.936+0000 7f05422ec640 1 -- 172.21.15.116:0/695272040 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f052806b370 msgr2=0x7f052806d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.936+0000 7f05422ec640 1 --2- 172.21.15.116:0/695272040 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f052806b370 0x7f052806d830 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f052c036000 tx=0x7f052c03c040 comp rx=0 tx=0).stop 2023-11-28T17:09:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.936+0000 7f05422ec640 1 -- 172.21.15.116:0/695272040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f053c0d1710 msgr2=0x7f053c159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.936+0000 7f05422ec640 1 --2- 172.21.15.116:0/695272040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f053c0d1710 0x7f053c159d40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f053000c480 tx=0x7f053000c940 comp rx=0 tx=0).stop 2023-11-28T17:09:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.936+0000 7f05422ec640 1 -- 172.21.15.116:0/695272040 shutdown_connections 2023-11-28T17:09:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.936+0000 7f05422ec640 1 --2- 172.21.15.116:0/695272040 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f052806b370 0x7f052806d830 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.936+0000 7f05422ec640 1 --2- 172.21.15.116:0/695272040 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f053c0d3840 0x7f053c157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.936+0000 7f05422ec640 1 --2- 172.21.15.116:0/695272040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f053c0d1710 0x7f053c159d40 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f053000c480 tx=0x7f053000c940 comp rx=0 tx=0).stop 2023-11-28T17:09:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.936+0000 7f05422ec640 1 --2- 172.21.15.116:0/695272040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f053c0ceba0 0x7f053c0b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.936+0000 7f05422ec640 1 -- 172.21.15.116:0/695272040 >> 172.21.15.116:0/695272040 conn(0x7f053c09c470 msgr2=0x7f053c0c8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.936+0000 7f05422ec640 1 -- 172.21.15.116:0/695272040 shutdown_connections 2023-11-28T17:09:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.936+0000 7f05422ec640 1 -- 172.21.15.116:0/695272040 wait complete. 2023-11-28T17:09:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.937+0000 7f10c2708640 1 -- 172.21.15.116:0/3742349631 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f10a806b300 msgr2=0x7f10a806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.937+0000 7f10c2708640 1 --2- 172.21.15.116:0/3742349631 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f10a806b300 0x7f10a806d7c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f10ac009c50 tx=0x7f10ac008040 comp rx=0 tx=0).stop 2023-11-28T17:09:08.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.937+0000 7f10c2708640 1 -- 172.21.15.116:0/3742349631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10bc0b2ad0 msgr2=0x7f10bc0c8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.937+0000 7f10c2708640 1 --2- 172.21.15.116:0/3742349631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10bc0b2ad0 0x7f10bc0c8b30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f10b0034040 tx=0x7f10b00046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.937+0000 7f10c2708640 1 -- 172.21.15.116:0/3742349631 shutdown_connections 2023-11-28T17:09:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.937+0000 7f10c2708640 1 --2- 172.21.15.116:0/3742349631 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f10a806b300 0x7f10a806d7c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.937+0000 7f10c2708640 1 --2- 172.21.15.116:0/3742349631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10bc0d1780 0x7f10bc0c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.937+0000 7f10c2708640 1 --2- 172.21.15.116:0/3742349631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10bc0b2ad0 0x7f10bc0c8b30 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f10b0034040 tx=0x7f10b00046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.937+0000 7f10c2708640 1 --2- 172.21.15.116:0/3742349631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f10bc0b20a0 0x7f10bc160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.937+0000 7f10c2708640 1 -- 172.21.15.116:0/3742349631 >> 172.21.15.116:0/3742349631 conn(0x7f10bc09c4e0 msgr2=0x7f10bc06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.938+0000 7f10c2708640 1 -- 172.21.15.116:0/3742349631 shutdown_connections 2023-11-28T17:09:08.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:08.938+0000 7f10c2708640 1 -- 172.21.15.116:0/3742349631 wait complete. 2023-11-28T17:09:08.950 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.3 2023-11-28T17:09:08.957 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2023-11-28T17:09:08.958 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2023-11-28T17:09:08.960 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.7 2023-11-28T17:09:08.960 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2023-11-28T17:09:09.917 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-28T17:09:09.920 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-28T17:09:09.938 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-28T17:09:09.951 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-28T17:09:09.958 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-28T17:09:09.958 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-28T17:09:09.960 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-28T17:09:09.961 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-28T17:09:10.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.190+0000 7f5e9636f640 1 Processor -- start 2023-11-28T17:09:10.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.190+0000 7f5e9636f640 1 -- start start 2023-11-28T17:09:10.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.192+0000 7f5e9636f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5e900ce140 0x7f5e900d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.192+0000 7f5e9636f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5e900d2d70 0x7f5e900b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.193+0000 7f5e9636f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e900b6640 con 0x7f5e9015d2c0 2023-11-28T17:09:10.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.193+0000 7f5e9636f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e900d31c0 con 0x7f5e900d2d70 2023-11-28T17:09:10.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.193+0000 7f5e9636f640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e900d3480 con 0x7f5e900ce140 2023-11-28T17:09:10.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.193+0000 7f5e8f7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5e900ce140 0x7f5e900d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.194+0000 7f5e8f7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5e900ce140 0x7f5e900d2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:59082/0 (socket says 172.21.15.116:59082) 2023-11-28T17:09:10.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.194+0000 7f5e8ffff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f5e9015d2c0 0x7f5e900cda30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.116:50644/0 (socket says 172.21.15.116:50644) 2023-11-28T17:09:10.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.194+0000 7f5e8ffff640 1 -- 172.21.15.116:0/1249279868 learned_addr learned my addr 172.21.15.116:0/1249279868 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:10.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.194+0000 7f5e9490c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5e900d2d70 0x7f5e900b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.195+0000 7f5e9490c640 1 -- 172.21.15.116:0/1249279868 >> v2:172.21.15.156:3300/0 conn(0x7f5e900ce140 msgr2=0x7f5e900d2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.195+0000 7f5e9490c640 1 --2- 172.21.15.116:0/1249279868 >> v2:172.21.15.156:3300/0 conn(0x7f5e900ce140 0x7f5e900d2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.195+0000 7f5e9490c640 1 -- 172.21.15.116:0/1249279868 >> v1:172.21.15.156:6789/0 conn(0x7f5e9015d2c0 legacy=0x7f5e900cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.195+0000 7f5e9490c640 1 -- 172.21.15.116:0/1249279868 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e900d3760 con 0x7f5e900d2d70 2023-11-28T17:09:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.198+0000 7f5e9490c640 1 --2- 172.21.15.116:0/1249279868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e900d2d70 0x7f5e900b6ad0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5e90160ea0 tx=0x7f5e8802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaa62a17db9635c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.198+0000 7fc0af80e640 1 Processor -- start 2023-11-28T17:09:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.198+0000 7f5e8effd640 1 -- 172.21.15.116:0/1249279868 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8802e070 con 0x7f5e900d2d70 2023-11-28T17:09:10.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.199+0000 7f5e8effd640 1 -- 172.21.15.116:0/1249279868 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e880315a0 con 0x7f5e900d2d70 2023-11-28T17:09:10.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.199+0000 7f5e8effd640 1 -- 172.21.15.116:0/1249279868 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88031930 con 0x7f5e900d2d70 2023-11-28T17:09:10.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.200+0000 7f5e9636f640 1 -- 172.21.15.116:0/1249279868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e900d2d70 msgr2=0x7f5e900b6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.200+0000 7f5e9636f640 1 --2- 172.21.15.116:0/1249279868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e900d2d70 0x7f5e900b6ad0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5e90160ea0 tx=0x7f5e8802b840 comp rx=0 tx=0).stop 2023-11-28T17:09:10.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.198+0000 7fc0af80e640 1 -- start start 2023-11-28T17:09:10.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.200+0000 7fc0af80e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc0a80b3850 0x7fc0a80cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.200+0000 7fc0af80e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc0a80cb650 0x7fc0a80703f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.201+0000 7fc0ad5aa640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc0a80b3850 0x7fc0a80cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.201+0000 7fc0ad5aa640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc0a80b3850 0x7fc0a80cd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49584/0 (socket says 172.21.15.116:49584) 2023-11-28T17:09:10.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.201+0000 7f5e9636f640 1 -- 172.21.15.116:0/1249279868 shutdown_connections 2023-11-28T17:09:10.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.201+0000 7f5e9636f640 1 --2- 172.21.15.116:0/1249279868 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e900d2d70 0x7f5e900b6ad0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.201+0000 7f5e9636f640 1 --2- 172.21.15.116:0/1249279868 >> v2:172.21.15.156:3300/0 conn(0x7f5e900ce140 0x7f5e900d2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.201+0000 7f5e9636f640 1 -- 172.21.15.116:0/1249279868 >> 172.21.15.116:0/1249279868 conn(0x7f5e90072010 msgr2=0x7f5e90072420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.202+0000 7f5e9636f640 1 -- 172.21.15.116:0/1249279868 shutdown_connections 2023-11-28T17:09:10.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.202+0000 7f5e9636f640 1 -- 172.21.15.116:0/1249279868 wait complete. 2023-11-28T17:09:10.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.201+0000 7fc0ad5aa640 1 -- 172.21.15.116:0/3556588158 learned_addr learned my addr 172.21.15.116:0/3556588158 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.201+0000 7fc0af80e640 1 -- 172.21.15.116:0/3556588158 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0a80b2770 con 0x7fc0a81594f0 2023-11-28T17:09:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.202+0000 7fc0af80e640 1 -- 172.21.15.116:0/3556588158 --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0a80ce090 con 0x7fc0a80cb650 2023-11-28T17:09:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.202+0000 7fc0addab640 1 --2- 172.21.15.116:0/3556588158 >> v2:172.21.15.156:3300/0 conn(0x7fc0a80cb650 0x7fc0a80703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.202+0000 7fc0af80e640 1 -- 172.21.15.116:0/3556588158 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a80cddb0 con 0x7fc0a80b3850 2023-11-28T17:09:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.203+0000 7fc0addab640 1 -- 172.21.15.116:0/3556588158 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc0a80b3850 msgr2=0x7fc0a80cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.203+0000 7fc0addab640 1 --2- 172.21.15.116:0/3556588158 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc0a80b3850 0x7fc0a80cd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.203+0000 7fc0addab640 1 -- 172.21.15.116:0/3556588158 >> v1:172.21.15.116:6789/0 conn(0x7fc0a81594f0 legacy=0x7fc0a80b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.203+0000 7fc0addab640 1 -- 172.21.15.116:0/3556588158 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a80b2120 con 0x7fc0a80cb650 2023-11-28T17:09:10.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.205+0000 7fc0addab640 1 --2- 172.21.15.116:0/3556588158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0a80cb650 0x7fc0a80703f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0009d90 tx=0x7fc0a002bcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71c40692b2089ac9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.206+0000 7fc09ffff640 1 -- 172.21.15.116:0/3556588158 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a002caa0 con 0x7fc0a80cb650 2023-11-28T17:09:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.206+0000 7fc09ffff640 1 -- 172.21.15.116:0/3556588158 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a002cc00 con 0x7fc0a80cb650 2023-11-28T17:09:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.206+0000 7fc09ffff640 1 -- 172.21.15.116:0/3556588158 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a0034440 con 0x7fc0a80cb650 2023-11-28T17:09:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.206+0000 7f5e9636f640 1 Processor -- start 2023-11-28T17:09:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.207+0000 7f5e9636f640 1 -- start start 2023-11-28T17:09:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.207+0000 7fc0af80e640 1 -- 172.21.15.116:0/3556588158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0a80cb650 msgr2=0x7fc0a80703f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.207+0000 7fc0af80e640 1 --2- 172.21.15.116:0/3556588158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0a80cb650 0x7fc0a80703f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0009d90 tx=0x7fc0a002bcf0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.207+0000 7fc0af80e640 1 -- 172.21.15.116:0/3556588158 shutdown_connections 2023-11-28T17:09:10.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.207+0000 7fc0af80e640 1 --2- 172.21.15.116:0/3556588158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0a80cb650 0x7fc0a80703f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.207+0000 7fc0af80e640 1 --2- 172.21.15.116:0/3556588158 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc0a80b3850 0x7fc0a80cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.207+0000 7fc0af80e640 1 -- 172.21.15.116:0/3556588158 >> 172.21.15.116:0/3556588158 conn(0x7fc0a806c250 msgr2=0x7fc0a806c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.208+0000 7fc0af80e640 1 -- 172.21.15.116:0/3556588158 shutdown_connections 2023-11-28T17:09:10.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.208+0000 7fc0af80e640 1 -- 172.21.15.116:0/3556588158 wait complete. 2023-11-28T17:09:10.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.209+0000 7f5e9636f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e900ce140 0x7f5e901f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.209+0000 7f5e9636f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e900d2d70 0x7f5e9015f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.209+0000 7f5e9636f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9015d2c0 0x7f5e901f53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.209+0000 7f5e9636f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900ca730 con 0x7f5e9015d2c0 2023-11-28T17:09:10.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.209+0000 7f5e9636f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900caa10 con 0x7f5e900ce140 2023-11-28T17:09:10.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.209+0000 7f5e9636f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900cacf0 con 0x7f5e900d2d70 2023-11-28T17:09:10.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.210+0000 7f5e9490c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9015d2c0 0x7f5e901f53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.211+0000 7f5e9490c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9015d2c0 0x7f5e901f53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51996/0 (socket says 172.21.15.116:51996) 2023-11-28T17:09:10.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.211+0000 7f5e9490c640 1 -- 172.21.15.116:0/3421873621 learned_addr learned my addr 172.21.15.116:0/3421873621 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:10.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.212+0000 7f5e8f7fe640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e900d2d70 0x7f5e9015f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.211+0000 7fc0af80e640 1 Processor -- start 2023-11-28T17:09:10.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.212+0000 7fc0af80e640 1 -- start start 2023-11-28T17:09:10.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.213+0000 7f5e9490c640 1 -- 172.21.15.116:0/3421873621 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e900d2d70 msgr2=0x7f5e9015f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.213+0000 7f5e9490c640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e900d2d70 0x7f5e9015f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.213+0000 7fc0af80e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc0a80b3850 0x7fc0a8158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.213+0000 7fc0af80e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a80cb650 0x7fc0a80b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.213+0000 7fc0af80e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0a8143800 0x7fc0a814e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.213+0000 7fc0af80e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a815a600 con 0x7fc0a80cb650 2023-11-28T17:09:10.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.213+0000 7fc0af80e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a814de00 con 0x7fc0a8143800 2023-11-28T17:09:10.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.213+0000 7fc0af80e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a80b09f0 con 0x7fc0a80b3850 2023-11-28T17:09:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.213+0000 7f5e8ffff640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e900ce140 0x7f5e901f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.213+0000 7f5e9490c640 1 -- 172.21.15.116:0/3421873621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e900ce140 msgr2=0x7f5e901f29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.214+0000 7f5e9490c640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e900ce140 0x7f5e901f29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.214+0000 7fc0addab640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0a8143800 0x7fc0a814e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.214+0000 7fc0addab640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0a8143800 0x7fc0a814e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:59112/0 (socket says 172.21.15.116:59112) 2023-11-28T17:09:10.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.214+0000 7fc0addab640 1 -- 172.21.15.116:0/3934204890 learned_addr learned my addr 172.21.15.116:0/3934204890 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:10.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.214+0000 7f5e9490c640 1 -- 172.21.15.116:0/3421873621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e88009400 con 0x7f5e9015d2c0 2023-11-28T17:09:10.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.214+0000 7f5e8f7fe640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e900d2d70 0x7f5e9015f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:09:10.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.215+0000 7f5e8ffff640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e900ce140 0x7f5e901f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:10.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.215+0000 7f5e9490c640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9015d2c0 0x7f5e901f53a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5e88009890 tx=0x7f5e880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.214+0000 7fc0ad5aa640 1 --2- 172.21.15.116:0/3934204890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc0a80b3850 0x7fc0a8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.214+0000 7fc0acda9640 1 --2- 172.21.15.116:0/3934204890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a80cb650 0x7fc0a80b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.215+0000 7fc0acda9640 1 -- 172.21.15.116:0/3934204890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc0a80b3850 msgr2=0x7fc0a8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.215+0000 7fc0acda9640 1 --2- 172.21.15.116:0/3934204890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc0a80b3850 0x7fc0a8158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.215+0000 7fc0acda9640 1 -- 172.21.15.116:0/3934204890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0a8143800 msgr2=0x7fc0a814e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.215+0000 7fc0acda9640 1 --2- 172.21.15.116:0/3934204890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0a8143800 0x7fc0a814e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.215+0000 7fc0acda9640 1 -- 172.21.15.116:0/3934204890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a0009a70 con 0x7fc0a80cb650 2023-11-28T17:09:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.216+0000 7fc0acda9640 1 --2- 172.21.15.116:0/3934204890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a80cb650 0x7fc0a80b04b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4001b30 tx=0x7fc0a400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.216+0000 7fc09dffb640 1 -- 172.21.15.116:0/3934204890 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a400ad50 con 0x7fc0a80cb650 2023-11-28T17:09:10.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.216+0000 7f5e8cff9640 1 -- 172.21.15.116:0/3421873621 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88032770 con 0x7f5e9015d2c0 2023-11-28T17:09:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.217+0000 7f5e8cff9640 1 -- 172.21.15.116:0/3421873621 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e880328d0 con 0x7f5e9015d2c0 2023-11-28T17:09:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.217+0000 7f5e8cff9640 1 -- 172.21.15.116:0/3421873621 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e88032c40 con 0x7f5e9015d2c0 2023-11-28T17:09:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.217+0000 7f5e9636f640 1 -- 172.21.15.116:0/3421873621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e900cafd0 con 0x7f5e9015d2c0 2023-11-28T17:09:10.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.218+0000 7f5e9636f640 1 -- 172.21.15.116:0/3421873621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9015fe80 con 0x7f5e9015d2c0 2023-11-28T17:09:10.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.217+0000 7fc09dffb640 1 -- 172.21.15.116:0/3934204890 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a40138e0 con 0x7fc0a80cb650 2023-11-28T17:09:10.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.217+0000 7fc09dffb640 1 -- 172.21.15.116:0/3934204890 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a400ad50 con 0x7fc0a80cb650 2023-11-28T17:09:10.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.217+0000 7fc0af80e640 1 -- 172.21.15.116:0/3934204890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a8159980 con 0x7fc0a80cb650 2023-11-28T17:09:10.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.219+0000 7fc0af80e640 1 -- 172.21.15.116:0/3934204890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a8159f40 con 0x7fc0a80cb650 2023-11-28T17:09:10.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.221+0000 7fc0af80e640 1 -- 172.21.15.116:0/3934204890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a815a740 con 0x7fc0a80cb650 2023-11-28T17:09:10.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.221+0000 7f5e8cff9640 1 -- 172.21.15.116:0/3421873621 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f5e8804e430 con 0x7f5e9015d2c0 2023-11-28T17:09:10.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.223+0000 7f5e9636f640 1 -- 172.21.15.116:0/3421873621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e90160010 con 0x7f5e9015d2c0 2023-11-28T17:09:10.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.224+0000 7f5e8cff9640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5e7006b2a0 0x7f5e7006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.224+0000 7f5e8cff9640 1 -- 172.21.15.116:0/3421873621 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e880abc70 con 0x7f5e9015d2c0 2023-11-28T17:09:10.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.225+0000 7f5e8ffff640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5e7006b2a0 0x7f5e7006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.224+0000 7fc09dffb640 1 -- 172.21.15.116:0/3934204890 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fc0a4013a50 con 0x7fc0a80cb650 2023-11-28T17:09:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.226+0000 7fc09dffb640 1 --2- 172.21.15.116:0/3934204890 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc08c06b330 0x7fc08c06d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.226+0000 7fc0ad5aa640 1 --2- 172.21.15.116:0/3934204890 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc08c06b330 0x7fc08c06d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.227+0000 7f5e8ffff640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5e7006b2a0 0x7f5e7006d760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5e78000f60 tx=0x7f5e78009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.228+0000 7fc0ad5aa640 1 --2- 172.21.15.116:0/3934204890 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc08c06b330 0x7fc08c06d7f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc098009b30 tx=0x7fc098009240 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.228+0000 7fc09dffb640 1 -- 172.21.15.116:0/3934204890 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0a405ff50 con 0x7fc0a80cb650 2023-11-28T17:09:10.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.237+0000 7f5e8cff9640 1 -- 172.21.15.116:0/3421873621 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e88076a40 con 0x7f5e9015d2c0 2023-11-28T17:09:10.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.245+0000 7fc09dffb640 1 -- 172.21.15.116:0/3934204890 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc0a405ca20 con 0x7fc0a80cb650 2023-11-28T17:09:10.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.269+0000 7f46aa14c640 1 Processor -- start 2023-11-28T17:09:10.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.269+0000 7f46aa14c640 1 -- start start 2023-11-28T17:09:10.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.271+0000 7f46aa14c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f46a415d420 0x7f46a40d1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.271+0000 7f46aa14c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f46a40ce760 0x7f46a40d1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.271+0000 7f46aa14c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a40c6b40 con 0x7f46a40d0d30 2023-11-28T17:09:10.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.271+0000 7f46aa14c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a415d800 con 0x7f46a40ce760 2023-11-28T17:09:10.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.271+0000 7f46aa14c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a40cece0 con 0x7f46a415d420 2023-11-28T17:09:10.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.272+0000 7f2870c2e640 1 Processor -- start 2023-11-28T17:09:10.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.272+0000 7f2870c2e640 1 -- start start 2023-11-28T17:09:10.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.272+0000 7f46a37fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f46a415d420 0x7f46a40d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.272+0000 7f46a37fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f46a415d420 0x7f46a40d1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49612/0 (socket says 172.21.15.116:49612) 2023-11-28T17:09:10.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.272+0000 7f46a37fe640 1 -- 172.21.15.116:0/2939636805 learned_addr learned my addr 172.21.15.116:0/2939636805 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:10.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.273+0000 7f469affd640 1 --2- 172.21.15.116:0/2939636805 >> v2:172.21.15.156:3300/0 conn(0x7f46a40ce760 0x7f46a40d1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.274+0000 7f46a2ffd640 1 -- 172.21.15.116:0/2939636805 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873523238 0 0) 0x7f46a40c6b40 con 0x7f46a40d0d30 2023-11-28T17:09:10.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.275+0000 7f46a2ffd640 1 -- 172.21.15.116:0/2939636805 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4690003520 con 0x7f46a40d0d30 2023-11-28T17:09:10.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.275+0000 7f469affd640 1 -- 172.21.15.116:0/2939636805 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f46a415d420 msgr2=0x7f46a40d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:10.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.275+0000 7f469affd640 1 --2- 172.21.15.116:0/2939636805 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f46a415d420 0x7f46a40d1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.275+0000 7f469affd640 1 -- 172.21.15.116:0/2939636805 >> v1:172.21.15.116:6789/0 conn(0x7f46a40d0d30 legacy=0x7f46a40cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.275+0000 7f469affd640 1 -- 172.21.15.116:0/2939636805 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a40c6db0 con 0x7f46a40ce760 2023-11-28T17:09:10.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.275+0000 7f2870c2e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640dfe90 0x7f28640525d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.275+0000 7f2870c2e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f28640398a0 0x7f286404fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.275+0000 7f2870c2e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2864052b10 0x7f2864050070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.276+0000 7f2870c2e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2864039da0 con 0x7f28640398a0 2023-11-28T17:09:10.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.276+0000 7f2870c2e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286403a080 con 0x7f2864052b10 2023-11-28T17:09:10.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.276+0000 7f2870c2e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286404f050 con 0x7f28640dfe90 2023-11-28T17:09:10.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.277+0000 7f99e4839640 1 Processor -- start 2023-11-28T17:09:10.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.277+0000 7f99e4839640 1 -- start start 2023-11-28T17:09:10.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.277+0000 7f286bfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2864052b10 0x7f2864050070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.278+0000 7f286b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640dfe90 0x7f28640525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.278+0000 7f469affd640 1 --2- 172.21.15.116:0/2939636805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46a40ce760 0x7f46a40d1a50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4694009d90 tx=0x7f469402bc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68c48207d7592060 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.278+0000 7f286b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640dfe90 0x7f28640525d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49618/0 (socket says 172.21.15.116:49618) 2023-11-28T17:09:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.278+0000 7f286b7fe640 1 -- 172.21.15.116:0/4220337095 learned_addr learned my addr 172.21.15.116:0/4220337095 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.278+0000 7f286affd640 1 --2- 172.21.15.116:0/4220337095 >> v2:172.21.15.116:3300/0 conn(0x7f28640398a0 0x7f286404fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.279+0000 7f286bfff640 1 -- 172.21.15.116:0/4220337095 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640dfe90 msgr2=0x7f28640525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.279+0000 7f286bfff640 1 --2- 172.21.15.116:0/4220337095 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640dfe90 0x7f28640525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.279+0000 7f286bfff640 1 -- 172.21.15.116:0/4220337095 >> v2:172.21.15.116:3300/0 conn(0x7f28640398a0 msgr2=0x7f286404fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.279+0000 7f286bfff640 1 --2- 172.21.15.116:0/4220337095 >> v2:172.21.15.116:3300/0 conn(0x7f28640398a0 0x7f286404fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.279+0000 7f286bfff640 1 -- 172.21.15.116:0/4220337095 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286404f330 con 0x7f2864052b10 2023-11-28T17:09:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.279+0000 7f99e4839640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99e0158e30 0x7f99e00b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.279+0000 7f286affd640 1 --2- 172.21.15.116:0/4220337095 >> v2:172.21.15.116:3300/0 conn(0x7f28640398a0 0x7f286404fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.279+0000 7f46a2ffd640 1 -- 172.21.15.116:0/2939636805 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469c001670 con 0x7f46a40ce760 2023-11-28T17:09:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.279+0000 7f46a2ffd640 1 -- 172.21.15.116:0/2939636805 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469402c7c0 con 0x7f46a40ce760 2023-11-28T17:09:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.279+0000 7f46a2ffd640 1 -- 172.21.15.116:0/2939636805 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469402cb50 con 0x7f46a40ce760 2023-11-28T17:09:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.280+0000 7f99e4839640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f99e00ce110 0x7f99e00ce560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.280+0000 7f99e4839640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e00ceaa0 con 0x7f99e00cc8a0 2023-11-28T17:09:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.280+0000 7f99e4839640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e00ccd00 con 0x7f99e00ce110 2023-11-28T17:09:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.280+0000 7f99e4839640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e0159920 con 0x7f99e0158e30 2023-11-28T17:09:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.280+0000 7f99dffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f99e00ce110 0x7f99e00ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.280+0000 7f99deffd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f99e00cc8a0 0x7f99e0159210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.116:50646/0 (socket says 172.21.15.116:50646) 2023-11-28T17:09:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.280+0000 7f99deffd640 1 -- 172.21.15.116:0/2089992018 learned_addr learned my addr 172.21.15.116:0/2089992018 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:10.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.280+0000 7f99df7fe640 1 --2- 172.21.15.116:0/2089992018 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99e0158e30 0x7f99e00b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.281+0000 7f46aa14c640 1 -- 172.21.15.116:0/2939636805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46a40ce760 msgr2=0x7f46a40d1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.281+0000 7f46aa14c640 1 --2- 172.21.15.116:0/2939636805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46a40ce760 0x7f46a40d1a50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4694009d90 tx=0x7f469402bc90 comp rx=0 tx=0).stop 2023-11-28T17:09:10.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.281+0000 7f99df7fe640 1 -- 172.21.15.116:0/2089992018 >> v2:172.21.15.116:3300/0 conn(0x7f99e00ce110 msgr2=0x7f99e00ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:10.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.281+0000 7f99de7fc640 1 -- 172.21.15.116:0/2089992018 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224211331 0 0) 0x7f99d0001110 con 0x7f99e00cc8a0 2023-11-28T17:09:10.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.281+0000 7f99df7fe640 1 --2- 172.21.15.116:0/2089992018 >> v2:172.21.15.116:3300/0 conn(0x7f99e00ce110 0x7f99e00ce560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.282+0000 7f46aa14c640 1 -- 172.21.15.116:0/2939636805 shutdown_connections 2023-11-28T17:09:10.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.282+0000 7f286bfff640 1 --2- 172.21.15.116:0/4220337095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2864052b10 0x7f2864050070 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2864052fa0 tx=0x7f286002d930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4dc8aa422b4c61d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.281+0000 7f99df7fe640 1 -- 172.21.15.116:0/2089992018 >> v1:172.21.15.156:6789/0 conn(0x7f99e00cc8a0 legacy=0x7f99e0159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.281+0000 7f99df7fe640 1 -- 172.21.15.116:0/2089992018 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e0159b40 con 0x7f99e0158e30 2023-11-28T17:09:10.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.282+0000 7f99dffff640 1 --2- 172.21.15.116:0/2089992018 >> v2:172.21.15.116:3300/0 conn(0x7f99e00ce110 0x7f99e00ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:09:10.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.282+0000 7f46aa14c640 1 --2- 172.21.15.116:0/2939636805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46a40ce760 0x7f46a40d1a50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.283+0000 7f46aa14c640 1 --2- 172.21.15.116:0/2939636805 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f46a415d420 0x7f46a40d1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.283+0000 7f286a7fc640 1 -- 172.21.15.116:0/4220337095 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2860033890 con 0x7f2864052b10 2023-11-28T17:09:10.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.283+0000 7f286a7fc640 1 -- 172.21.15.116:0/4220337095 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28600339f0 con 0x7f2864052b10 2023-11-28T17:09:10.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.283+0000 7f286a7fc640 1 -- 172.21.15.116:0/4220337095 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2860033d80 con 0x7f2864052b10 2023-11-28T17:09:10.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.283+0000 7f46aa14c640 1 -- 172.21.15.116:0/2939636805 >> 172.21.15.116:0/2939636805 conn(0x7f46a40d2f50 msgr2=0x7f46a40d3360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.283+0000 7f46aa14c640 1 -- 172.21.15.116:0/2939636805 shutdown_connections 2023-11-28T17:09:10.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.283+0000 7f46aa14c640 1 -- 172.21.15.116:0/2939636805 wait complete. 2023-11-28T17:09:10.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.284+0000 7f99df7fe640 1 --2- 172.21.15.116:0/2089992018 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99e0158e30 0x7f99e00b47e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f99d80097a0 tx=0x7f99d802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bba144b7c2e477b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.284+0000 7f2870c2e640 1 -- 172.21.15.116:0/4220337095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2864052b10 msgr2=0x7f2864050070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.284+0000 7f2870c2e640 1 --2- 172.21.15.116:0/4220337095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2864052b10 0x7f2864050070 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2864052fa0 tx=0x7f286002d930 comp rx=0 tx=0).stop 2023-11-28T17:09:10.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.284+0000 7f6cae8d3640 1 Processor -- start 2023-11-28T17:09:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.284+0000 7f2870c2e640 1 -- 172.21.15.116:0/4220337095 shutdown_connections 2023-11-28T17:09:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.284+0000 7f2870c2e640 1 --2- 172.21.15.116:0/4220337095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2864052b10 0x7f2864050070 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.284+0000 7f2870c2e640 1 --2- 172.21.15.116:0/4220337095 >> v2:172.21.15.116:3300/0 conn(0x7f28640398a0 0x7f286404fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.284+0000 7f2870c2e640 1 --2- 172.21.15.116:0/4220337095 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640dfe90 0x7f28640525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.284+0000 7f2870c2e640 1 -- 172.21.15.116:0/4220337095 >> 172.21.15.116:0/4220337095 conn(0x7f2864009f80 msgr2=0x7f286400a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.286+0000 7f46aa14c640 1 Processor -- start 2023-11-28T17:09:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.286+0000 7f46aa14c640 1 -- start start 2023-11-28T17:09:10.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.285+0000 7f99de7fc640 1 -- 172.21.15.116:0/2089992018 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8031780 con 0x7f99e0158e30 2023-11-28T17:09:10.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.285+0000 7f99de7fc640 1 -- 172.21.15.116:0/2089992018 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d80318e0 con 0x7f99e0158e30 2023-11-28T17:09:10.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.285+0000 7f99de7fc640 1 -- 172.21.15.116:0/2089992018 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8031c70 con 0x7f99e0158e30 2023-11-28T17:09:10.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.286+0000 7f99e4839640 1 -- 172.21.15.116:0/2089992018 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99e0158e30 msgr2=0x7f99e00b47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.286+0000 7f99e4839640 1 --2- 172.21.15.116:0/2089992018 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99e0158e30 0x7f99e00b47e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f99d80097a0 tx=0x7f99d802b820 comp rx=0 tx=0).stop 2023-11-28T17:09:10.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.285+0000 7f2870c2e640 1 -- 172.21.15.116:0/4220337095 shutdown_connections 2023-11-28T17:09:10.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.285+0000 7f6cae8d3640 1 -- start start 2023-11-28T17:09:10.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.286+0000 7f6cae8d3640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6ca80b20a0 0x7f6ca80d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.287+0000 7f6cae8d3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6ca80b2ad0 0x7f6ca80b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.287+0000 7f6cae8d3640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ca80cdcb0 con 0x7f6ca80d1780 2023-11-28T17:09:10.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.287+0000 7f6cae8d3640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca80d39e0 con 0x7f6ca80b2ad0 2023-11-28T17:09:10.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.287+0000 7f6cae8d3640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ca80d3cc0 con 0x7f6ca80b20a0 2023-11-28T17:09:10.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.287+0000 7f6ca7fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6ca80d1780 0x7f6ca80d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54652/0 (socket says 172.21.15.116:54652) 2023-11-28T17:09:10.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.287+0000 7f6ca7fff640 1 -- 172.21.15.116:0/1229788556 learned_addr learned my addr 172.21.15.116:0/1229788556 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:10.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.287+0000 7f99e4839640 1 -- 172.21.15.116:0/2089992018 shutdown_connections 2023-11-28T17:09:10.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.287+0000 7f99e4839640 1 --2- 172.21.15.116:0/2089992018 >> v2:172.21.15.116:3300/0 conn(0x7f99e00ce110 0x7f99e00ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.287+0000 7f99e4839640 1 --2- 172.21.15.116:0/2089992018 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99e0158e30 0x7f99e00b47e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.287+0000 7f99e4839640 1 -- 172.21.15.116:0/2089992018 >> 172.21.15.116:0/2089992018 conn(0x7f99e00aea80 msgr2=0x7f99e0072250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.287+0000 7feb11636640 1 Processor -- start 2023-11-28T17:09:10.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.288+0000 7f99e4839640 1 -- 172.21.15.116:0/2089992018 shutdown_connections 2023-11-28T17:09:10.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.285+0000 7f2870c2e640 1 -- 172.21.15.116:0/4220337095 wait complete. 2023-11-28T17:09:10.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.288+0000 7feb11636640 1 -- start start 2023-11-28T17:09:10.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.288+0000 7f99e4839640 1 -- 172.21.15.116:0/2089992018 wait complete. 2023-11-28T17:09:10.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.288+0000 7f46aa14c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46a40ce760 0x7f46a4151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.288+0000 7f46aa14c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46a40d0d30 0x7f46a40b8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.288+0000 7f46aa14c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f46a415d420 0x7f46a4156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.288+0000 7f46aa14c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a415eeb0 con 0x7f46a40ce760 2023-11-28T17:09:10.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.288+0000 7f46aa14c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a415f190 con 0x7f46a40d0d30 2023-11-28T17:09:10.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.288+0000 7f46aa14c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a415f470 con 0x7f46a415d420 2023-11-28T17:09:10.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.287+0000 7f6ca77fe640 1 --2- 172.21.15.116:0/1229788556 >> v2:172.21.15.156:3300/0 conn(0x7f6ca80b20a0 0x7f6ca80d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.288+0000 7f2870c2e640 1 Processor -- start 2023-11-28T17:09:10.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.289+0000 7f2870c2e640 1 -- start start 2023-11-28T17:09:10.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.288+0000 7f6cace70640 1 --2- 172.21.15.116:0/1229788556 >> v2:172.21.15.116:3300/0 conn(0x7f6ca80b2ad0 0x7f6ca80b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.289+0000 7f6ca6ffd640 1 -- 172.21.15.116:0/1229788556 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133473566 0 0) 0x7f6ca80cdcb0 con 0x7f6ca80d1780 2023-11-28T17:09:10.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.289+0000 7f6ca6ffd640 1 -- 172.21.15.116:0/1229788556 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c90003520 con 0x7f6ca80d1780 2023-11-28T17:09:10.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.289+0000 7f6ca77fe640 1 -- 172.21.15.116:0/1229788556 >> v2:172.21.15.116:3300/0 conn(0x7f6ca80b2ad0 msgr2=0x7f6ca80b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:10.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.289+0000 7f6ca77fe640 1 --2- 172.21.15.116:0/1229788556 >> v2:172.21.15.116:3300/0 conn(0x7f6ca80b2ad0 0x7f6ca80b2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.289+0000 7f6ca77fe640 1 -- 172.21.15.116:0/1229788556 >> v1:172.21.15.116:6789/0 conn(0x7f6ca80d1780 legacy=0x7f6ca80d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.289+0000 7f6ca77fe640 1 -- 172.21.15.116:0/1229788556 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca80cdf50 con 0x7f6ca80b20a0 2023-11-28T17:09:10.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.290+0000 7feb11636640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feb0c0722a0 0x7feb0c15f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.290+0000 7feb0b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feb0c0722a0 0x7feb0c15f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.290+0000 7feb0b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feb0c0722a0 0x7feb0c15f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52048/0 (socket says 172.21.15.116:52048) 2023-11-28T17:09:10.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.290+0000 7feb0b7fe640 1 -- 172.21.15.116:0/1137660894 learned_addr learned my addr 172.21.15.116:0/1137660894 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:10.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.290+0000 7f2870c2e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640398a0 0x7f286404a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.289+0000 7f469affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46a40d0d30 0x7f46a40b8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.289+0000 7f469affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46a40d0d30 0x7f46a40b8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:59148/0 (socket says 172.21.15.116:59148) 2023-11-28T17:09:10.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.289+0000 7f469affd640 1 -- 172.21.15.116:0/1957618393 learned_addr learned my addr 172.21.15.116:0/1957618393 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:10.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.289+0000 7f46a37fe640 1 --2- 172.21.15.116:0/1957618393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46a40ce760 0x7f46a4151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.290+0000 7f46a3fff640 1 --2- 172.21.15.116:0/1957618393 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f46a415d420 0x7f46a4156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.291+0000 7f46a37fe640 1 -- 172.21.15.116:0/1957618393 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f46a415d420 msgr2=0x7f46a4156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.291+0000 7f46a37fe640 1 --2- 172.21.15.116:0/1957618393 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f46a415d420 0x7f46a4156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.291+0000 7f46a37fe640 1 -- 172.21.15.116:0/1957618393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46a40d0d30 msgr2=0x7f46a40b8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.291+0000 7f46a37fe640 1 --2- 172.21.15.116:0/1957618393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46a40d0d30 0x7f46a40b8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.291+0000 7feb11636640 1 -- 172.21.15.116:0/1137660894 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0c0cecf0 con 0x7feb0c15ec40 2023-11-28T17:09:10.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.291+0000 7feb11636640 1 -- 172.21.15.116:0/1137660894 --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0c070870 con 0x7feb0c0703f0 2023-11-28T17:09:10.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.291+0000 7feb11636640 1 -- 172.21.15.116:0/1137660894 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0c070af0 con 0x7feb0c0722a0 2023-11-28T17:09:10.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.291+0000 7f46a37fe640 1 -- 172.21.15.116:0/1957618393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4694009a70 con 0x7f46a40ce760 2023-11-28T17:09:10.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.290+0000 7f2870c2e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2864052b10 0x7f28640dcb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.291+0000 7f2870c2e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28640dfe90 0x7f28640dd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.291+0000 7f2870c2e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28640de2d0 con 0x7f2864052b10 2023-11-28T17:09:10.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.291+0000 7f2870c2e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2864185ea0 con 0x7f28640dfe90 2023-11-28T17:09:10.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.291+0000 7f2870c2e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28640dd6c0 con 0x7f28640398a0 2023-11-28T17:09:10.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7f286b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640398a0 0x7f286404a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7f286affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2864052b10 0x7f28640dcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.291+0000 7f6ca77fe640 1 --2- 172.21.15.116:0/1229788556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca80b20a0 0x7f6ca80d01e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8162190 tx=0x7f6c9c02b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4df43ea804ab6b1f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7f46a37fe640 1 --2- 172.21.15.116:0/1957618393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46a40ce760 0x7f46a4151d60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f468c00e870 tx=0x7f468c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7f286bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28640dfe90 0x7f28640dd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7f286b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640398a0 0x7f286404a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49648/0 (socket says 172.21.15.116:49648) 2023-11-28T17:09:10.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7f286b7fe640 1 -- 172.21.15.116:0/2994199901 learned_addr learned my addr 172.21.15.116:0/2994199901 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:10.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7f46a0ff9640 1 -- 172.21.15.116:0/1957618393 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c00ee60 con 0x7f46a40ce760 2023-11-28T17:09:10.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7feb0b7fe640 1 -- 172.21.15.116:0/1137660894 >> v1:172.21.15.156:6789/0 conn(0x7feb0c0703f0 legacy=0x7feb0c071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7feb09ffb640 1 -- 172.21.15.116:0/1137660894 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435618729 0 0) 0x7feb04001110 con 0x7feb0c15ec40 2023-11-28T17:09:10.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7feb0b7fe640 1 -- 172.21.15.116:0/1137660894 >> v1:172.21.15.116:6789/0 conn(0x7feb0c15ec40 legacy=0x7feb0c15b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7f46a0ff9640 1 -- 172.21.15.116:0/1957618393 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468c00c940 con 0x7f46a40ce760 2023-11-28T17:09:10.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7f46a0ff9640 1 -- 172.21.15.116:0/1957618393 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c00ccd0 con 0x7f46a40ce760 2023-11-28T17:09:10.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7f46aa14c640 1 -- 172.21.15.116:0/1957618393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a415f750 con 0x7f46a40ce760 2023-11-28T17:09:10.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7feb0b7fe640 1 -- 172.21.15.116:0/1137660894 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c070dd0 con 0x7feb0c0722a0 2023-11-28T17:09:10.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7f6ca6ffd640 1 -- 172.21.15.116:0/1229788556 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c031520 con 0x7f6ca80b20a0 2023-11-28T17:09:10.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.293+0000 7f6ca6ffd640 1 -- 172.21.15.116:0/1229788556 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9c031680 con 0x7f6ca80b20a0 2023-11-28T17:09:10.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.293+0000 7f6ca6ffd640 1 -- 172.21.15.116:0/1229788556 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c0319f0 con 0x7f6ca80b20a0 2023-11-28T17:09:10.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7f99e4839640 1 Processor -- start 2023-11-28T17:09:10.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.292+0000 7f46aa14c640 1 -- 172.21.15.116:0/1957618393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a41522a0 con 0x7f46a40ce760 2023-11-28T17:09:10.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.293+0000 7f286b7fe640 1 -- 172.21.15.116:0/2994199901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28640dfe90 msgr2=0x7f28640dd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.293+0000 7f286b7fe640 1 --2- 172.21.15.116:0/2994199901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28640dfe90 0x7f28640dd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.294+0000 7f46aa14c640 1 -- 172.21.15.116:0/1957618393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a40cefa0 con 0x7f46a40ce760 2023-11-28T17:09:10.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.293+0000 7f286b7fe640 1 -- 172.21.15.116:0/2994199901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2864052b10 msgr2=0x7f28640dcb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.293+0000 7f286b7fe640 1 --2- 172.21.15.116:0/2994199901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2864052b10 0x7f28640dcb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.293+0000 7f286b7fe640 1 -- 172.21.15.116:0/2994199901 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2860009400 con 0x7f28640398a0 2023-11-28T17:09:10.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.293+0000 7f286b7fe640 1 --2- 172.21.15.116:0/2994199901 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640398a0 0x7f286404a2c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2858009fd0 tx=0x7f285800ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.294+0000 7feb0b7fe640 1 --2- 172.21.15.116:0/1137660894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c0722a0 0x7feb0c15f020 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7feafc009e50 tx=0x7feafc02bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77c3aacbbf29879 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.294+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1229788556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca80b20a0 msgr2=0x7f6ca80d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.294+0000 7f6cae8d3640 1 --2- 172.21.15.116:0/1229788556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca80b20a0 0x7f6ca80d01e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8162190 tx=0x7f6c9c02b7c0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.294+0000 7f99e4839640 1 -- start start 2023-11-28T17:09:10.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.295+0000 7f2857fff640 1 -- 172.21.15.116:0/2994199901 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285800c940 con 0x7f28640398a0 2023-11-28T17:09:10.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.295+0000 7f2857fff640 1 -- 172.21.15.116:0/2994199901 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285800caa0 con 0x7f28640398a0 2023-11-28T17:09:10.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.295+0000 7f2857fff640 1 -- 172.21.15.116:0/2994199901 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285800ce30 con 0x7f28640398a0 2023-11-28T17:09:10.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.295+0000 7f2870c2e640 1 -- 172.21.15.116:0/2994199901 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28640dd9a0 con 0x7f28640398a0 2023-11-28T17:09:10.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.295+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1229788556 shutdown_connections 2023-11-28T17:09:10.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.295+0000 7f6cae8d3640 1 --2- 172.21.15.116:0/1229788556 >> v2:172.21.15.116:3300/0 conn(0x7f6ca80b2ad0 0x7f6ca80b2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.295+0000 7f6cae8d3640 1 --2- 172.21.15.116:0/1229788556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca80b20a0 0x7f6ca80d01e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.295+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1229788556 >> 172.21.15.116:0/1229788556 conn(0x7f6ca809c4e0 msgr2=0x7f6ca80d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.295+0000 7f2870c2e640 1 -- 172.21.15.116:0/2994199901 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28640ddf60 con 0x7f28640398a0 2023-11-28T17:09:10.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.295+0000 7f99e4839640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e00cc8a0 0x7f99e014c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.296+0000 7f99e4839640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99e00ce110 0x7f99e0140760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.296+0000 7f99e4839640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99e0158e30 0x7f99e015b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.296+0000 7f99e4839640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e0140010 con 0x7f99e00cc8a0 2023-11-28T17:09:10.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.296+0000 7f99e4839640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e013f140 con 0x7f99e0158e30 2023-11-28T17:09:10.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.296+0000 7f99e4839640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e013f400 con 0x7f99e00ce110 2023-11-28T17:09:10.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.296+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1229788556 shutdown_connections 2023-11-28T17:09:10.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.296+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1229788556 wait complete. 2023-11-28T17:09:10.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.296+0000 7f99dffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99e0158e30 0x7f99e015b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.296+0000 7f99dffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99e0158e30 0x7f99e015b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:59152/0 (socket says 172.21.15.116:59152) 2023-11-28T17:09:10.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.297+0000 7f99dffff640 1 -- 172.21.15.116:0/1962744190 learned_addr learned my addr 172.21.15.116:0/1962744190 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:10.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.297+0000 7f99deffd640 1 --2- 172.21.15.116:0/1962744190 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99e00ce110 0x7f99e0140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.297+0000 7f99df7fe640 1 --2- 172.21.15.116:0/1962744190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e00cc8a0 0x7f99e014c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.297+0000 7f2855ffb640 1 -- 172.21.15.116:0/2994199901 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28640e1430 con 0x7f28640398a0 2023-11-28T17:09:10.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.297+0000 7f46a0ff9640 1 -- 172.21.15.116:0/1957618393 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f468c013470 con 0x7f46a40ce760 2023-11-28T17:09:10.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.297+0000 7f99df7fe640 1 -- 172.21.15.116:0/1962744190 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99e00ce110 msgr2=0x7f99e0140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.297+0000 7f99df7fe640 1 --2- 172.21.15.116:0/1962744190 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99e00ce110 0x7f99e0140760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.297+0000 7f99df7fe640 1 -- 172.21.15.116:0/1962744190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99e0158e30 msgr2=0x7f99e015b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.297+0000 7f99df7fe640 1 --2- 172.21.15.116:0/1962744190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99e0158e30 0x7f99e015b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.297+0000 7f99df7fe640 1 -- 172.21.15.116:0/1962744190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c8009a70 con 0x7f99e00cc8a0 2023-11-28T17:09:10.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.298+0000 7f46a0ff9640 1 --2- 172.21.15.116:0/1957618393 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f468406b2a0 0x7f468406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.298+0000 7f46a0ff9640 1 -- 172.21.15.116:0/1957618393 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f468c08a1e0 con 0x7f46a40ce760 2023-11-28T17:09:10.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.298+0000 7f99df7fe640 1 --2- 172.21.15.116:0/1962744190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e00cc8a0 0x7f99e014c490 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f99e0072bf0 tx=0x7f99d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.298+0000 7feb09ffb640 1 -- 172.21.15.116:0/1137660894 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc02cb30 con 0x7feb0c0722a0 2023-11-28T17:09:10.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.299+0000 7feb09ffb640 1 -- 172.21.15.116:0/1137660894 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feafc02cc90 con 0x7feb0c0722a0 2023-11-28T17:09:10.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.299+0000 7feb09ffb640 1 -- 172.21.15.116:0/1137660894 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc034610 con 0x7feb0c0722a0 2023-11-28T17:09:10.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.299+0000 7f6cae8d3640 1 Processor -- start 2023-11-28T17:09:10.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.299+0000 7f469affd640 1 --2- 172.21.15.116:0/1957618393 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f468406b2a0 0x7f468406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.299+0000 7feb11636640 1 -- 172.21.15.116:0/1137660894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c0722a0 msgr2=0x7feb0c15f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.299+0000 7feb11636640 1 --2- 172.21.15.116:0/1137660894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c0722a0 0x7feb0c15f020 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7feafc009e50 tx=0x7feafc02bf90 comp rx=0 tx=0).stop 2023-11-28T17:09:10.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.300+0000 7feb11636640 1 -- 172.21.15.116:0/1137660894 shutdown_connections 2023-11-28T17:09:10.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.300+0000 7feb11636640 1 --2- 172.21.15.116:0/1137660894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c0722a0 0x7feb0c15f020 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.300+0000 7feb11636640 1 -- 172.21.15.116:0/1137660894 >> 172.21.15.116:0/1137660894 conn(0x7feb0c06c250 msgr2=0x7feb0c06c660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.300+0000 7f6cae8d3640 1 -- start start 2023-11-28T17:09:10.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.300+0000 7f469affd640 1 --2- 172.21.15.116:0/1957618393 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f468406b2a0 0x7f468406d760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f46a4159700 tx=0x7f4694030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.301+0000 7feb11636640 1 -- 172.21.15.116:0/1137660894 shutdown_connections 2023-11-28T17:09:10.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.301+0000 7feb11636640 1 -- 172.21.15.116:0/1137660894 wait complete. 2023-11-28T17:09:10.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.301+0000 7f6cae8d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca80b20a0 0x7f6ca8071000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.301+0000 7f6cae8d3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca80b2ad0 0x7f6ca80b77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.301+0000 7f6cae8d3640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca80d1780 0x7f6ca80b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.301+0000 7f6cae8d3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca81eae20 con 0x7f6ca80b20a0 2023-11-28T17:09:10.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.301+0000 7f6cae8d3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80cc5b0 con 0x7f6ca80b2ad0 2023-11-28T17:09:10.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.301+0000 7f6cae8d3640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca80cc890 con 0x7f6ca80d1780 2023-11-28T17:09:10.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.302+0000 7f6ca7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca80b20a0 0x7f6ca8071000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.302+0000 7f6ca7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca80b20a0 0x7f6ca8071000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52078/0 (socket says 172.21.15.116:52078) 2023-11-28T17:09:10.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.302+0000 7f6ca7fff640 1 -- 172.21.15.116:0/1190807032 learned_addr learned my addr 172.21.15.116:0/1190807032 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:10.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.302+0000 7f6cace70640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca80d1780 0x7f6ca80b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.301+0000 7f2857fff640 1 -- 172.21.15.116:0/2994199901 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2858013470 con 0x7f28640398a0 2023-11-28T17:09:10.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.302+0000 7f2857fff640 1 --2- 172.21.15.116:0/2994199901 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f285006b370 0x7f285006d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.302+0000 7f6ca77fe640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca80b2ad0 0x7f6ca80b77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.302+0000 7f6ca7fff640 1 -- 172.21.15.116:0/1190807032 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca80d1780 msgr2=0x7f6ca80b7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.302+0000 7f6ca7fff640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca80d1780 0x7f6ca80b7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.302+0000 7f6ca7fff640 1 -- 172.21.15.116:0/1190807032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca80b2ad0 msgr2=0x7f6ca80b77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.302+0000 7f6ca7fff640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca80b2ad0 0x7f6ca80b77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.302+0000 7f6ca7fff640 1 -- 172.21.15.116:0/1190807032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c009400 con 0x7f6ca80b20a0 2023-11-28T17:09:10.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.303+0000 7f6ca7fff640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca80b20a0 0x7f6ca8071000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800c460 tx=0x7f6c9800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.303+0000 7f2857fff640 1 -- 172.21.15.116:0/2994199901 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2858089c50 con 0x7f28640398a0 2023-11-28T17:09:10.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.303+0000 7feb11636640 1 Processor -- start 2023-11-28T17:09:10.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.303+0000 7f286affd640 1 --2- 172.21.15.116:0/2994199901 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f285006b370 0x7f285006d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.303+0000 7f6ca77fe640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca80b2ad0 0x7f6ca80b77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:10.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.304+0000 7feb11636640 1 -- start start 2023-11-28T17:09:10.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.304+0000 7f6ca4ff9640 1 -- 172.21.15.116:0/1190807032 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c980074e0 con 0x7f6ca80b20a0 2023-11-28T17:09:10.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.304+0000 7f286affd640 1 --2- 172.21.15.116:0/2994199901 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f285006b370 0x7f285006d830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f286c0641c0 tx=0x7f286c06d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.304+0000 7f6ca4ff9640 1 -- 172.21.15.116:0/1190807032 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c98007640 con 0x7f6ca80b20a0 2023-11-28T17:09:10.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.304+0000 7f6ca4ff9640 1 -- 172.21.15.116:0/1190807032 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c980049e0 con 0x7f6ca80b20a0 2023-11-28T17:09:10.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.304+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1190807032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca80ccb70 con 0x7f6ca80b20a0 2023-11-28T17:09:10.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.305+0000 7f46a0ff9640 1 -- 172.21.15.116:0/1957618393 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f468c0550e0 con 0x7f46a40ce760 2023-11-28T17:09:10.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.304+0000 7f99d7fff640 1 -- 172.21.15.116:0/1962744190 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8032820 con 0x7f99e00cc8a0 2023-11-28T17:09:10.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.304+0000 7f99d7fff640 1 -- 172.21.15.116:0/1962744190 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d80329a0 con 0x7f99e00cc8a0 2023-11-28T17:09:10.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.305+0000 7feb11636640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb0c0703f0 0x7feb0c0cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.305+0000 7feb11636640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb0c0722a0 0x7feb0c0b2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.305+0000 7feb11636640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c15ec40 0x7feb0c159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.305+0000 7feb11636640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c0cc000 con 0x7feb0c15ec40 2023-11-28T17:09:10.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.305+0000 7feb11636640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c0cb9d0 con 0x7feb0c0703f0 2023-11-28T17:09:10.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.305+0000 7feb11636640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c15a310 con 0x7feb0c0722a0 2023-11-28T17:09:10.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.306+0000 7feb0affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb0c0703f0 0x7feb0c0cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.306+0000 7feb0a7fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb0c0722a0 0x7feb0c0b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.305+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1190807032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca80cd130 con 0x7f6ca80b20a0 2023-11-28T17:09:10.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.305+0000 7f99d7fff640 1 -- 172.21.15.116:0/1962744190 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8032d50 con 0x7f99e00cc8a0 2023-11-28T17:09:10.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.305+0000 7f99e4839640 1 -- 172.21.15.116:0/1962744190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d8009400 con 0x7f99e00cc8a0 2023-11-28T17:09:10.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.305+0000 7f99e4839640 1 -- 172.21.15.116:0/1962744190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e013f9c0 con 0x7f99e00cc8a0 2023-11-28T17:09:10.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.306+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1190807032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca80693d0 con 0x7f6ca80b20a0 2023-11-28T17:09:10.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.306+0000 7feb0b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c15ec40 0x7feb0c159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.306+0000 7feb0b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c15ec40 0x7feb0c159dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52088/0 (socket says 172.21.15.116:52088) 2023-11-28T17:09:10.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.306+0000 7feb0b7fe640 1 -- 172.21.15.116:0/321043137 learned_addr learned my addr 172.21.15.116:0/321043137 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:10.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.307+0000 7f99e4839640 1 -- 172.21.15.116:0/1962744190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e004e880 con 0x7f99e00cc8a0 2023-11-28T17:09:10.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.307+0000 7f6cc942e640 1 Processor -- start 2023-11-28T17:09:10.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.307+0000 7f6cc942e640 1 -- start start 2023-11-28T17:09:10.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.308+0000 7f6cc942e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cbc050160 0x7f6cbc038040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.308+0000 7f6cc942e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6cbc053c30 0x7f6cbc0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.308+0000 7f6cc942e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cbc054480 con 0x7f6cbc0df1e0 2023-11-28T17:09:10.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.308+0000 7f6cc942e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cbc0541a0 con 0x7f6cbc053c30 2023-11-28T17:09:10.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.308+0000 7f6cc942e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cbc005ef0 con 0x7f6cbc050160 2023-11-28T17:09:10.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.307+0000 7feb0a7fc640 1 -- 172.21.15.116:0/321043137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb0c0703f0 msgr2=0x7feb0c0cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.307+0000 7feb0a7fc640 1 --2- 172.21.15.116:0/321043137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb0c0703f0 0x7feb0c0cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.307+0000 7feb0a7fc640 1 -- 172.21.15.116:0/321043137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c15ec40 msgr2=0x7feb0c159dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.307+0000 7feb0a7fc640 1 --2- 172.21.15.116:0/321043137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c15ec40 0x7feb0c159dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.307+0000 7feb0a7fc640 1 -- 172.21.15.116:0/321043137 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feafc009b30 con 0x7feb0c0722a0 2023-11-28T17:09:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.309+0000 7f6cc3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6cbc0df1e0 0x7f6cbc04fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54682/0 (socket says 172.21.15.116:54682) 2023-11-28T17:09:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.309+0000 7f6cc3fff640 1 -- 172.21.15.116:0/1393021359 learned_addr learned my addr 172.21.15.116:0/1393021359 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.309+0000 7f6cc8c2d640 1 --2- 172.21.15.116:0/1393021359 >> v2:172.21.15.116:3300/0 conn(0x7f6cbc053c30 0x7f6cbc0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.309+0000 7f6cc37fe640 1 --2- 172.21.15.116:0/1393021359 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cbc050160 0x7f6cbc038040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.309+0000 7f99d7fff640 1 -- 172.21.15.116:0/1962744190 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f99d804e450 con 0x7f99e00cc8a0 2023-11-28T17:09:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.310+0000 7f99d7fff640 1 --2- 172.21.15.116:0/1962744190 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f99c006b370 0x7f99c006d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.310+0000 7f6cc8c2d640 1 -- 172.21.15.116:0/1393021359 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cbc050160 msgr2=0x7f6cbc038040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.310+0000 7f6cc8c2d640 1 --2- 172.21.15.116:0/1393021359 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cbc050160 0x7f6cbc038040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.310+0000 7f6cc8c2d640 1 -- 172.21.15.116:0/1393021359 >> v1:172.21.15.116:6789/0 conn(0x7f6cbc0df1e0 legacy=0x7f6cbc04fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.310+0000 7f99d7fff640 1 -- 172.21.15.116:0/1962744190 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99d80ab9d0 con 0x7f99e00cc8a0 2023-11-28T17:09:10.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.310+0000 7f6cc8c2d640 1 -- 172.21.15.116:0/1393021359 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc0061d0 con 0x7f6cbc053c30 2023-11-28T17:09:10.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.311+0000 7f6ca4ff9640 1 -- 172.21.15.116:0/1190807032 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6c98004b40 con 0x7f6ca80b20a0 2023-11-28T17:09:10.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.311+0000 7feb0a7fc640 1 --2- 172.21.15.116:0/321043137 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb0c0722a0 0x7feb0c0b2260 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feb0000c390 tx=0x7feb0000c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.311+0000 7f99deffd640 1 --2- 172.21.15.116:0/1962744190 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f99c006b370 0x7f99c006d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.312+0000 7f6ca4ff9640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6c9406b2f0 0x7f6c9406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.312+0000 7f6cc8c2d640 1 --2- 172.21.15.116:0/1393021359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cbc053c30 0x7f6cbc0069d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6cb00097a0 tx=0x7f6cb002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89fde99151ed1000 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.312+0000 7f6ca4ff9640 1 -- 172.21.15.116:0/1190807032 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c98089950 con 0x7f6ca80b20a0 2023-11-28T17:09:10.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.313+0000 7f6ca77fe640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6c9406b2f0 0x7f6c9406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.313+0000 7f99deffd640 1 --2- 172.21.15.116:0/1962744190 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f99c006b370 0x7f99c006d830 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f99e00b3550 tx=0x7f99d0007b90 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.314+0000 7f6ca77fe640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6c9406b2f0 0x7f6c9406d7b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c035000 tx=0x7f6c9c030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.315+0000 7f6ca4ff9640 1 -- 172.21.15.116:0/1190807032 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6c98054850 con 0x7f6ca80b20a0 2023-11-28T17:09:10.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.315+0000 7feaf37fe640 1 -- 172.21.15.116:0/321043137 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb000052d0 con 0x7feb0c0722a0 2023-11-28T17:09:10.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.316+0000 7feaf37fe640 1 -- 172.21.15.116:0/321043137 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb00005430 con 0x7feb0c0722a0 2023-11-28T17:09:10.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.316+0000 7feb11636640 1 -- 172.21.15.116:0/321043137 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0c15a5b0 con 0x7feb0c0722a0 2023-11-28T17:09:10.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.316+0000 7feaf37fe640 1 -- 172.21.15.116:0/321043137 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb00003830 con 0x7feb0c0722a0 2023-11-28T17:09:10.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.316+0000 7feb11636640 1 -- 172.21.15.116:0/321043137 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0c0cbc20 con 0x7feb0c0722a0 2023-11-28T17:09:10.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.316+0000 7f2857fff640 1 -- 172.21.15.116:0/2994199901 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2858054b50 con 0x7f28640398a0 2023-11-28T17:09:10.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.318+0000 7feb11636640 1 -- 172.21.15.116:0/321043137 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0c0693d0 con 0x7feb0c0722a0 2023-11-28T17:09:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.321+0000 7feaf37fe640 1 -- 172.21.15.116:0/321043137 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7feb00003990 con 0x7feb0c0722a0 2023-11-28T17:09:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.322+0000 7feaf37fe640 1 --2- 172.21.15.116:0/321043137 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feaf806b2a0 0x7feaf806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.322+0000 7feaf37fe640 1 -- 172.21.15.116:0/321043137 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb00089030 con 0x7feb0c0722a0 2023-11-28T17:09:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.323+0000 7feb0affd640 1 --2- 172.21.15.116:0/321043137 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feaf806b2a0 0x7feaf806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.324+0000 7f6cc2ffd640 1 -- 172.21.15.116:0/1393021359 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb0031780 con 0x7f6cbc053c30 2023-11-28T17:09:10.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.324+0000 7feb0affd640 1 --2- 172.21.15.116:0/321043137 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feaf806b2a0 0x7feaf806d760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feb04001370 tx=0x7feb04009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.324+0000 7f6cc2ffd640 1 -- 172.21.15.116:0/1393021359 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb00318e0 con 0x7f6cbc053c30 2023-11-28T17:09:10.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.324+0000 7f99d7fff640 1 -- 172.21.15.116:0/1962744190 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99d80769f0 con 0x7f99e00cc8a0 2023-11-28T17:09:10.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.324+0000 7f6cc2ffd640 1 -- 172.21.15.116:0/1393021359 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb0031c70 con 0x7f6cbc053c30 2023-11-28T17:09:10.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.325+0000 7f6cc942e640 1 -- 172.21.15.116:0/1393021359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cbc053c30 msgr2=0x7f6cbc0069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.325+0000 7f6cc942e640 1 --2- 172.21.15.116:0/1393021359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cbc053c30 0x7f6cbc0069d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6cb00097a0 tx=0x7f6cb002b820 comp rx=0 tx=0).stop 2023-11-28T17:09:10.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.327+0000 7f6cc942e640 1 -- 172.21.15.116:0/1393021359 shutdown_connections 2023-11-28T17:09:10.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.327+0000 7f6cc942e640 1 --2- 172.21.15.116:0/1393021359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cbc053c30 0x7f6cbc0069d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.327+0000 7f6cc942e640 1 --2- 172.21.15.116:0/1393021359 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cbc050160 0x7f6cbc038040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.327+0000 7f6cc942e640 1 -- 172.21.15.116:0/1393021359 >> 172.21.15.116:0/1393021359 conn(0x7f6cbc009f70 msgr2=0x7f6cbc00a350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.327+0000 7f6cc942e640 1 -- 172.21.15.116:0/1393021359 shutdown_connections 2023-11-28T17:09:10.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.327+0000 7f6cc942e640 1 -- 172.21.15.116:0/1393021359 wait complete. 2023-11-28T17:09:10.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.330+0000 7f6cc942e640 1 Processor -- start 2023-11-28T17:09:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.330+0000 7f6cc942e640 1 -- start start 2023-11-28T17:09:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.331+0000 7f6cc942e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cbc050160 0x7f6cbc049fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.331+0000 7f6cc942e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cbc053c30 0x7f6cbc0d6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.332+0000 7f6cc942e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cbc0df1e0 0x7f6cbc0dae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.332+0000 7feaf37fe640 1 -- 172.21.15.116:0/321043137 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb00053fc0 con 0x7feb0c0722a0 2023-11-28T17:09:10.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.332+0000 7f6cc942e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cbc04af40 con 0x7f6cbc0df1e0 2023-11-28T17:09:10.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.332+0000 7f6cc942e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cbc0d6490 con 0x7f6cbc050160 2023-11-28T17:09:10.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.332+0000 7f6cc942e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cbc04a6a0 con 0x7f6cbc053c30 2023-11-28T17:09:10.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.332+0000 7f6cc8c2d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cbc0df1e0 0x7f6cbc0dae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.332+0000 7f6cc3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cbc050160 0x7f6cbc049fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.332+0000 7f6cc3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cbc050160 0x7f6cbc049fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:59190/0 (socket says 172.21.15.116:59190) 2023-11-28T17:09:10.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.332+0000 7f6cc3fff640 1 -- 172.21.15.116:0/554321615 learned_addr learned my addr 172.21.15.116:0/554321615 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:10.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.333+0000 7f6cc3fff640 1 -- 172.21.15.116:0/554321615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cbc053c30 msgr2=0x7f6cbc0d6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.333+0000 7f6cc3fff640 1 --2- 172.21.15.116:0/554321615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cbc053c30 0x7f6cbc0d6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.333+0000 7f6cc3fff640 1 -- 172.21.15.116:0/554321615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cbc0df1e0 msgr2=0x7f6cbc0dae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.333+0000 7f6cc3fff640 1 --2- 172.21.15.116:0/554321615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cbc0df1e0 0x7f6cbc0dae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.333+0000 7f6cc3fff640 1 -- 172.21.15.116:0/554321615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb0009400 con 0x7f6cbc050160 2023-11-28T17:09:10.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.334+0000 7f6cc3fff640 1 --2- 172.21.15.116:0/554321615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cbc050160 0x7f6cbc049fc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4068b70 tx=0x7f6cc4067f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.335+0000 7f6cc0ff9640 1 -- 172.21.15.116:0/554321615 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc406bc20 con 0x7f6cbc050160 2023-11-28T17:09:10.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.335+0000 7f6cc0ff9640 1 -- 172.21.15.116:0/554321615 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc406bd80 con 0x7f6cbc050160 2023-11-28T17:09:10.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.335+0000 7f6cc0ff9640 1 -- 172.21.15.116:0/554321615 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4072400 con 0x7f6cbc050160 2023-11-28T17:09:10.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.336+0000 7f6cc942e640 1 -- 172.21.15.116:0/554321615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cbc04a980 con 0x7f6cbc050160 2023-11-28T17:09:10.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.336+0000 7f6cc942e640 1 -- 172.21.15.116:0/554321615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cbc04c550 con 0x7f6cbc050160 2023-11-28T17:09:10.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.338+0000 7f6cc942e640 1 -- 172.21.15.116:0/554321615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cbc005350 con 0x7f6cbc050160 2023-11-28T17:09:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.340+0000 7f6cc0ff9640 1 -- 172.21.15.116:0/554321615 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6cc4072560 con 0x7f6cbc050160 2023-11-28T17:09:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.342+0000 7f6cc0ff9640 1 --2- 172.21.15.116:0/554321615 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6ca806b370 0x7f6ca806d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.342+0000 7f6cc0ff9640 1 -- 172.21.15.116:0/554321615 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6cc40b2e40 con 0x7f6cbc050160 2023-11-28T17:09:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.343+0000 7f6cc37fe640 1 --2- 172.21.15.116:0/554321615 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6ca806b370 0x7f6ca806d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.345+0000 7f6cc37fe640 1 --2- 172.21.15.116:0/554321615 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6ca806b370 0x7f6ca806d830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc04b660 tx=0x7f6cb8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:10.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.354+0000 7f6cc0ff9640 1 -- 172.21.15.116:0/554321615 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6cc40cea60 con 0x7f6cbc050160 2023-11-28T17:09:10.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.632+0000 7f5e9636f640 1 -- 172.21.15.116:0/3421873621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f5e900693d0 con 0x7f5e9015d2c0 2023-11-28T17:09:10.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.633+0000 7f5e8cff9640 1 -- 172.21.15.116:0/3421873621 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5e8807a060 con 0x7f5e9015d2c0 2023-11-28T17:09:10.634 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-28T17:09:10.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.636+0000 7f5e9636f640 1 -- 172.21.15.116:0/3421873621 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5e7006b2a0 msgr2=0x7f5e7006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.637+0000 7f5e9636f640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5e7006b2a0 0x7f5e7006d760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5e78000f60 tx=0x7f5e78009040 comp rx=0 tx=0).stop 2023-11-28T17:09:10.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.637+0000 7f5e9636f640 1 -- 172.21.15.116:0/3421873621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9015d2c0 msgr2=0x7f5e901f53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.637+0000 7f5e9636f640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9015d2c0 0x7f5e901f53a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5e88009890 tx=0x7f5e880046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.638+0000 7f5e9636f640 1 -- 172.21.15.116:0/3421873621 shutdown_connections 2023-11-28T17:09:10.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.638+0000 7f5e9636f640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5e7006b2a0 0x7f5e7006d760 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.638+0000 7f5e9636f640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9015d2c0 0x7f5e901f53a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.638+0000 7f5e9636f640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e900d2d70 0x7f5e9015f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.638+0000 7f5e9636f640 1 --2- 172.21.15.116:0/3421873621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e900ce140 0x7f5e901f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.638+0000 7f5e9636f640 1 -- 172.21.15.116:0/3421873621 >> 172.21.15.116:0/3421873621 conn(0x7f5e90072010 msgr2=0x7f5e900cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.639+0000 7f5e9636f640 1 -- 172.21.15.116:0/3421873621 shutdown_connections 2023-11-28T17:09:10.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.639+0000 7f5e9636f640 1 -- 172.21.15.116:0/3421873621 wait complete. 2023-11-28T17:09:10.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.654+0000 7fc0af80e640 1 -- 172.21.15.116:0/3934204890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc0a815a740 con 0x7fc0a80cb650 2023-11-28T17:09:10.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.656+0000 7fc09dffb640 1 -- 172.21.15.116:0/3934204890 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc0a815a740 con 0x7fc0a80cb650 2023-11-28T17:09:10.660 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-28T17:09:10.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.659+0000 7fc0af80e640 1 -- 172.21.15.116:0/3934204890 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc08c06b330 msgr2=0x7fc08c06d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.659+0000 7fc0af80e640 1 --2- 172.21.15.116:0/3934204890 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc08c06b330 0x7fc08c06d7f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc098009b30 tx=0x7fc098009240 comp rx=0 tx=0).stop 2023-11-28T17:09:10.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.659+0000 7fc0af80e640 1 -- 172.21.15.116:0/3934204890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a80cb650 msgr2=0x7fc0a80b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.660+0000 7fc0af80e640 1 --2- 172.21.15.116:0/3934204890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a80cb650 0x7fc0a80b04b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4001b30 tx=0x7fc0a400a430 comp rx=0 tx=0).stop 2023-11-28T17:09:10.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.660+0000 7fc0af80e640 1 -- 172.21.15.116:0/3934204890 shutdown_connections 2023-11-28T17:09:10.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.660+0000 7fc0af80e640 1 --2- 172.21.15.116:0/3934204890 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc08c06b330 0x7fc08c06d7f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.660+0000 7fc0af80e640 1 --2- 172.21.15.116:0/3934204890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0a8143800 0x7fc0a814e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.661+0000 7fc0af80e640 1 --2- 172.21.15.116:0/3934204890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0a80cb650 0x7fc0a80b04b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.661+0000 7fc0af80e640 1 --2- 172.21.15.116:0/3934204890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc0a80b3850 0x7fc0a8158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.661+0000 7fc0af80e640 1 -- 172.21.15.116:0/3934204890 >> 172.21.15.116:0/3934204890 conn(0x7fc0a806c250 msgr2=0x7fc0a80ccc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.664+0000 7fc0af80e640 1 -- 172.21.15.116:0/3934204890 shutdown_connections 2023-11-28T17:09:10.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.664+0000 7fc0af80e640 1 -- 172.21.15.116:0/3934204890 wait complete. 2023-11-28T17:09:10.667 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2023-11-28T17:09:10.667 DEBUG:teuthology.parallel:result is None 2023-11-28T17:09:10.691 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2023-11-28T17:09:10.691 DEBUG:teuthology.parallel:result is None 2023-11-28T17:09:10.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.694+0000 7f99e4839640 1 -- 172.21.15.116:0/1962744190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f99e006d690 con 0x7f99e00cc8a0 2023-11-28T17:09:10.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.695+0000 7f99d7fff640 1 -- 172.21.15.116:0/1962744190 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f99d807a010 con 0x7f99e00cc8a0 2023-11-28T17:09:10.696 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-28T17:09:10.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.697+0000 7f99e4839640 1 -- 172.21.15.116:0/1962744190 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f99c006b370 msgr2=0x7f99c006d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.697+0000 7f99e4839640 1 --2- 172.21.15.116:0/1962744190 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f99c006b370 0x7f99c006d830 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f99e00b3550 tx=0x7f99d0007b90 comp rx=0 tx=0).stop 2023-11-28T17:09:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.697+0000 7f99e4839640 1 -- 172.21.15.116:0/1962744190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e00cc8a0 msgr2=0x7f99e014c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.697+0000 7f99e4839640 1 --2- 172.21.15.116:0/1962744190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e00cc8a0 0x7f99e014c490 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f99e0072bf0 tx=0x7f99d80046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.697+0000 7f99e4839640 1 -- 172.21.15.116:0/1962744190 shutdown_connections 2023-11-28T17:09:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.697+0000 7f99e4839640 1 --2- 172.21.15.116:0/1962744190 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f99c006b370 0x7f99c006d830 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.697+0000 7f99e4839640 1 --2- 172.21.15.116:0/1962744190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99e0158e30 0x7f99e015b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.697+0000 7f99e4839640 1 --2- 172.21.15.116:0/1962744190 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f99e00ce110 0x7f99e0140760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.697+0000 7f99e4839640 1 --2- 172.21.15.116:0/1962744190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e00cc8a0 0x7f99e014c490 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.697+0000 7f99e4839640 1 -- 172.21.15.116:0/1962744190 >> 172.21.15.116:0/1962744190 conn(0x7f99e00aea80 msgr2=0x7f99e006daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.697+0000 7f99e4839640 1 -- 172.21.15.116:0/1962744190 shutdown_connections 2023-11-28T17:09:10.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.698+0000 7f99e4839640 1 -- 172.21.15.116:0/1962744190 wait complete. 2023-11-28T17:09:10.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.703+0000 7f2855ffb640 1 -- 172.21.15.116:0/2994199901 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2864005360 con 0x7f28640398a0 2023-11-28T17:09:10.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.704+0000 7f2857fff640 1 -- 172.21.15.116:0/2994199901 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2858058170 con 0x7f28640398a0 2023-11-28T17:09:10.706 INFO:teuthology.orchestra.run.smithi116.stdout:55834574855 2023-11-28T17:09:10.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.707+0000 7f2855ffb640 1 -- 172.21.15.116:0/2994199901 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f285006b370 msgr2=0x7f285006d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.707+0000 7f2855ffb640 1 --2- 172.21.15.116:0/2994199901 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f285006b370 0x7f285006d830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f286c0641c0 tx=0x7f286c06d040 comp rx=0 tx=0).stop 2023-11-28T17:09:10.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.707+0000 7f2855ffb640 1 -- 172.21.15.116:0/2994199901 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640398a0 msgr2=0x7f286404a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.707+0000 7f2855ffb640 1 --2- 172.21.15.116:0/2994199901 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640398a0 0x7f286404a2c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2858009fd0 tx=0x7f285800ed40 comp rx=0 tx=0).stop 2023-11-28T17:09:10.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.707+0000 7f2855ffb640 1 -- 172.21.15.116:0/2994199901 shutdown_connections 2023-11-28T17:09:10.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.707+0000 7f2855ffb640 1 --2- 172.21.15.116:0/2994199901 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f285006b370 0x7f285006d830 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f286c0641c0 tx=0x7f286c06d040 comp rx=0 tx=0).stop 2023-11-28T17:09:10.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.707+0000 7f2855ffb640 1 --2- 172.21.15.116:0/2994199901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28640dfe90 0x7f28640dd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.707+0000 7f2855ffb640 1 --2- 172.21.15.116:0/2994199901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2864052b10 0x7f28640dcb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.707+0000 7f2855ffb640 1 --2- 172.21.15.116:0/2994199901 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640398a0 0x7f286404a2c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.707+0000 7f2855ffb640 1 -- 172.21.15.116:0/2994199901 >> 172.21.15.116:0/2994199901 conn(0x7f2864009f80 msgr2=0x7f286404db70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.707+0000 7f2855ffb640 1 -- 172.21.15.116:0/2994199901 shutdown_connections 2023-11-28T17:09:10.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.708+0000 7f2855ffb640 1 -- 172.21.15.116:0/2994199901 wait complete. 2023-11-28T17:09:10.716 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.3 2023-11-28T17:09:10.716 DEBUG:teuthology.parallel:result is None 2023-11-28T17:09:10.725 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2023-11-28T17:09:10.725 DEBUG:teuthology.parallel:result is None 2023-11-28T17:09:10.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.725+0000 7f46aa14c640 1 -- 172.21.15.116:0/1957618393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f46a40cefa0 con 0x7f46a40ce760 2023-11-28T17:09:10.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.726+0000 7f46a0ff9640 1 -- 172.21.15.116:0/1957618393 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f46a40cefa0 con 0x7f46a40ce760 2023-11-28T17:09:10.727 INFO:teuthology.orchestra.run.smithi116.stdout:55834574854 2023-11-28T17:09:10.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.729+0000 7f46aa14c640 1 -- 172.21.15.116:0/1957618393 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f468406b2a0 msgr2=0x7f468406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.729+0000 7f46aa14c640 1 --2- 172.21.15.116:0/1957618393 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f468406b2a0 0x7f468406d760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f46a4159700 tx=0x7f4694030000 comp rx=0 tx=0).stop 2023-11-28T17:09:10.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.729+0000 7f46aa14c640 1 -- 172.21.15.116:0/1957618393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46a40ce760 msgr2=0x7f46a4151d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.729+0000 7f46aa14c640 1 --2- 172.21.15.116:0/1957618393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46a40ce760 0x7f46a4151d60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f468c00e870 tx=0x7f468c00ed30 comp rx=0 tx=0).stop 2023-11-28T17:09:10.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.729+0000 7f46aa14c640 1 -- 172.21.15.116:0/1957618393 shutdown_connections 2023-11-28T17:09:10.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.729+0000 7f46aa14c640 1 --2- 172.21.15.116:0/1957618393 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f468406b2a0 0x7f468406d760 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.729+0000 7f46aa14c640 1 --2- 172.21.15.116:0/1957618393 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f46a415d420 0x7f46a4156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.729+0000 7f46aa14c640 1 --2- 172.21.15.116:0/1957618393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46a40d0d30 0x7f46a40b8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.729+0000 7f46aa14c640 1 --2- 172.21.15.116:0/1957618393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46a40ce760 0x7f46a4151d60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.729+0000 7f46aa14c640 1 -- 172.21.15.116:0/1957618393 >> 172.21.15.116:0/1957618393 conn(0x7f46a40d2f50 msgr2=0x7f46a40c8810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.729+0000 7f46aa14c640 1 -- 172.21.15.116:0/1957618393 shutdown_connections 2023-11-28T17:09:10.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.729+0000 7f46aa14c640 1 -- 172.21.15.116:0/1957618393 wait complete. 2023-11-28T17:09:10.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.729+0000 7feb11636640 1 -- 172.21.15.116:0/321043137 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7feb0c06d020 con 0x7feb0c0722a0 2023-11-28T17:09:10.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.730+0000 7feaf37fe640 1 -- 172.21.15.116:0/321043137 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feb000575e0 con 0x7feb0c0722a0 2023-11-28T17:09:10.732 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-28T17:09:10.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.732+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1190807032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f6ca80d40b0 con 0x7f6ca80b20a0 2023-11-28T17:09:10.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.732+0000 7f6ca4ff9640 1 -- 172.21.15.116:0/1190807032 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6c98022020 con 0x7f6ca80b20a0 2023-11-28T17:09:10.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.732+0000 7feb11636640 1 -- 172.21.15.116:0/321043137 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feaf806b2a0 msgr2=0x7feaf806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.732+0000 7feb11636640 1 --2- 172.21.15.116:0/321043137 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feaf806b2a0 0x7feaf806d760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7feb04001370 tx=0x7feb04009040 comp rx=0 tx=0).stop 2023-11-28T17:09:10.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.732+0000 7feb11636640 1 -- 172.21.15.116:0/321043137 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb0c0722a0 msgr2=0x7feb0c0b2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.732+0000 7feb11636640 1 --2- 172.21.15.116:0/321043137 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb0c0722a0 0x7feb0c0b2260 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feb0000c390 tx=0x7feb0000c850 comp rx=0 tx=0).stop 2023-11-28T17:09:10.734 INFO:teuthology.orchestra.run.smithi116.stdout:55834574853 2023-11-28T17:09:10.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.733+0000 7feb11636640 1 -- 172.21.15.116:0/321043137 shutdown_connections 2023-11-28T17:09:10.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.733+0000 7feb11636640 1 --2- 172.21.15.116:0/321043137 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feaf806b2a0 0x7feaf806d760 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.733+0000 7feb11636640 1 --2- 172.21.15.116:0/321043137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb0c15ec40 0x7feb0c159dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.733+0000 7feb11636640 1 --2- 172.21.15.116:0/321043137 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb0c0722a0 0x7feb0c0b2260 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.733+0000 7feb11636640 1 --2- 172.21.15.116:0/321043137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb0c0703f0 0x7feb0c0cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.733+0000 7feb11636640 1 -- 172.21.15.116:0/321043137 >> 172.21.15.116:0/321043137 conn(0x7feb0c06c250 msgr2=0x7feb0c1545a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.733+0000 7feb11636640 1 -- 172.21.15.116:0/321043137 shutdown_connections 2023-11-28T17:09:10.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.733+0000 7feb11636640 1 -- 172.21.15.116:0/321043137 wait complete. 2023-11-28T17:09:10.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.736+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1190807032 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6c9406b2f0 msgr2=0x7f6c9406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.736+0000 7f6cae8d3640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6c9406b2f0 0x7f6c9406d7b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c035000 tx=0x7f6c9c030000 comp rx=0 tx=0).stop 2023-11-28T17:09:10.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.736+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1190807032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca80b20a0 msgr2=0x7f6ca8071000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.736+0000 7f6cae8d3640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca80b20a0 0x7f6ca8071000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800c460 tx=0x7f6c9800c920 comp rx=0 tx=0).stop 2023-11-28T17:09:10.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.736+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1190807032 shutdown_connections 2023-11-28T17:09:10.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.736+0000 7f6cae8d3640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6c9406b2f0 0x7f6c9406d7b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.736+0000 7f6cae8d3640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ca80d1780 0x7f6ca80b7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.736+0000 7f6cae8d3640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca80b2ad0 0x7f6ca80b77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.736+0000 7f6cae8d3640 1 --2- 172.21.15.116:0/1190807032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ca80b20a0 0x7f6ca8071000 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.736+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1190807032 >> 172.21.15.116:0/1190807032 conn(0x7f6ca809c4e0 msgr2=0x7f6ca806bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.736+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1190807032 shutdown_connections 2023-11-28T17:09:10.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.736+0000 7f6cae8d3640 1 -- 172.21.15.116:0/1190807032 wait complete. 2023-11-28T17:09:10.750 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2023-11-28T17:09:10.750 DEBUG:teuthology.parallel:result is None 2023-11-28T17:09:10.755 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.7 2023-11-28T17:09:10.755 DEBUG:teuthology.parallel:result is None 2023-11-28T17:09:10.759 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2023-11-28T17:09:10.759 DEBUG:teuthology.parallel:result is None 2023-11-28T17:09:10.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.761+0000 7f6cc942e640 1 -- 172.21.15.116:0/554321615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6cbc052ba0 con 0x7f6cbc050160 2023-11-28T17:09:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.762+0000 7f6cc0ff9640 1 -- 172.21.15.116:0/554321615 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6cc40d2080 con 0x7f6cbc050160 2023-11-28T17:09:10.764 INFO:teuthology.orchestra.run.smithi116.stdout:55834574855 2023-11-28T17:09:10.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.764+0000 7f6cc942e640 1 -- 172.21.15.116:0/554321615 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6ca806b370 msgr2=0x7f6ca806d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.764+0000 7f6cc942e640 1 --2- 172.21.15.116:0/554321615 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6ca806b370 0x7f6ca806d830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc04b660 tx=0x7f6cb8008040 comp rx=0 tx=0).stop 2023-11-28T17:09:10.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.764+0000 7f6cc942e640 1 -- 172.21.15.116:0/554321615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cbc050160 msgr2=0x7f6cbc049fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:10.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.764+0000 7f6cc942e640 1 --2- 172.21.15.116:0/554321615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cbc050160 0x7f6cbc049fc0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4068b70 tx=0x7f6cc4067f40 comp rx=0 tx=0).stop 2023-11-28T17:09:10.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.764+0000 7f6cc942e640 1 -- 172.21.15.116:0/554321615 shutdown_connections 2023-11-28T17:09:10.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.764+0000 7f6cc942e640 1 --2- 172.21.15.116:0/554321615 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6ca806b370 0x7f6ca806d830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.764+0000 7f6cc942e640 1 --2- 172.21.15.116:0/554321615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cbc0df1e0 0x7f6cbc0dae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.764+0000 7f6cc942e640 1 --2- 172.21.15.116:0/554321615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cbc053c30 0x7f6cbc0d6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.764+0000 7f6cc942e640 1 --2- 172.21.15.116:0/554321615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cbc050160 0x7f6cbc049fc0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:10.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.764+0000 7f6cc942e640 1 -- 172.21.15.116:0/554321615 >> 172.21.15.116:0/554321615 conn(0x7f6cbc009f70 msgr2=0x7f6cbc0e1920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:10.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.765+0000 7f6cc942e640 1 -- 172.21.15.116:0/554321615 shutdown_connections 2023-11-28T17:09:10.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:10.765+0000 7f6cc942e640 1 -- 172.21.15.116:0/554321615 wait complete. 2023-11-28T17:09:10.781 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2023-11-28T17:09:10.781 DEBUG:teuthology.parallel:result is None 2023-11-28T17:09:10.781 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-28T17:09:10.781 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:09:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.031+0000 7f82babd5640 1 Processor -- start 2023-11-28T17:09:11.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.031+0000 7f82babd5640 1 -- start start 2023-11-28T17:09:11.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.033+0000 7f82babd5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82b40b3940 0x7f82b406c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.033+0000 7f82babd5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b40b29c0 con 0x7f82b4162490 2023-11-28T17:09:11.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.033+0000 7f82babd5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b4162af0 con 0x7f82b40b3d20 2023-11-28T17:09:11.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.033+0000 7f82babd5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b40b2090 con 0x7f82b40b3940 2023-11-28T17:09:11.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.033+0000 7f82b9172640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f82b4162490 0x7f82b40703f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54686/0 (socket says 172.21.15.116:54686) 2023-11-28T17:09:11.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.033+0000 7f82b9172640 1 -- 172.21.15.116:0/3984748063 learned_addr learned my addr 172.21.15.116:0/3984748063 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:11.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.033+0000 7f82b37fe640 1 -- 172.21.15.116:0/3984748063 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989534954 0 0) 0x7f82b40b29c0 con 0x7f82b4162490 2023-11-28T17:09:11.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.033+0000 7f82b37fe640 1 -- 172.21.15.116:0/3984748063 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c003520 con 0x7f82b4162490 2023-11-28T17:09:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.033+0000 7f82b8971640 1 --2- 172.21.15.116:0/3984748063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82b40b3940 0x7f82b406c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b37fe640 1 -- 172.21.15.116:0/3984748063 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3454888858 0 0) 0x7f829c003520 con 0x7f82b4162490 2023-11-28T17:09:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b37fe640 1 -- 172.21.15.116:0/3984748063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82b40b3940 msgr2=0x7f82b406c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b37fe640 1 --2- 172.21.15.116:0/3984748063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82b40b3940 0x7f82b406c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b37fe640 1 -- 172.21.15.116:0/3984748063 >> v1:172.21.15.156:6789/0 conn(0x7f82b40b3d20 legacy=0x7f82b4161d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b37fe640 1 -- 172.21.15.116:0/3984748063 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b40b2370 con 0x7f82b4162490 2023-11-28T17:09:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b37fe640 1 -- 172.21.15.116:0/3984748063 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f82ac0019f0 con 0x7f82b4162490 2023-11-28T17:09:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b37fe640 1 -- 172.21.15.116:0/3984748063 >> v1:172.21.15.116:6789/0 conn(0x7f82b4162490 legacy=0x7f82b40703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b37fe640 1 --2- 172.21.15.116:0/3984748063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f829c0039a0 0x7f829c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b37fe640 1 --2- 172.21.15.116:0/3984748063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c0047c0 0x7f829c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b37fe640 1 --2- 172.21.15.116:0/3984748063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82b40b3940 0x7f829c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b37fe640 1 -- 172.21.15.116:0/3984748063 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b40b2090 con 0x7f829c0039a0 2023-11-28T17:09:11.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b37fe640 1 -- 172.21.15.116:0/3984748063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c00bb10 con 0x7f829c0047c0 2023-11-28T17:09:11.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b37fe640 1 -- 172.21.15.116:0/3984748063 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c00bcd0 con 0x7f82b40b3940 2023-11-28T17:09:11.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b8971640 1 --2- 172.21.15.116:0/3984748063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f829c0039a0 0x7f829c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.034+0000 7f82b3fff640 1 --2- 172.21.15.116:0/3984748063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c0047c0 0x7f829c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.035+0000 7f82b9172640 1 --2- 172.21.15.116:0/3984748063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82b40b3940 0x7f829c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.035+0000 7f82b8971640 1 -- 172.21.15.116:0/3984748063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82b40b3940 msgr2=0x7f829c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:11.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.035+0000 7f82b8971640 1 --2- 172.21.15.116:0/3984748063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82b40b3940 0x7f829c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.035+0000 7f82b8971640 1 -- 172.21.15.116:0/3984748063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c0047c0 msgr2=0x7f829c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:11.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.035+0000 7f82b8971640 1 --2- 172.21.15.116:0/3984748063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c0047c0 0x7f829c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.035+0000 7f82b8971640 1 -- 172.21.15.116:0/3984748063 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c003520 con 0x7f829c0039a0 2023-11-28T17:09:11.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.035+0000 7f82b9172640 1 --2- 172.21.15.116:0/3984748063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82b40b3940 0x7f829c005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:11.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.035+0000 7f82b3fff640 1 --2- 172.21.15.116:0/3984748063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c0047c0 0x7f829c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:09:11.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.036+0000 7f82b8971640 1 --2- 172.21.15.116:0/3984748063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f829c0039a0 0x7f829c003dc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f82a800ba30 tx=0x7f829c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c2defc8a6fc9041 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:11.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.036+0000 7f82b37fe640 1 -- 172.21.15.116:0/3984748063 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8032640 con 0x7f829c0039a0 2023-11-28T17:09:11.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.036+0000 7f82b37fe640 1 -- 172.21.15.116:0/3984748063 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a80327a0 con 0x7f829c0039a0 2023-11-28T17:09:11.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.036+0000 7f82b37fe640 1 -- 172.21.15.116:0/3984748063 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8032b30 con 0x7f829c0039a0 2023-11-28T17:09:11.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.036+0000 7f82babd5640 1 -- 172.21.15.116:0/3984748063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f829c0039a0 msgr2=0x7f829c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.036+0000 7f82babd5640 1 --2- 172.21.15.116:0/3984748063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f829c0039a0 0x7f829c003dc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f82a800ba30 tx=0x7f829c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.036+0000 7f82b9172640 1 -- 172.21.15.116:0/3984748063 reap_dead start 2023-11-28T17:09:11.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.037+0000 7f82babd5640 1 -- 172.21.15.116:0/3984748063 shutdown_connections 2023-11-28T17:09:11.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.037+0000 7f82babd5640 1 -- 172.21.15.116:0/3984748063 >> 172.21.15.116:0/3984748063 conn(0x7f82b406a4c0 msgr2=0x7f82b406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:11.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.037+0000 7f82babd5640 1 -- 172.21.15.116:0/3984748063 shutdown_connections 2023-11-28T17:09:11.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.037+0000 7f82babd5640 1 -- 172.21.15.116:0/3984748063 wait complete. 2023-11-28T17:09:11.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.038+0000 7f82babd5640 1 Processor -- start 2023-11-28T17:09:11.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.038+0000 7f82babd5640 1 -- start start 2023-11-28T17:09:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.039+0000 7f82babd5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c0039a0 0x7f82b41fb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.039+0000 7f82babd5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82b40abb70 0x7f82b4155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.039+0000 7f82babd5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82b4155fa0 0x7f82b40ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.039+0000 7f82babd5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b40acd20 con 0x7f82b4155fa0 2023-11-28T17:09:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.039+0000 7f82babd5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b40ad1b0 con 0x7f829c0039a0 2023-11-28T17:09:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.039+0000 7f82babd5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b41fba20 con 0x7f82b40abb70 2023-11-28T17:09:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.039+0000 7f82b9172640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82b4155fa0 0x7f82b40ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.040+0000 7f82b9172640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82b4155fa0 0x7f82b40ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52130/0 (socket says 172.21.15.116:52130) 2023-11-28T17:09:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.040+0000 7f82b9172640 1 -- 172.21.15.116:0/961133846 learned_addr learned my addr 172.21.15.116:0/961133846 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.040+0000 7f82b8971640 1 --2- 172.21.15.116:0/961133846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c0039a0 0x7f82b41fb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.040+0000 7f82b3fff640 1 --2- 172.21.15.116:0/961133846 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82b40abb70 0x7f82b4155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.040+0000 7f82b9172640 1 -- 172.21.15.116:0/961133846 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82b40abb70 msgr2=0x7f82b4155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.040+0000 7f82b9172640 1 --2- 172.21.15.116:0/961133846 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82b40abb70 0x7f82b4155a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.040+0000 7f82b9172640 1 -- 172.21.15.116:0/961133846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c0039a0 msgr2=0x7f82b41fb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.040+0000 7f82b9172640 1 --2- 172.21.15.116:0/961133846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c0039a0 0x7f82b41fb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.040+0000 7f82b9172640 1 -- 172.21.15.116:0/961133846 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b40ac110 con 0x7f82b4155fa0 2023-11-28T17:09:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.040+0000 7f82b9172640 1 --2- 172.21.15.116:0/961133846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82b4155fa0 0x7f82b40ccd20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f82ac00dc10 tx=0x7f82ac00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.040+0000 7f82b17fa640 1 -- 172.21.15.116:0/961133846 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac00dee0 con 0x7f82b4155fa0 2023-11-28T17:09:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.040+0000 7f82b17fa640 1 -- 172.21.15.116:0/961133846 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ac00bb30 con 0x7f82b4155fa0 2023-11-28T17:09:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.040+0000 7f82b17fa640 1 -- 172.21.15.116:0/961133846 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac00bea0 con 0x7f82b4155fa0 2023-11-28T17:09:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.040+0000 7f82babd5640 1 -- 172.21.15.116:0/961133846 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b40ac3f0 con 0x7f82b4155fa0 2023-11-28T17:09:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.041+0000 7f82babd5640 1 -- 172.21.15.116:0/961133846 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b40ac9b0 con 0x7f82b4155fa0 2023-11-28T17:09:11.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.042+0000 7f82babd5640 1 -- 172.21.15.116:0/961133846 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b40acb70 con 0x7f82b4155fa0 2023-11-28T17:09:11.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.042+0000 7f82b17fa640 1 -- 172.21.15.116:0/961133846 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f82ac007110 con 0x7f82b4155fa0 2023-11-28T17:09:11.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.043+0000 7f82b17fa640 1 --2- 172.21.15.116:0/961133846 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f82a006b2d0 0x7f82a006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.043+0000 7f82b17fa640 1 -- 172.21.15.116:0/961133846 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82ac089f90 con 0x7f82b4155fa0 2023-11-28T17:09:11.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.043+0000 7f82b8971640 1 --2- 172.21.15.116:0/961133846 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f82a006b2d0 0x7f82a006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.044+0000 7f82b8971640 1 --2- 172.21.15.116:0/961133846 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f82a006b2d0 0x7f82a006d790 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f82a8037000 tx=0x7f82a803d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:11.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.048+0000 7f82b17fa640 1 -- 172.21.15.116:0/961133846 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82ac054df0 con 0x7f82b4155fa0 2023-11-28T17:09:11.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.222+0000 7f82babd5640 1 -- 172.21.15.116:0/961133846 --> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f82b4070b00 con 0x7f82a006b2d0 2023-11-28T17:09:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.237+0000 7f82b17fa640 1 -- 172.21.15.116:0/961133846 <== mgr.4098 v2:172.21.15.116:6832/1077400178 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264398 (secure 0 0 0) 0x7f82a8032910 con 0x7f82a006b2d0 2023-11-28T17:09:11.242 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:11.243 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-28T17:09:11.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.241+0000 7f82babd5640 1 -- 172.21.15.116:0/961133846 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f82a006b2d0 msgr2=0x7f82a006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.241+0000 7f82babd5640 1 --2- 172.21.15.116:0/961133846 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f82a006b2d0 0x7f82a006d790 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f82a8037000 tx=0x7f82a803d040 comp rx=0 tx=0).stop 2023-11-28T17:09:11.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.241+0000 7f82babd5640 1 -- 172.21.15.116:0/961133846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82b4155fa0 msgr2=0x7f82b40ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.241+0000 7f82babd5640 1 --2- 172.21.15.116:0/961133846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82b4155fa0 0x7f82b40ccd20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f82ac00dc10 tx=0x7f82ac00b4a0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.242+0000 7f82babd5640 1 -- 172.21.15.116:0/961133846 shutdown_connections 2023-11-28T17:09:11.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.242+0000 7f82babd5640 1 --2- 172.21.15.116:0/961133846 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f82a006b2d0 0x7f82a006d790 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.242+0000 7f82babd5640 1 --2- 172.21.15.116:0/961133846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82b4155fa0 0x7f82b40ccd20 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.242+0000 7f82babd5640 1 --2- 172.21.15.116:0/961133846 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82b40abb70 0x7f82b4155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.242+0000 7f82babd5640 1 --2- 172.21.15.116:0/961133846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f829c0039a0 0x7f82b41fb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.242+0000 7f82babd5640 1 -- 172.21.15.116:0/961133846 >> 172.21.15.116:0/961133846 conn(0x7f82b406a4c0 msgr2=0x7f82b4161030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.242+0000 7f82babd5640 1 -- 172.21.15.116:0/961133846 shutdown_connections 2023-11-28T17:09:11.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.242+0000 7f82babd5640 1 -- 172.21.15.116:0/961133846 wait complete. 2023-11-28T17:09:11.260 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-11-28T17:09:11.042788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235648,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739072,"statfs":{"total":773094113280,"available":772852809728,"internally_reserved":0,"allocated":2891776,"data_stored":1752836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003611"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513632+0000","last_change":"2023-11-28T17:09:01.014584+0000","last_active":"2023-11-28T17:09:02.513632+0000","last_peered":"2023-11-28T17:09:02.513632+0000","last_clean":"2023-11-28T17:09:02.513632+0000","last_became_active":"2023-11-28T17:09:01.014291+0000","last_became_peered":"2023-11-28T17:09:01.014291+0000","last_unstale":"2023-11-28T17:09:02.513632+0000","last_undegraded":"2023-11-28T17:09:02.513632+0000","last_fullsized":"2023-11-28T17:09:02.513632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:22.787341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223151+0000","last_change":"2023-11-28T17:09:01.009980+0000","last_active":"2023-11-28T17:09:02.223151+0000","last_peered":"2023-11-28T17:09:02.223151+0000","last_clean":"2023-11-28T17:09:02.223151+0000","last_became_active":"2023-11-28T17:09:01.009702+0000","last_became_peered":"2023-11-28T17:09:01.009702+0000","last_unstale":"2023-11-28T17:09:02.223151+0000","last_undegraded":"2023-11-28T17:09:02.223151+0000","last_fullsized":"2023-11-28T17:09:02.223151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:06:59.199376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222772+0000","last_change":"2023-11-28T17:09:01.010483+0000","last_active":"2023-11-28T17:09:02.222772+0000","last_peered":"2023-11-28T17:09:02.222772+0000","last_clean":"2023-11-28T17:09:02.222772+0000","last_became_active":"2023-11-28T17:09:01.010032+0000","last_became_peered":"2023-11-28T17:09:01.010032+0000","last_unstale":"2023-11-28T17:09:02.222772+0000","last_undegraded":"2023-11-28T17:09:02.222772+0000","last_fullsized":"2023-11-28T17:09:02.222772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:46:24.200692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513159+0000","last_change":"2023-11-28T17:09:00.998473+0000","last_active":"2023-11-28T17:09:02.513159+0000","last_peered":"2023-11-28T17:09:02.513159+0000","last_clean":"2023-11-28T17:09:02.513159+0000","last_became_active":"2023-11-28T17:09:00.998206+0000","last_became_peered":"2023-11-28T17:09:00.998206+0000","last_unstale":"2023-11-28T17:09:02.513159+0000","last_undegraded":"2023-11-28T17:09:02.513159+0000","last_fullsized":"2023-11-28T17:09:02.513159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:31.569896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512712+0000","last_change":"2023-11-28T17:09:01.019053+0000","last_active":"2023-11-28T17:09:02.512712+0000","last_peered":"2023-11-28T17:09:02.512712+0000","last_clean":"2023-11-28T17:09:02.512712+0000","last_became_active":"2023-11-28T17:09:01.018689+0000","last_became_peered":"2023-11-28T17:09:01.018689+0000","last_unstale":"2023-11-28T17:09:02.512712+0000","last_undegraded":"2023-11-28T17:09:02.512712+0000","last_fullsized":"2023-11-28T17:09:02.512712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:06:59.199376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513060+0000","last_change":"2023-11-28T17:09:01.000056+0000","last_active":"2023-11-28T17:09:02.513060+0000","last_peered":"2023-11-28T17:09:02.513060+0000","last_clean":"2023-11-28T17:09:02.513060+0000","last_became_active":"2023-11-28T17:09:00.999547+0000","last_became_peered":"2023-11-28T17:09:00.999547+0000","last_unstale":"2023-11-28T17:09:02.513060+0000","last_undegraded":"2023-11-28T17:09:02.513060+0000","last_fullsized":"2023-11-28T17:09:02.513060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:12.314413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.511764+0000","last_change":"2023-11-28T17:09:01.012699+0000","last_active":"2023-11-28T17:09:02.511764+0000","last_peered":"2023-11-28T17:09:02.511764+0000","last_clean":"2023-11-28T17:09:02.511764+0000","last_became_active":"2023-11-28T17:09:01.012442+0000","last_became_peered":"2023-11-28T17:09:01.012442+0000","last_unstale":"2023-11-28T17:09:02.511764+0000","last_undegraded":"2023-11-28T17:09:02.511764+0000","last_fullsized":"2023-11-28T17:09:02.511764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:46:24.200692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223215+0000","last_change":"2023-11-28T17:09:01.016792+0000","last_active":"2023-11-28T17:09:02.223215+0000","last_peered":"2023-11-28T17:09:02.223215+0000","last_clean":"2023-11-28T17:09:02.223215+0000","last_became_active":"2023-11-28T17:09:01.016034+0000","last_became_peered":"2023-11-28T17:09:01.016034+0000","last_unstale":"2023-11-28T17:09:02.223215+0000","last_undegraded":"2023-11-28T17:09:02.223215+0000","last_fullsized":"2023-11-28T17:09:02.223215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:52.224230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515192+0000","last_change":"2023-11-28T17:09:01.007271+0000","last_active":"2023-11-28T17:09:02.515192+0000","last_peered":"2023-11-28T17:09:02.515192+0000","last_clean":"2023-11-28T17:09:02.515192+0000","last_became_active":"2023-11-28T17:09:01.006929+0000","last_became_peered":"2023-11-28T17:09:01.006929+0000","last_unstale":"2023-11-28T17:09:02.515192+0000","last_undegraded":"2023-11-28T17:09:02.515192+0000","last_fullsized":"2023-11-28T17:09:02.515192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:20.109182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515425+0000","last_change":"2023-11-28T17:09:01.012176+0000","last_active":"2023-11-28T17:09:02.515425+0000","last_peered":"2023-11-28T17:09:02.515425+0000","last_clean":"2023-11-28T17:09:02.515425+0000","last_became_active":"2023-11-28T17:09:01.011821+0000","last_became_peered":"2023-11-28T17:09:01.011821+0000","last_unstale":"2023-11-28T17:09:02.515425+0000","last_undegraded":"2023-11-28T17:09:02.515425+0000","last_fullsized":"2023-11-28T17:09:02.515425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:55.626548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513365+0000","last_change":"2023-11-28T17:09:01.008348+0000","last_active":"2023-11-28T17:09:02.513365+0000","last_peered":"2023-11-28T17:09:02.513365+0000","last_clean":"2023-11-28T17:09:02.513365+0000","last_became_active":"2023-11-28T17:09:01.008029+0000","last_became_peered":"2023-11-28T17:09:01.008029+0000","last_unstale":"2023-11-28T17:09:02.513365+0000","last_undegraded":"2023-11-28T17:09:02.513365+0000","last_fullsized":"2023-11-28T17:09:02.513365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:52.224230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512353+0000","last_change":"2023-11-28T17:09:01.013836+0000","last_active":"2023-11-28T17:09:02.512353+0000","last_peered":"2023-11-28T17:09:02.512353+0000","last_clean":"2023-11-28T17:09:02.512353+0000","last_became_active":"2023-11-28T17:09:01.011870+0000","last_became_peered":"2023-11-28T17:09:01.011870+0000","last_unstale":"2023-11-28T17:09:02.512353+0000","last_undegraded":"2023-11-28T17:09:02.512353+0000","last_fullsized":"2023-11-28T17:09:02.512353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:23:27.238395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988425+0000","last_change":"2023-11-28T17:09:01.007854+0000","last_active":"2023-11-28T17:09:01.988425+0000","last_peered":"2023-11-28T17:09:01.988425+0000","last_clean":"2023-11-28T17:09:01.988425+0000","last_became_active":"2023-11-28T17:09:01.007561+0000","last_became_peered":"2023-11-28T17:09:01.007561+0000","last_unstale":"2023-11-28T17:09:01.988425+0000","last_undegraded":"2023-11-28T17:09:01.988425+0000","last_fullsized":"2023-11-28T17:09:01.988425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:53.375742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986992+0000","last_change":"2023-11-28T17:09:01.006142+0000","last_active":"2023-11-28T17:09:01.986992+0000","last_peered":"2023-11-28T17:09:01.986992+0000","last_clean":"2023-11-28T17:09:01.986992+0000","last_became_active":"2023-11-28T17:09:01.005823+0000","last_became_peered":"2023-11-28T17:09:01.005823+0000","last_unstale":"2023-11-28T17:09:01.986992+0000","last_undegraded":"2023-11-28T17:09:01.986992+0000","last_fullsized":"2023-11-28T17:09:01.986992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:32.388198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222442+0000","last_change":"2023-11-28T17:09:01.016320+0000","last_active":"2023-11-28T17:09:02.222442+0000","last_peered":"2023-11-28T17:09:02.222442+0000","last_clean":"2023-11-28T17:09:02.222442+0000","last_became_active":"2023-11-28T17:09:01.016012+0000","last_became_peered":"2023-11-28T17:09:01.016012+0000","last_unstale":"2023-11-28T17:09:02.222442+0000","last_undegraded":"2023-11-28T17:09:02.222442+0000","last_fullsized":"2023-11-28T17:09:02.222442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:44.651188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986526+0000","last_change":"2023-11-28T17:09:00.996901+0000","last_active":"2023-11-28T17:09:01.986526+0000","last_peered":"2023-11-28T17:09:01.986526+0000","last_clean":"2023-11-28T17:09:01.986526+0000","last_became_active":"2023-11-28T17:09:00.996627+0000","last_became_peered":"2023-11-28T17:09:00.996627+0000","last_unstale":"2023-11-28T17:09:01.986526+0000","last_undegraded":"2023-11-28T17:09:01.986526+0000","last_fullsized":"2023-11-28T17:09:01.986526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:45.964344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988654+0000","last_change":"2023-11-28T17:09:01.015527+0000","last_active":"2023-11-28T17:09:01.988654+0000","last_peered":"2023-11-28T17:09:01.988654+0000","last_clean":"2023-11-28T17:09:01.988654+0000","last_became_active":"2023-11-28T17:09:01.015238+0000","last_became_peered":"2023-11-28T17:09:01.015238+0000","last_unstale":"2023-11-28T17:09:01.988654+0000","last_undegraded":"2023-11-28T17:09:01.988654+0000","last_fullsized":"2023-11-28T17:09:01.988654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:44:13.494591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512305+0000","last_change":"2023-11-28T17:09:01.015000+0000","last_active":"2023-11-28T17:09:02.512305+0000","last_peered":"2023-11-28T17:09:02.512305+0000","last_clean":"2023-11-28T17:09:02.512305+0000","last_became_active":"2023-11-28T17:09:01.014649+0000","last_became_peered":"2023-11-28T17:09:01.014649+0000","last_unstale":"2023-11-28T17:09:02.512305+0000","last_undegraded":"2023-11-28T17:09:02.512305+0000","last_fullsized":"2023-11-28T17:09:02.512305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:31.569896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988495+0000","last_change":"2023-11-28T17:09:00.997869+0000","last_active":"2023-11-28T17:09:01.988495+0000","last_peered":"2023-11-28T17:09:01.988495+0000","last_clean":"2023-11-28T17:09:01.988495+0000","last_became_active":"2023-11-28T17:09:00.997614+0000","last_became_peered":"2023-11-28T17:09:00.997614+0000","last_unstale":"2023-11-28T17:09:01.988495+0000","last_undegraded":"2023-11-28T17:09:01.988495+0000","last_fullsized":"2023-11-28T17:09:01.988495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:25:26.955817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513938+0000","last_change":"2023-11-28T17:09:01.011584+0000","last_active":"2023-11-28T17:09:02.513938+0000","last_peered":"2023-11-28T17:09:02.513938+0000","last_clean":"2023-11-28T17:09:02.513938+0000","last_became_active":"2023-11-28T17:09:01.010359+0000","last_became_peered":"2023-11-28T17:09:01.010359+0000","last_unstale":"2023-11-28T17:09:02.513938+0000","last_undegraded":"2023-11-28T17:09:02.513938+0000","last_fullsized":"2023-11-28T17:09:02.513938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:45.964344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.989818+0000","last_change":"2023-11-28T17:09:00.999976+0000","last_active":"2023-11-28T17:09:01.989818+0000","last_peered":"2023-11-28T17:09:01.989818+0000","last_clean":"2023-11-28T17:09:01.989818+0000","last_became_active":"2023-11-28T17:09:00.999647+0000","last_became_peered":"2023-11-28T17:09:00.999647+0000","last_unstale":"2023-11-28T17:09:01.989818+0000","last_undegraded":"2023-11-28T17:09:01.989818+0000","last_fullsized":"2023-11-28T17:09:01.989818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:44.651188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.514610+0000","last_change":"2023-11-28T17:09:01.009510+0000","last_active":"2023-11-28T17:09:02.514610+0000","last_peered":"2023-11-28T17:09:02.514610+0000","last_clean":"2023-11-28T17:09:02.514610+0000","last_became_active":"2023-11-28T17:09:01.009184+0000","last_became_peered":"2023-11-28T17:09:01.009184+0000","last_unstale":"2023-11-28T17:09:02.514610+0000","last_undegraded":"2023-11-28T17:09:02.514610+0000","last_fullsized":"2023-11-28T17:09:02.514610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:12.314413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222577+0000","last_change":"2023-11-28T17:09:01.015602+0000","last_active":"2023-11-28T17:09:02.222577+0000","last_peered":"2023-11-28T17:09:02.222577+0000","last_clean":"2023-11-28T17:09:02.222577+0000","last_became_active":"2023-11-28T17:09:01.012170+0000","last_became_peered":"2023-11-28T17:09:01.012170+0000","last_unstale":"2023-11-28T17:09:02.222577+0000","last_undegraded":"2023-11-28T17:09:02.222577+0000","last_fullsized":"2023-11-28T17:09:02.222577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:07.593383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986512+0000","last_change":"2023-11-28T17:09:01.016062+0000","last_active":"2023-11-28T17:09:01.986512+0000","last_peered":"2023-11-28T17:09:01.986512+0000","last_clean":"2023-11-28T17:09:01.986512+0000","last_became_active":"2023-11-28T17:09:01.015802+0000","last_became_peered":"2023-11-28T17:09:01.015802+0000","last_unstale":"2023-11-28T17:09:01.986512+0000","last_undegraded":"2023-11-28T17:09:01.986512+0000","last_fullsized":"2023-11-28T17:09:01.986512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:34.895213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223456+0000","last_change":"2023-11-28T17:09:01.016389+0000","last_active":"2023-11-28T17:09:02.223456+0000","last_peered":"2023-11-28T17:09:02.223456+0000","last_clean":"2023-11-28T17:09:02.223456+0000","last_became_active":"2023-11-28T17:09:01.011680+0000","last_became_peered":"2023-11-28T17:09:01.011680+0000","last_unstale":"2023-11-28T17:09:02.223456+0000","last_undegraded":"2023-11-28T17:09:02.223456+0000","last_fullsized":"2023-11-28T17:09:02.223456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:22.787341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515917+0000","last_change":"2023-11-28T17:09:01.006281+0000","last_active":"2023-11-28T17:09:02.515917+0000","last_peered":"2023-11-28T17:09:02.515917+0000","last_clean":"2023-11-28T17:09:02.515917+0000","last_became_active":"2023-11-28T17:09:01.000480+0000","last_became_peered":"2023-11-28T17:09:01.000480+0000","last_unstale":"2023-11-28T17:09:02.515917+0000","last_undegraded":"2023-11-28T17:09:02.515917+0000","last_fullsized":"2023-11-28T17:09:02.515917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:25.324961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.989120+0000","last_change":"2023-11-28T17:09:01.002090+0000","last_active":"2023-11-28T17:09:01.989120+0000","last_peered":"2023-11-28T17:09:01.989120+0000","last_clean":"2023-11-28T17:09:01.989120+0000","last_became_active":"2023-11-28T17:09:01.001728+0000","last_became_peered":"2023-11-28T17:09:01.001728+0000","last_unstale":"2023-11-28T17:09:01.989120+0000","last_undegraded":"2023-11-28T17:09:01.989120+0000","last_fullsized":"2023-11-28T17:09:01.989120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:11.188332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988810+0000","last_change":"2023-11-28T17:09:01.013978+0000","last_active":"2023-11-28T17:09:01.988810+0000","last_peered":"2023-11-28T17:09:01.988810+0000","last_clean":"2023-11-28T17:09:01.988810+0000","last_became_active":"2023-11-28T17:09:01.013646+0000","last_became_peered":"2023-11-28T17:09:01.013646+0000","last_unstale":"2023-11-28T17:09:01.988810+0000","last_undegraded":"2023-11-28T17:09:01.988810+0000","last_fullsized":"2023-11-28T17:09:01.988810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:00.035340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222920+0000","last_change":"2023-11-28T17:09:01.013635+0000","last_active":"2023-11-28T17:09:02.222920+0000","last_peered":"2023-11-28T17:09:02.222920+0000","last_clean":"2023-11-28T17:09:02.222920+0000","last_became_active":"2023-11-28T17:09:01.013310+0000","last_became_peered":"2023-11-28T17:09:01.013310+0000","last_unstale":"2023-11-28T17:09:02.222920+0000","last_undegraded":"2023-11-28T17:09:02.222920+0000","last_fullsized":"2023-11-28T17:09:02.222920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:23:27.238395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987149+0000","last_change":"2023-11-28T17:09:01.016976+0000","last_active":"2023-11-28T17:09:01.987149+0000","last_peered":"2023-11-28T17:09:01.987149+0000","last_clean":"2023-11-28T17:09:01.987149+0000","last_became_active":"2023-11-28T17:09:01.016690+0000","last_became_peered":"2023-11-28T17:09:01.016690+0000","last_unstale":"2023-11-28T17:09:01.987149+0000","last_undegraded":"2023-11-28T17:09:01.987149+0000","last_fullsized":"2023-11-28T17:09:01.987149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:55.626548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.511885+0000","last_change":"2023-11-28T17:09:01.012474+0000","last_active":"2023-11-28T17:09:02.511885+0000","last_peered":"2023-11-28T17:09:02.511885+0000","last_clean":"2023-11-28T17:09:02.511885+0000","last_became_active":"2023-11-28T17:09:01.012226+0000","last_became_peered":"2023-11-28T17:09:01.012226+0000","last_unstale":"2023-11-28T17:09:02.511885+0000","last_undegraded":"2023-11-28T17:09:02.511885+0000","last_fullsized":"2023-11-28T17:09:02.511885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:36.438412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512524+0000","last_change":"2023-11-28T17:09:00.994555+0000","last_active":"2023-11-28T17:09:02.512524+0000","last_peered":"2023-11-28T17:09:02.512524+0000","last_clean":"2023-11-28T17:09:02.512524+0000","last_became_active":"2023-11-28T17:09:00.993870+0000","last_became_peered":"2023-11-28T17:09:00.993870+0000","last_unstale":"2023-11-28T17:09:02.512524+0000","last_undegraded":"2023-11-28T17:09:02.512524+0000","last_fullsized":"2023-11-28T17:09:02.512524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:36.438412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.514893+0000","last_change":"2023-11-28T17:09:01.018035+0000","last_active":"2023-11-28T17:09:02.514893+0000","last_peered":"2023-11-28T17:09:02.514893+0000","last_clean":"2023-11-28T17:09:02.514893+0000","last_became_active":"2023-11-28T17:09:01.017735+0000","last_became_peered":"2023-11-28T17:09:01.017735+0000","last_unstale":"2023-11-28T17:09:02.514893+0000","last_undegraded":"2023-11-28T17:09:02.514893+0000","last_fullsized":"2023-11-28T17:09:02.514893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:32.388198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.271905+0000","last_change":"2023-11-28T17:09:00.266501+0000","last_active":"2023-11-28T17:09:01.271905+0000","last_peered":"2023-11-28T17:09:01.271905+0000","last_clean":"2023-11-28T17:09:01.271905+0000","last_became_active":"2023-11-28T17:09:00.266177+0000","last_became_peered":"2023-11-28T17:09:00.266177+0000","last_unstale":"2023-11-28T17:09:01.271905+0000","last_undegraded":"2023-11-28T17:09:01.271905+0000","last_fullsized":"2023-11-28T17:09:01.271905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:08.376086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.990190+0000","last_change":"2023-11-28T17:09:01.008791+0000","last_active":"2023-11-28T17:09:01.990190+0000","last_peered":"2023-11-28T17:09:01.990190+0000","last_clean":"2023-11-28T17:09:01.990190+0000","last_became_active":"2023-11-28T17:09:01.008266+0000","last_became_peered":"2023-11-28T17:09:01.008266+0000","last_unstale":"2023-11-28T17:09:01.990190+0000","last_undegraded":"2023-11-28T17:09:01.990190+0000","last_fullsized":"2023-11-28T17:09:01.990190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:50.317730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.275936+0000","last_change":"2023-11-28T17:09:00.269760+0000","last_active":"2023-11-28T17:09:01.275936+0000","last_peered":"2023-11-28T17:09:01.275936+0000","last_clean":"2023-11-28T17:09:01.275936+0000","last_became_active":"2023-11-28T17:09:00.269082+0000","last_became_peered":"2023-11-28T17:09:00.269082+0000","last_unstale":"2023-11-28T17:09:01.275936+0000","last_undegraded":"2023-11-28T17:09:01.275936+0000","last_fullsized":"2023-11-28T17:09:01.275936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:51.213109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.986115+0000","last_change":"2023-11-28T17:09:00.262912+0000","last_active":"2023-11-28T17:09:00.986115+0000","last_peered":"2023-11-28T17:09:00.986115+0000","last_clean":"2023-11-28T17:09:00.986115+0000","last_became_active":"2023-11-28T17:09:00.260568+0000","last_became_peered":"2023-11-28T17:09:00.260568+0000","last_unstale":"2023-11-28T17:09:00.986115+0000","last_undegraded":"2023-11-28T17:09:00.986115+0000","last_fullsized":"2023-11-28T17:09:00.986115+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:30.558775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223014+0000","last_change":"2023-11-28T17:09:01.012823+0000","last_active":"2023-11-28T17:09:02.223014+0000","last_peered":"2023-11-28T17:09:02.223014+0000","last_clean":"2023-11-28T17:09:02.223014+0000","last_became_active":"2023-11-28T17:09:01.012514+0000","last_became_peered":"2023-11-28T17:09:01.012514+0000","last_unstale":"2023-11-28T17:09:02.223014+0000","last_undegraded":"2023-11-28T17:09:02.223014+0000","last_fullsized":"2023-11-28T17:09:02.223014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:24.354633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.985898+0000","last_change":"2023-11-28T17:09:00.262108+0000","last_active":"2023-11-28T17:09:00.985898+0000","last_peered":"2023-11-28T17:09:00.985898+0000","last_clean":"2023-11-28T17:09:00.985898+0000","last_became_active":"2023-11-28T17:09:00.261621+0000","last_became_peered":"2023-11-28T17:09:00.261621+0000","last_unstale":"2023-11-28T17:09:00.985898+0000","last_undegraded":"2023-11-28T17:09:00.985898+0000","last_fullsized":"2023-11-28T17:09:00.985898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:14:52.843064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988589+0000","last_change":"2023-11-28T17:09:01.011726+0000","last_active":"2023-11-28T17:09:01.988589+0000","last_peered":"2023-11-28T17:09:01.988589+0000","last_clean":"2023-11-28T17:09:01.988589+0000","last_became_active":"2023-11-28T17:09:01.010904+0000","last_became_peered":"2023-11-28T17:09:01.010904+0000","last_unstale":"2023-11-28T17:09:01.988589+0000","last_undegraded":"2023-11-28T17:09:01.988589+0000","last_fullsized":"2023-11-28T17:09:01.988589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:14:53.854185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.990133+0000","last_change":"2023-11-28T17:09:00.262360+0000","last_active":"2023-11-28T17:09:00.990133+0000","last_peered":"2023-11-28T17:09:00.990133+0000","last_clean":"2023-11-28T17:09:00.990133+0000","last_became_active":"2023-11-28T17:09:00.261943+0000","last_became_peered":"2023-11-28T17:09:00.261943+0000","last_unstale":"2023-11-28T17:09:00.990133+0000","last_undegraded":"2023-11-28T17:09:00.990133+0000","last_fullsized":"2023-11-28T17:09:00.990133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:52.998404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515909+0000","last_change":"2023-11-28T17:09:01.007941+0000","last_active":"2023-11-28T17:09:02.515909+0000","last_peered":"2023-11-28T17:09:02.515909+0000","last_clean":"2023-11-28T17:09:02.515909+0000","last_became_active":"2023-11-28T17:09:01.002743+0000","last_became_peered":"2023-11-28T17:09:01.002743+0000","last_unstale":"2023-11-28T17:09:02.515909+0000","last_undegraded":"2023-11-28T17:09:02.515909+0000","last_fullsized":"2023-11-28T17:09:02.515909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:20.109182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.997425+0000","last_change":"2023-11-28T17:09:00.259873+0000","last_active":"2023-11-28T17:09:00.997425+0000","last_peered":"2023-11-28T17:09:00.997425+0000","last_clean":"2023-11-28T17:09:00.997425+0000","last_became_active":"2023-11-28T17:09:00.259506+0000","last_became_peered":"2023-11-28T17:09:00.259506+0000","last_unstale":"2023-11-28T17:09:00.997425+0000","last_undegraded":"2023-11-28T17:09:00.997425+0000","last_fullsized":"2023-11-28T17:09:00.997425+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:19.098061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223644+0000","last_change":"2023-11-28T17:09:01.017611+0000","last_active":"2023-11-28T17:09:02.223644+0000","last_peered":"2023-11-28T17:09:02.223644+0000","last_clean":"2023-11-28T17:09:02.223644+0000","last_became_active":"2023-11-28T17:09:01.017269+0000","last_became_peered":"2023-11-28T17:09:01.017269+0000","last_unstale":"2023-11-28T17:09:02.223644+0000","last_undegraded":"2023-11-28T17:09:02.223644+0000","last_fullsized":"2023-11-28T17:09:02.223644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:45.964344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.272853+0000","last_change":"2023-11-28T17:09:00.264904+0000","last_active":"2023-11-28T17:09:01.272853+0000","last_peered":"2023-11-28T17:09:01.272853+0000","last_clean":"2023-11-28T17:09:01.272853+0000","last_became_active":"2023-11-28T17:09:00.264645+0000","last_became_peered":"2023-11-28T17:09:00.264645+0000","last_unstale":"2023-11-28T17:09:01.272853+0000","last_undegraded":"2023-11-28T17:09:01.272853+0000","last_fullsized":"2023-11-28T17:09:01.272853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:25:59.069890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987426+0000","last_change":"2023-11-28T17:09:01.005502+0000","last_active":"2023-11-28T17:09:01.987426+0000","last_peered":"2023-11-28T17:09:01.987426+0000","last_clean":"2023-11-28T17:09:01.987426+0000","last_became_active":"2023-11-28T17:09:01.004783+0000","last_became_peered":"2023-11-28T17:09:01.004783+0000","last_unstale":"2023-11-28T17:09:01.987426+0000","last_undegraded":"2023-11-28T17:09:01.987426+0000","last_fullsized":"2023-11-28T17:09:01.987426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:10:50.736315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276609+0000","last_change":"2023-11-28T17:09:00.263414+0000","last_active":"2023-11-28T17:09:01.276609+0000","last_peered":"2023-11-28T17:09:01.276609+0000","last_clean":"2023-11-28T17:09:01.276609+0000","last_became_active":"2023-11-28T17:09:00.260801+0000","last_became_peered":"2023-11-28T17:09:00.260801+0000","last_unstale":"2023-11-28T17:09:01.276609+0000","last_undegraded":"2023-11-28T17:09:01.276609+0000","last_fullsized":"2023-11-28T17:09:01.276609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:31:18.731288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513670+0000","last_change":"2023-11-28T17:09:01.019149+0000","last_active":"2023-11-28T17:09:02.513670+0000","last_peered":"2023-11-28T17:09:02.513670+0000","last_clean":"2023-11-28T17:09:02.513670+0000","last_became_active":"2023-11-28T17:09:01.018834+0000","last_became_peered":"2023-11-28T17:09:01.018834+0000","last_unstale":"2023-11-28T17:09:02.513670+0000","last_undegraded":"2023-11-28T17:09:02.513670+0000","last_fullsized":"2023-11-28T17:09:02.513670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:34.895213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.275750+0000","last_change":"2023-11-28T17:09:00.271771+0000","last_active":"2023-11-28T17:09:01.275750+0000","last_peered":"2023-11-28T17:09:01.275750+0000","last_clean":"2023-11-28T17:09:01.275750+0000","last_became_active":"2023-11-28T17:09:00.269930+0000","last_became_peered":"2023-11-28T17:09:00.269930+0000","last_unstale":"2023-11-28T17:09:01.275750+0000","last_undegraded":"2023-11-28T17:09:01.275750+0000","last_fullsized":"2023-11-28T17:09:01.275750+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:33.502573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987179+0000","last_change":"2023-11-28T17:09:00.999606+0000","last_active":"2023-11-28T17:09:01.987179+0000","last_peered":"2023-11-28T17:09:01.987179+0000","last_clean":"2023-11-28T17:09:01.987179+0000","last_became_active":"2023-11-28T17:09:00.999346+0000","last_became_peered":"2023-11-28T17:09:00.999346+0000","last_unstale":"2023-11-28T17:09:01.987179+0000","last_undegraded":"2023-11-28T17:09:01.987179+0000","last_fullsized":"2023-11-28T17:09:01.987179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:25.324961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.991340+0000","last_change":"2023-11-28T17:09:00.269952+0000","last_active":"2023-11-28T17:09:00.991340+0000","last_peered":"2023-11-28T17:09:00.991340+0000","last_clean":"2023-11-28T17:09:00.991340+0000","last_became_active":"2023-11-28T17:09:00.269539+0000","last_became_peered":"2023-11-28T17:09:00.269539+0000","last_unstale":"2023-11-28T17:09:00.991340+0000","last_undegraded":"2023-11-28T17:09:00.991340+0000","last_fullsized":"2023-11-28T17:09:00.991340+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:58.974841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987385+0000","last_change":"2023-11-28T17:09:01.009043+0000","last_active":"2023-11-28T17:09:01.987385+0000","last_peered":"2023-11-28T17:09:01.987385+0000","last_clean":"2023-11-28T17:09:01.987385+0000","last_became_active":"2023-11-28T17:09:01.008739+0000","last_became_peered":"2023-11-28T17:09:01.008739+0000","last_unstale":"2023-11-28T17:09:01.987385+0000","last_undegraded":"2023-11-28T17:09:01.987385+0000","last_fullsized":"2023-11-28T17:09:01.987385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:49.417143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.999207+0000","last_change":"2023-11-28T17:09:00.270474+0000","last_active":"2023-11-28T17:09:00.999207+0000","last_peered":"2023-11-28T17:09:00.999207+0000","last_clean":"2023-11-28T17:09:00.999207+0000","last_became_active":"2023-11-28T17:09:00.270277+0000","last_became_peered":"2023-11-28T17:09:00.270277+0000","last_unstale":"2023-11-28T17:09:00.999207+0000","last_undegraded":"2023-11-28T17:09:00.999207+0000","last_fullsized":"2023-11-28T17:09:00.999207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:52.364620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.511767+0000","last_change":"2023-11-28T17:09:01.016931+0000","last_active":"2023-11-28T17:09:02.511767+0000","last_peered":"2023-11-28T17:09:02.511767+0000","last_clean":"2023-11-28T17:09:02.511767+0000","last_became_active":"2023-11-28T17:09:01.016673+0000","last_became_peered":"2023-11-28T17:09:01.016673+0000","last_unstale":"2023-11-28T17:09:02.511767+0000","last_undegraded":"2023-11-28T17:09:02.511767+0000","last_fullsized":"2023-11-28T17:09:02.511767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:36.438412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.274020+0000","last_change":"2023-11-28T17:09:00.262584+0000","last_active":"2023-11-28T17:09:01.274020+0000","last_peered":"2023-11-28T17:09:01.274020+0000","last_clean":"2023-11-28T17:09:01.274020+0000","last_became_active":"2023-11-28T17:09:00.262259+0000","last_became_peered":"2023-11-28T17:09:00.262259+0000","last_unstale":"2023-11-28T17:09:01.274020+0000","last_undegraded":"2023-11-28T17:09:01.274020+0000","last_fullsized":"2023-11-28T17:09:01.274020+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:31.377077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986947+0000","last_change":"2023-11-28T17:09:01.003188+0000","last_active":"2023-11-28T17:09:01.986947+0000","last_peered":"2023-11-28T17:09:01.986947+0000","last_clean":"2023-11-28T17:09:01.986947+0000","last_became_active":"2023-11-28T17:09:01.000643+0000","last_became_peered":"2023-11-28T17:09:01.000643+0000","last_unstale":"2023-11-28T17:09:01.986947+0000","last_undegraded":"2023-11-28T17:09:01.986947+0000","last_fullsized":"2023-11-28T17:09:01.986947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:49:27.242284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273771+0000","last_change":"2023-11-28T17:09:00.272398+0000","last_active":"2023-11-28T17:09:01.273771+0000","last_peered":"2023-11-28T17:09:01.273771+0000","last_clean":"2023-11-28T17:09:01.273771+0000","last_became_active":"2023-11-28T17:09:00.271617+0000","last_became_peered":"2023-11-28T17:09:00.271617+0000","last_unstale":"2023-11-28T17:09:01.273771+0000","last_undegraded":"2023-11-28T17:09:01.273771+0000","last_fullsized":"2023-11-28T17:09:01.273771+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:10.177210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.516054+0000","last_change":"2023-11-28T17:09:01.012920+0000","last_active":"2023-11-28T17:09:02.516054+0000","last_peered":"2023-11-28T17:09:02.516054+0000","last_clean":"2023-11-28T17:09:02.516054+0000","last_became_active":"2023-11-28T17:09:01.012651+0000","last_became_peered":"2023-11-28T17:09:01.012651+0000","last_unstale":"2023-11-28T17:09:02.516054+0000","last_undegraded":"2023-11-28T17:09:02.516054+0000","last_fullsized":"2023-11-28T17:09:02.516054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:53.375742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273900+0000","last_change":"2023-11-28T17:09:00.271952+0000","last_active":"2023-11-28T17:09:01.273900+0000","last_peered":"2023-11-28T17:09:01.273900+0000","last_clean":"2023-11-28T17:09:01.273900+0000","last_became_active":"2023-11-28T17:09:00.271641+0000","last_became_peered":"2023-11-28T17:09:00.271641+0000","last_unstale":"2023-11-28T17:09:01.273900+0000","last_undegraded":"2023-11-28T17:09:01.273900+0000","last_fullsized":"2023-11-28T17:09:01.273900+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:52.364620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513991+0000","last_change":"2023-11-28T17:09:01.014412+0000","last_active":"2023-11-28T17:09:02.513991+0000","last_peered":"2023-11-28T17:09:02.513991+0000","last_clean":"2023-11-28T17:09:02.513991+0000","last_became_active":"2023-11-28T17:09:01.014145+0000","last_became_peered":"2023-11-28T17:09:01.014145+0000","last_unstale":"2023-11-28T17:09:02.513991+0000","last_undegraded":"2023-11-28T17:09:02.513991+0000","last_fullsized":"2023-11-28T17:09:02.513991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:49:27.242284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273346+0000","last_change":"2023-11-28T17:09:00.264424+0000","last_active":"2023-11-28T17:09:01.273346+0000","last_peered":"2023-11-28T17:09:01.273346+0000","last_clean":"2023-11-28T17:09:01.273346+0000","last_became_active":"2023-11-28T17:09:00.263195+0000","last_became_peered":"2023-11-28T17:09:00.263195+0000","last_unstale":"2023-11-28T17:09:01.273346+0000","last_undegraded":"2023-11-28T17:09:01.273346+0000","last_fullsized":"2023-11-28T17:09:01.273346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:35.427291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513525+0000","last_change":"2023-11-28T17:09:01.019072+0000","last_active":"2023-11-28T17:09:02.513525+0000","last_peered":"2023-11-28T17:09:02.513525+0000","last_clean":"2023-11-28T17:09:02.513525+0000","last_became_active":"2023-11-28T17:09:01.018785+0000","last_became_peered":"2023-11-28T17:09:01.018785+0000","last_unstale":"2023-11-28T17:09:02.513525+0000","last_undegraded":"2023-11-28T17:09:02.513525+0000","last_fullsized":"2023-11-28T17:09:02.513525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:59.985963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.984805+0000","last_change":"2023-11-28T17:09:00.268057+0000","last_active":"2023-11-28T17:09:00.984805+0000","last_peered":"2023-11-28T17:09:00.984805+0000","last_clean":"2023-11-28T17:09:00.984805+0000","last_became_active":"2023-11-28T17:09:00.264298+0000","last_became_peered":"2023-11-28T17:09:00.264298+0000","last_unstale":"2023-11-28T17:09:00.984805+0000","last_undegraded":"2023-11-28T17:09:00.984805+0000","last_fullsized":"2023-11-28T17:09:00.984805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:35.427291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988219+0000","last_change":"2023-11-28T17:09:01.009062+0000","last_active":"2023-11-28T17:09:01.988219+0000","last_peered":"2023-11-28T17:09:01.988219+0000","last_clean":"2023-11-28T17:09:01.988219+0000","last_became_active":"2023-11-28T17:09:01.008759+0000","last_became_peered":"2023-11-28T17:09:01.008759+0000","last_unstale":"2023-11-28T17:09:01.988219+0000","last_undegraded":"2023-11-28T17:09:01.988219+0000","last_fullsized":"2023-11-28T17:09:01.988219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:09:50.534329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.986684+0000","last_change":"2023-11-28T17:08:51.392782+0000","last_active":"2023-11-28T17:09:00.986684+0000","last_peered":"2023-11-28T17:09:00.986684+0000","last_clean":"2023-11-28T17:09:00.986684+0000","last_became_active":"2023-11-28T17:08:51.390415+0000","last_became_peered":"2023-11-28T17:08:51.390415+0000","last_unstale":"2023-11-28T17:09:00.986684+0000","last_undegraded":"2023-11-28T17:09:00.986684+0000","last_fullsized":"2023-11-28T17:09:00.986684+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_clean_scrub_stamp":"2023-11-28T17:08:49.967682+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:03:32.638685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.985042+0000","last_change":"2023-11-28T17:09:00.267795+0000","last_active":"2023-11-28T17:09:00.985042+0000","last_peered":"2023-11-28T17:09:00.985042+0000","last_clean":"2023-11-28T17:09:00.985042+0000","last_became_active":"2023-11-28T17:09:00.264764+0000","last_became_peered":"2023-11-28T17:09:00.264764+0000","last_unstale":"2023-11-28T17:09:00.985042+0000","last_undegraded":"2023-11-28T17:09:00.985042+0000","last_fullsized":"2023-11-28T17:09:00.985042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:59.024219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988457+0000","last_change":"2023-11-28T17:09:01.004564+0000","last_active":"2023-11-28T17:09:01.988457+0000","last_peered":"2023-11-28T17:09:01.988457+0000","last_clean":"2023-11-28T17:09:01.988457+0000","last_became_active":"2023-11-28T17:09:01.004189+0000","last_became_peered":"2023-11-28T17:09:01.004189+0000","last_unstale":"2023-11-28T17:09:01.988457+0000","last_undegraded":"2023-11-28T17:09:01.988457+0000","last_fullsized":"2023-11-28T17:09:01.988457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:09.387207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.986785+0000","last_change":"2023-11-28T17:09:00.269480+0000","last_active":"2023-11-28T17:09:00.986785+0000","last_peered":"2023-11-28T17:09:00.986785+0000","last_clean":"2023-11-28T17:09:00.986785+0000","last_became_active":"2023-11-28T17:09:00.269032+0000","last_became_peered":"2023-11-28T17:09:00.269032+0000","last_unstale":"2023-11-28T17:09:00.986785+0000","last_undegraded":"2023-11-28T17:09:00.986785+0000","last_fullsized":"2023-11-28T17:09:00.986785+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:49.306608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987055+0000","last_change":"2023-11-28T17:09:01.007809+0000","last_active":"2023-11-28T17:09:01.987055+0000","last_peered":"2023-11-28T17:09:01.987055+0000","last_clean":"2023-11-28T17:09:01.987055+0000","last_became_active":"2023-11-28T17:09:01.007508+0000","last_became_peered":"2023-11-28T17:09:01.007508+0000","last_unstale":"2023-11-28T17:09:01.987055+0000","last_undegraded":"2023-11-28T17:09:01.987055+0000","last_fullsized":"2023-11-28T17:09:01.987055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:20.109182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.001078+0000","last_change":"2023-11-28T17:09:00.271320+0000","last_active":"2023-11-28T17:09:01.001078+0000","last_peered":"2023-11-28T17:09:01.001078+0000","last_clean":"2023-11-28T17:09:01.001078+0000","last_became_active":"2023-11-28T17:09:00.271019+0000","last_became_peered":"2023-11-28T17:09:00.271019+0000","last_unstale":"2023-11-28T17:09:01.001078+0000","last_undegraded":"2023-11-28T17:09:01.001078+0000","last_fullsized":"2023-11-28T17:09:01.001078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:41:19.939145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222456+0000","last_change":"2023-11-28T17:09:01.009687+0000","last_active":"2023-11-28T17:09:02.222456+0000","last_peered":"2023-11-28T17:09:02.222456+0000","last_clean":"2023-11-28T17:09:02.222456+0000","last_became_active":"2023-11-28T17:09:01.009329+0000","last_became_peered":"2023-11-28T17:09:01.009329+0000","last_unstale":"2023-11-28T17:09:02.222456+0000","last_undegraded":"2023-11-28T17:09:02.222456+0000","last_fullsized":"2023-11-28T17:09:02.222456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:50.317730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.991649+0000","last_change":"2023-11-28T17:09:00.269333+0000","last_active":"2023-11-28T17:09:00.991649+0000","last_peered":"2023-11-28T17:09:00.991649+0000","last_clean":"2023-11-28T17:09:00.991649+0000","last_became_active":"2023-11-28T17:09:00.267481+0000","last_became_peered":"2023-11-28T17:09:00.267481+0000","last_unstale":"2023-11-28T17:09:00.991649+0000","last_undegraded":"2023-11-28T17:09:00.991649+0000","last_fullsized":"2023-11-28T17:09:00.991649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:21.776220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512505+0000","last_change":"2023-11-28T17:09:01.012458+0000","last_active":"2023-11-28T17:09:02.512505+0000","last_peered":"2023-11-28T17:09:02.512505+0000","last_clean":"2023-11-28T17:09:02.512505+0000","last_became_active":"2023-11-28T17:09:01.011811+0000","last_became_peered":"2023-11-28T17:09:01.011811+0000","last_unstale":"2023-11-28T17:09:02.512505+0000","last_undegraded":"2023-11-28T17:09:02.512505+0000","last_fullsized":"2023-11-28T17:09:02.512505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:24.354633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.999956+0000","last_change":"2023-11-28T17:09:00.263517+0000","last_active":"2023-11-28T17:09:00.999956+0000","last_peered":"2023-11-28T17:09:00.999956+0000","last_clean":"2023-11-28T17:09:00.999956+0000","last_became_active":"2023-11-28T17:09:00.263243+0000","last_became_peered":"2023-11-28T17:09:00.263243+0000","last_unstale":"2023-11-28T17:09:00.999956+0000","last_undegraded":"2023-11-28T17:09:00.999956+0000","last_fullsized":"2023-11-28T17:09:00.999956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:10.177210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515382+0000","last_change":"2023-11-28T17:09:00.997629+0000","last_active":"2023-11-28T17:09:02.515382+0000","last_peered":"2023-11-28T17:09:02.515382+0000","last_clean":"2023-11-28T17:09:02.515382+0000","last_became_active":"2023-11-28T17:09:00.997360+0000","last_became_peered":"2023-11-28T17:09:00.997360+0000","last_unstale":"2023-11-28T17:09:02.515382+0000","last_undegraded":"2023-11-28T17:09:02.515382+0000","last_fullsized":"2023-11-28T17:09:02.515382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:32.388198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.988631+0000","last_change":"2023-11-28T17:09:00.268166+0000","last_active":"2023-11-28T17:09:00.988631+0000","last_peered":"2023-11-28T17:09:00.988631+0000","last_clean":"2023-11-28T17:09:00.988631+0000","last_became_active":"2023-11-28T17:09:00.267609+0000","last_became_peered":"2023-11-28T17:09:00.267609+0000","last_unstale":"2023-11-28T17:09:00.988631+0000","last_undegraded":"2023-11-28T17:09:00.988631+0000","last_fullsized":"2023-11-28T17:09:00.988631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:23:26.227274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.989182+0000","last_change":"2023-11-28T17:09:01.017987+0000","last_active":"2023-11-28T17:09:01.989182+0000","last_peered":"2023-11-28T17:09:01.989182+0000","last_clean":"2023-11-28T17:09:01.989182+0000","last_became_active":"2023-11-28T17:09:01.017491+0000","last_became_peered":"2023-11-28T17:09:01.017491+0000","last_unstale":"2023-11-28T17:09:01.989182+0000","last_undegraded":"2023-11-28T17:09:01.989182+0000","last_fullsized":"2023-11-28T17:09:01.989182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:09.387207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.990667+0000","last_change":"2023-11-28T17:09:00.267582+0000","last_active":"2023-11-28T17:09:01.990667+0000","last_peered":"2023-11-28T17:09:01.990667+0000","last_clean":"2023-11-28T17:09:01.990667+0000","last_became_active":"2023-11-28T17:09:00.267245+0000","last_became_peered":"2023-11-28T17:09:00.267245+0000","last_unstale":"2023-11-28T17:09:01.990667+0000","last_undegraded":"2023-11-28T17:09:01.990667+0000","last_fullsized":"2023-11-28T17:09:01.990667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:59.024219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.990398+0000","last_change":"2023-11-28T17:09:00.997521+0000","last_active":"2023-11-28T17:09:01.990398+0000","last_peered":"2023-11-28T17:09:01.990398+0000","last_clean":"2023-11-28T17:09:01.990398+0000","last_became_active":"2023-11-28T17:09:00.997041+0000","last_became_peered":"2023-11-28T17:09:00.997041+0000","last_unstale":"2023-11-28T17:09:01.990398+0000","last_undegraded":"2023-11-28T17:09:01.990398+0000","last_fullsized":"2023-11-28T17:09:01.990398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:07.593383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276430+0000","last_change":"2023-11-28T17:09:00.264837+0000","last_active":"2023-11-28T17:09:01.276430+0000","last_peered":"2023-11-28T17:09:01.276430+0000","last_clean":"2023-11-28T17:09:01.276430+0000","last_became_active":"2023-11-28T17:09:00.264570+0000","last_became_peered":"2023-11-28T17:09:00.264570+0000","last_unstale":"2023-11-28T17:09:01.276430+0000","last_undegraded":"2023-11-28T17:09:01.276430+0000","last_fullsized":"2023-11-28T17:09:01.276430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:23.343512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.990475+0000","last_change":"2023-11-28T17:09:01.006414+0000","last_active":"2023-11-28T17:09:01.990475+0000","last_peered":"2023-11-28T17:09:01.990475+0000","last_clean":"2023-11-28T17:09:01.990475+0000","last_became_active":"2023-11-28T17:09:01.005570+0000","last_became_peered":"2023-11-28T17:09:01.005570+0000","last_unstale":"2023-11-28T17:09:01.990475+0000","last_undegraded":"2023-11-28T17:09:01.990475+0000","last_fullsized":"2023-11-28T17:09:01.990475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:46:24.200692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.998379+0000","last_change":"2023-11-28T17:09:00.257189+0000","last_active":"2023-11-28T17:09:00.998379+0000","last_peered":"2023-11-28T17:09:00.998379+0000","last_clean":"2023-11-28T17:09:00.998379+0000","last_became_active":"2023-11-28T17:09:00.256862+0000","last_became_peered":"2023-11-28T17:09:00.256862+0000","last_unstale":"2023-11-28T17:09:00.998379+0000","last_undegraded":"2023-11-28T17:09:00.998379+0000","last_fullsized":"2023-11-28T17:09:00.998379+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:24.313840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987313+0000","last_change":"2023-11-28T17:09:01.006232+0000","last_active":"2023-11-28T17:09:01.987313+0000","last_peered":"2023-11-28T17:09:01.987313+0000","last_clean":"2023-11-28T17:09:01.987313+0000","last_became_active":"2023-11-28T17:09:01.005930+0000","last_became_peered":"2023-11-28T17:09:01.005930+0000","last_unstale":"2023-11-28T17:09:01.987313+0000","last_undegraded":"2023-11-28T17:09:01.987313+0000","last_fullsized":"2023-11-28T17:09:01.987313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:06:27.104365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.991766+0000","last_change":"2023-11-28T17:09:00.266555+0000","last_active":"2023-11-28T17:09:00.991766+0000","last_peered":"2023-11-28T17:09:00.991766+0000","last_clean":"2023-11-28T17:09:00.991766+0000","last_became_active":"2023-11-28T17:09:00.266279+0000","last_became_peered":"2023-11-28T17:09:00.266279+0000","last_unstale":"2023-11-28T17:09:00.991766+0000","last_undegraded":"2023-11-28T17:09:00.991766+0000","last_fullsized":"2023-11-28T17:09:00.991766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:19.098061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988255+0000","last_change":"2023-11-28T17:09:01.007494+0000","last_active":"2023-11-28T17:09:01.988255+0000","last_peered":"2023-11-28T17:09:01.988255+0000","last_clean":"2023-11-28T17:09:01.988255+0000","last_became_active":"2023-11-28T17:09:01.007121+0000","last_became_peered":"2023-11-28T17:09:01.007121+0000","last_unstale":"2023-11-28T17:09:01.988255+0000","last_undegraded":"2023-11-28T17:09:01.988255+0000","last_fullsized":"2023-11-28T17:09:01.988255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:11.188332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.984942+0000","last_change":"2023-11-28T17:09:00.267476+0000","last_active":"2023-11-28T17:09:00.984942+0000","last_peered":"2023-11-28T17:09:00.984942+0000","last_clean":"2023-11-28T17:09:00.984942+0000","last_became_active":"2023-11-28T17:09:00.267138+0000","last_became_peered":"2023-11-28T17:09:00.267138+0000","last_unstale":"2023-11-28T17:09:00.984942+0000","last_undegraded":"2023-11-28T17:09:00.984942+0000","last_fullsized":"2023-11-28T17:09:00.984942+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:11.303292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512236+0000","last_change":"2023-11-28T17:09:01.008773+0000","last_active":"2023-11-28T17:09:02.512236+0000","last_peered":"2023-11-28T17:09:02.512236+0000","last_clean":"2023-11-28T17:09:02.512236+0000","last_became_active":"2023-11-28T17:09:01.008241+0000","last_became_peered":"2023-11-28T17:09:01.008241+0000","last_unstale":"2023-11-28T17:09:02.512236+0000","last_undegraded":"2023-11-28T17:09:02.512236+0000","last_fullsized":"2023-11-28T17:09:02.512236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:12.314413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276631+0000","last_change":"2023-11-28T17:09:00.267671+0000","last_active":"2023-11-28T17:09:01.276631+0000","last_peered":"2023-11-28T17:09:01.276631+0000","last_clean":"2023-11-28T17:09:01.276631+0000","last_became_active":"2023-11-28T17:09:00.264445+0000","last_became_peered":"2023-11-28T17:09:00.264445+0000","last_unstale":"2023-11-28T17:09:01.276631+0000","last_undegraded":"2023-11-28T17:09:01.276631+0000","last_fullsized":"2023-11-28T17:09:01.276631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:52.364620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987244+0000","last_change":"2023-11-28T17:09:01.003981+0000","last_active":"2023-11-28T17:09:01.987244+0000","last_peered":"2023-11-28T17:09:01.987244+0000","last_clean":"2023-11-28T17:09:01.987244+0000","last_became_active":"2023-11-28T17:09:01.003508+0000","last_became_peered":"2023-11-28T17:09:01.003508+0000","last_unstale":"2023-11-28T17:09:01.987244+0000","last_undegraded":"2023-11-28T17:09:01.987244+0000","last_fullsized":"2023-11-28T17:09:01.987244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:48:31.212857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988121+0000","last_change":"2023-11-28T17:09:01.012859+0000","last_active":"2023-11-28T17:09:01.988121+0000","last_peered":"2023-11-28T17:09:01.988121+0000","last_clean":"2023-11-28T17:09:01.988121+0000","last_became_active":"2023-11-28T17:09:01.010930+0000","last_became_peered":"2023-11-28T17:09:01.010930+0000","last_unstale":"2023-11-28T17:09:01.988121+0000","last_undegraded":"2023-11-28T17:09:01.988121+0000","last_fullsized":"2023-11-28T17:09:01.988121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:46:01.399129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.268661+0000","last_change":"2023-11-28T17:09:00.259200+0000","last_active":"2023-11-28T17:09:01.268661+0000","last_peered":"2023-11-28T17:09:01.268661+0000","last_clean":"2023-11-28T17:09:01.268661+0000","last_became_active":"2023-11-28T17:09:00.258886+0000","last_became_peered":"2023-11-28T17:09:00.258886+0000","last_unstale":"2023-11-28T17:09:01.268661+0000","last_undegraded":"2023-11-28T17:09:01.268661+0000","last_fullsized":"2023-11-28T17:09:01.268661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:43.640066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.514743+0000","last_change":"2023-11-28T17:09:01.015028+0000","last_active":"2023-11-28T17:09:02.514743+0000","last_peered":"2023-11-28T17:09:02.514743+0000","last_clean":"2023-11-28T17:09:02.514743+0000","last_became_active":"2023-11-28T17:09:01.014670+0000","last_became_peered":"2023-11-28T17:09:01.014670+0000","last_unstale":"2023-11-28T17:09:02.514743+0000","last_undegraded":"2023-11-28T17:09:02.514743+0000","last_fullsized":"2023-11-28T17:09:02.514743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:31.569896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.268602+0000","last_change":"2023-11-28T17:09:00.263220+0000","last_active":"2023-11-28T17:09:01.268602+0000","last_peered":"2023-11-28T17:09:01.268602+0000","last_clean":"2023-11-28T17:09:01.268602+0000","last_became_active":"2023-11-28T17:09:00.262930+0000","last_became_peered":"2023-11-28T17:09:00.262930+0000","last_unstale":"2023-11-28T17:09:01.268602+0000","last_undegraded":"2023-11-28T17:09:01.268602+0000","last_fullsized":"2023-11-28T17:09:01.268602+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:06.582262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223461+0000","last_change":"2023-11-28T17:09:01.014772+0000","last_active":"2023-11-28T17:09:02.223461+0000","last_peered":"2023-11-28T17:09:02.223461+0000","last_clean":"2023-11-28T17:09:02.223461+0000","last_became_active":"2023-11-28T17:09:01.013249+0000","last_became_peered":"2023-11-28T17:09:01.013249+0000","last_unstale":"2023-11-28T17:09:02.223461+0000","last_undegraded":"2023-11-28T17:09:02.223461+0000","last_fullsized":"2023-11-28T17:09:02.223461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:34.895213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273335+0000","last_change":"2023-11-28T17:09:00.270822+0000","last_active":"2023-11-28T17:09:01.273335+0000","last_peered":"2023-11-28T17:09:01.273335+0000","last_clean":"2023-11-28T17:09:01.273335+0000","last_became_active":"2023-11-28T17:09:00.268995+0000","last_became_peered":"2023-11-28T17:09:00.268995+0000","last_unstale":"2023-11-28T17:09:01.273335+0000","last_undegraded":"2023-11-28T17:09:01.273335+0000","last_fullsized":"2023-11-28T17:09:01.273335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:24.313840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223456+0000","last_change":"2023-11-28T17:09:01.012994+0000","last_active":"2023-11-28T17:09:02.223456+0000","last_peered":"2023-11-28T17:09:02.223456+0000","last_clean":"2023-11-28T17:09:02.223456+0000","last_became_active":"2023-11-28T17:09:01.012496+0000","last_became_peered":"2023-11-28T17:09:01.012496+0000","last_unstale":"2023-11-28T17:09:02.223456+0000","last_undegraded":"2023-11-28T17:09:02.223456+0000","last_fullsized":"2023-11-28T17:09:02.223456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:59.985963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276064+0000","last_change":"2023-11-28T17:09:00.265200+0000","last_active":"2023-11-28T17:09:01.276064+0000","last_peered":"2023-11-28T17:09:01.276064+0000","last_clean":"2023-11-28T17:09:01.276064+0000","last_became_active":"2023-11-28T17:09:00.264913+0000","last_became_peered":"2023-11-28T17:09:00.264913+0000","last_unstale":"2023-11-28T17:09:01.276064+0000","last_undegraded":"2023-11-28T17:09:01.276064+0000","last_fullsized":"2023-11-28T17:09:01.276064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:24.313840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7270000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4510000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.409}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3579999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7809999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6140000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":806912,"data_stored":661370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2569999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7130000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":73194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.056}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.875}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.944}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2120000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2650000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5379999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3069999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6280000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2879999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3570000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3740000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5620000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7170000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.788}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3260000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6949999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5720000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.169}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":70966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.456}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.516}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.421}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T17:09:11.266 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:09:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.512+0000 7f6677fff640 1 Processor -- start 2023-11-28T17:09:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.512+0000 7f6677fff640 1 -- start start 2023-11-28T17:09:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.514+0000 7f6677fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f66780701f0 0x7f667815ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.514+0000 7f6677fff640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780cd8e0 con 0x7f66780cf2a0 2023-11-28T17:09:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.514+0000 7f6677fff640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66780706a0 con 0x7f6678159f30 2023-11-28T17:09:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.514+0000 7f6677fff640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f667815b3d0 con 0x7f66780701f0 2023-11-28T17:09:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.514+0000 7f66767fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f66780cf2a0 0x7f66780b1f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54690/0 (socket says 172.21.15.116:54690) 2023-11-28T17:09:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.514+0000 7f66767fc640 1 -- 172.21.15.116:0/4161488024 learned_addr learned my addr 172.21.15.116:0/4161488024 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.514+0000 7f6675ffb640 1 -- 172.21.15.116:0/4161488024 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580583380 0 0) 0x7f66780cd8e0 con 0x7f66780cf2a0 2023-11-28T17:09:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.514+0000 7f6675ffb640 1 -- 172.21.15.116:0/4161488024 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6660003520 con 0x7f66780cf2a0 2023-11-28T17:09:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.514+0000 7f66777fe640 1 --2- 172.21.15.116:0/4161488024 >> v2:172.21.15.156:3300/0 conn(0x7f66780701f0 0x7f667815ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.514+0000 7f6675ffb640 1 -- 172.21.15.116:0/4161488024 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3972982725 0 0) 0x7f6660003520 con 0x7f66780cf2a0 2023-11-28T17:09:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.514+0000 7f6675ffb640 1 -- 172.21.15.116:0/4161488024 >> v2:172.21.15.156:3300/0 conn(0x7f66780701f0 msgr2=0x7f667815ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6675ffb640 1 --2- 172.21.15.116:0/4161488024 >> v2:172.21.15.156:3300/0 conn(0x7f66780701f0 0x7f667815ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6675ffb640 1 -- 172.21.15.116:0/4161488024 >> v1:172.21.15.156:6789/0 conn(0x7f6678159f30 legacy=0x7f66780ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6675ffb640 1 -- 172.21.15.116:0/4161488024 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6678070920 con 0x7f66780cf2a0 2023-11-28T17:09:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6675ffb640 1 -- 172.21.15.116:0/4161488024 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f666c001670 con 0x7f66780cf2a0 2023-11-28T17:09:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6675ffb640 1 -- 172.21.15.116:0/4161488024 >> v1:172.21.15.116:6789/0 conn(0x7f66780cf2a0 legacy=0x7f66780b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6675ffb640 1 --2- 172.21.15.116:0/4161488024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66600039c0 0x7f6660003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6675ffb640 1 --2- 172.21.15.116:0/4161488024 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f66600047c0 0x7f6660004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6675ffb640 1 --2- 172.21.15.116:0/4161488024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66600051d0 0x7f666000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6675ffb640 1 -- 172.21.15.116:0/4161488024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666000c0d0 con 0x7f66600039c0 2023-11-28T17:09:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6675ffb640 1 -- 172.21.15.116:0/4161488024 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666000c240 con 0x7f66600051d0 2023-11-28T17:09:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6675ffb640 1 -- 172.21.15.116:0/4161488024 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f666000c400 con 0x7f66600047c0 2023-11-28T17:09:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6676ffd640 1 --2- 172.21.15.116:0/4161488024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66600039c0 0x7f6660003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6676ffd640 1 -- 172.21.15.116:0/4161488024 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f66600047c0 msgr2=0x7f6660004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6676ffd640 1 --2- 172.21.15.116:0/4161488024 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f66600047c0 0x7f6660004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6676ffd640 1 -- 172.21.15.116:0/4161488024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66600051d0 msgr2=0x7f666000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:11.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6676ffd640 1 --2- 172.21.15.116:0/4161488024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66600051d0 0x7f666000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f6676ffd640 1 -- 172.21.15.116:0/4161488024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6660003520 con 0x7f66600039c0 2023-11-28T17:09:11.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.515+0000 7f66777fe640 1 -- 172.21.15.116:0/4161488024 reap_dead start 2023-11-28T17:09:11.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.516+0000 7f6676ffd640 1 --2- 172.21.15.116:0/4161488024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66600039c0 0x7f6660003dc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6668003d80 tx=0x7f66600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6fdee83b676f4fe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:11.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.516+0000 7f6675ffb640 1 -- 172.21.15.116:0/4161488024 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6668032600 con 0x7f66600039c0 2023-11-28T17:09:11.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.516+0000 7f6675ffb640 1 -- 172.21.15.116:0/4161488024 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666802e050 con 0x7f66600039c0 2023-11-28T17:09:11.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.516+0000 7f6675ffb640 1 -- 172.21.15.116:0/4161488024 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66680329e0 con 0x7f66600039c0 2023-11-28T17:09:11.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.517+0000 7f6677fff640 1 -- 172.21.15.116:0/4161488024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66600039c0 msgr2=0x7f6660003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.517+0000 7f6677fff640 1 --2- 172.21.15.116:0/4161488024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66600039c0 0x7f6660003dc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6668003d80 tx=0x7f66600057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.517+0000 7f6677fff640 1 -- 172.21.15.116:0/4161488024 shutdown_connections 2023-11-28T17:09:11.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.517+0000 7f6677fff640 1 --2- 172.21.15.116:0/4161488024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66600039c0 0x7f6660003dc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.517+0000 7f6677fff640 1 -- 172.21.15.116:0/4161488024 >> 172.21.15.116:0/4161488024 conn(0x7f66780b0950 msgr2=0x7f66780b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:11.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.517+0000 7f6677fff640 1 -- 172.21.15.116:0/4161488024 shutdown_connections 2023-11-28T17:09:11.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.517+0000 7f6677fff640 1 -- 172.21.15.116:0/4161488024 wait complete. 2023-11-28T17:09:11.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.519+0000 7f6677fff640 1 Processor -- start 2023-11-28T17:09:11.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.519+0000 7f6677fff640 1 -- start start 2023-11-28T17:09:11.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.520+0000 7f6677fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66600039c0 0x7f667815be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.520+0000 7f6677fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6678143d80 0x7f6678148510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.520+0000 7f6677fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6678148a50 0x7f667814f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.520+0000 7f6677fff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66780ac1c0 con 0x7f66600039c0 2023-11-28T17:09:11.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.520+0000 7f6677fff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66780acd40 con 0x7f6678143d80 2023-11-28T17:09:11.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.520+0000 7f6677fff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66780ac410 con 0x7f6678148a50 2023-11-28T17:09:11.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.520+0000 7f6676ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66600039c0 0x7f667815be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.520+0000 7f6676ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66600039c0 0x7f667815be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52152/0 (socket says 172.21.15.116:52152) 2023-11-28T17:09:11.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.520+0000 7f6676ffd640 1 -- 172.21.15.116:0/2497246198 learned_addr learned my addr 172.21.15.116:0/2497246198 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:11.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.521+0000 7f6676ffd640 1 -- 172.21.15.116:0/2497246198 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6678148a50 msgr2=0x7f667814f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.521+0000 7f6676ffd640 1 --2- 172.21.15.116:0/2497246198 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6678148a50 0x7f667814f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.521+0000 7f6676ffd640 1 -- 172.21.15.116:0/2497246198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6678143d80 msgr2=0x7f6678148510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.521+0000 7f6676ffd640 1 --2- 172.21.15.116:0/2497246198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6678143d80 0x7f6678148510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.521+0000 7f6676ffd640 1 -- 172.21.15.116:0/2497246198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780ac6f0 con 0x7f66600039c0 2023-11-28T17:09:11.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.521+0000 7f6676ffd640 1 --2- 172.21.15.116:0/2497246198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66600039c0 0x7f667815be50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6668003d80 tx=0x7f66680036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:11.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.521+0000 7f665ffff640 1 -- 172.21.15.116:0/2497246198 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666803c410 con 0x7f66600039c0 2023-11-28T17:09:11.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.521+0000 7f665ffff640 1 -- 172.21.15.116:0/2497246198 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666803c570 con 0x7f66600039c0 2023-11-28T17:09:11.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.521+0000 7f6677fff640 1 -- 172.21.15.116:0/2497246198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66780ac9d0 con 0x7f66600039c0 2023-11-28T17:09:11.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.521+0000 7f665ffff640 1 -- 172.21.15.116:0/2497246198 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666803c8e0 con 0x7f66600039c0 2023-11-28T17:09:11.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.521+0000 7f6677fff640 1 -- 172.21.15.116:0/2497246198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6678144200 con 0x7f66600039c0 2023-11-28T17:09:11.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.522+0000 7f6677fff640 1 -- 172.21.15.116:0/2497246198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66780ac550 con 0x7f66600039c0 2023-11-28T17:09:11.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.523+0000 7f665ffff640 1 -- 172.21.15.116:0/2497246198 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6668039050 con 0x7f66600039c0 2023-11-28T17:09:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.524+0000 7f665ffff640 1 --2- 172.21.15.116:0/2497246198 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f666406b2a0 0x7f666406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.524+0000 7f665ffff640 1 -- 172.21.15.116:0/2497246198 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66680aaf40 con 0x7f66600039c0 2023-11-28T17:09:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.524+0000 7f66767fc640 1 --2- 172.21.15.116:0/2497246198 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f666406b2a0 0x7f666406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.525+0000 7f66767fc640 1 --2- 172.21.15.116:0/2497246198 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f666406b2a0 0x7f666406d760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6678149e40 tx=0x7f666c001600 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.528+0000 7f665ffff640 1 -- 172.21.15.116:0/2497246198 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66780ac550 con 0x7f66600039c0 2023-11-28T17:09:11.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.702+0000 7f6677fff640 1 -- 172.21.15.116:0/2497246198 --> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f66780ac1c0 con 0x7f666406b2a0 2023-11-28T17:09:11.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.722+0000 7f665ffff640 1 -- 172.21.15.116:0/2497246198 <== mgr.4098 v2:172.21.15.116:6832/1077400178 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264398 (secure 0 0 0) 0x7f66780ac1c0 con 0x7f666406b2a0 2023-11-28T17:09:11.727 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:11.729 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-28T17:09:11.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.728+0000 7f6677fff640 1 -- 172.21.15.116:0/2497246198 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f666406b2a0 msgr2=0x7f666406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.728+0000 7f6677fff640 1 --2- 172.21.15.116:0/2497246198 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f666406b2a0 0x7f666406d760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6678149e40 tx=0x7f666c001600 comp rx=0 tx=0).stop 2023-11-28T17:09:11.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.728+0000 7f6677fff640 1 -- 172.21.15.116:0/2497246198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66600039c0 msgr2=0x7f667815be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.728+0000 7f6677fff640 1 --2- 172.21.15.116:0/2497246198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66600039c0 0x7f667815be50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6668003d80 tx=0x7f66680036f0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.728+0000 7f6677fff640 1 -- 172.21.15.116:0/2497246198 shutdown_connections 2023-11-28T17:09:11.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.728+0000 7f6677fff640 1 --2- 172.21.15.116:0/2497246198 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f666406b2a0 0x7f666406d760 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.728+0000 7f6677fff640 1 --2- 172.21.15.116:0/2497246198 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6678148a50 0x7f667814f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.728+0000 7f6677fff640 1 --2- 172.21.15.116:0/2497246198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6678143d80 0x7f6678148510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.728+0000 7f6677fff640 1 --2- 172.21.15.116:0/2497246198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66600039c0 0x7f667815be50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.728+0000 7f6677fff640 1 -- 172.21.15.116:0/2497246198 >> 172.21.15.116:0/2497246198 conn(0x7f66780b0950 msgr2=0x7f6678159380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:11.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.728+0000 7f6677fff640 1 -- 172.21.15.116:0/2497246198 shutdown_connections 2023-11-28T17:09:11.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.729+0000 7f6677fff640 1 -- 172.21.15.116:0/2497246198 wait complete. 2023-11-28T17:09:11.746 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-11-28T17:09:11.042788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235648,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739072,"statfs":{"total":773094113280,"available":772852809728,"internally_reserved":0,"allocated":2891776,"data_stored":1752836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003611"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513632+0000","last_change":"2023-11-28T17:09:01.014584+0000","last_active":"2023-11-28T17:09:02.513632+0000","last_peered":"2023-11-28T17:09:02.513632+0000","last_clean":"2023-11-28T17:09:02.513632+0000","last_became_active":"2023-11-28T17:09:01.014291+0000","last_became_peered":"2023-11-28T17:09:01.014291+0000","last_unstale":"2023-11-28T17:09:02.513632+0000","last_undegraded":"2023-11-28T17:09:02.513632+0000","last_fullsized":"2023-11-28T17:09:02.513632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:22.787341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223151+0000","last_change":"2023-11-28T17:09:01.009980+0000","last_active":"2023-11-28T17:09:02.223151+0000","last_peered":"2023-11-28T17:09:02.223151+0000","last_clean":"2023-11-28T17:09:02.223151+0000","last_became_active":"2023-11-28T17:09:01.009702+0000","last_became_peered":"2023-11-28T17:09:01.009702+0000","last_unstale":"2023-11-28T17:09:02.223151+0000","last_undegraded":"2023-11-28T17:09:02.223151+0000","last_fullsized":"2023-11-28T17:09:02.223151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:06:59.199376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222772+0000","last_change":"2023-11-28T17:09:01.010483+0000","last_active":"2023-11-28T17:09:02.222772+0000","last_peered":"2023-11-28T17:09:02.222772+0000","last_clean":"2023-11-28T17:09:02.222772+0000","last_became_active":"2023-11-28T17:09:01.010032+0000","last_became_peered":"2023-11-28T17:09:01.010032+0000","last_unstale":"2023-11-28T17:09:02.222772+0000","last_undegraded":"2023-11-28T17:09:02.222772+0000","last_fullsized":"2023-11-28T17:09:02.222772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:46:24.200692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513159+0000","last_change":"2023-11-28T17:09:00.998473+0000","last_active":"2023-11-28T17:09:02.513159+0000","last_peered":"2023-11-28T17:09:02.513159+0000","last_clean":"2023-11-28T17:09:02.513159+0000","last_became_active":"2023-11-28T17:09:00.998206+0000","last_became_peered":"2023-11-28T17:09:00.998206+0000","last_unstale":"2023-11-28T17:09:02.513159+0000","last_undegraded":"2023-11-28T17:09:02.513159+0000","last_fullsized":"2023-11-28T17:09:02.513159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:31.569896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512712+0000","last_change":"2023-11-28T17:09:01.019053+0000","last_active":"2023-11-28T17:09:02.512712+0000","last_peered":"2023-11-28T17:09:02.512712+0000","last_clean":"2023-11-28T17:09:02.512712+0000","last_became_active":"2023-11-28T17:09:01.018689+0000","last_became_peered":"2023-11-28T17:09:01.018689+0000","last_unstale":"2023-11-28T17:09:02.512712+0000","last_undegraded":"2023-11-28T17:09:02.512712+0000","last_fullsized":"2023-11-28T17:09:02.512712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:06:59.199376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513060+0000","last_change":"2023-11-28T17:09:01.000056+0000","last_active":"2023-11-28T17:09:02.513060+0000","last_peered":"2023-11-28T17:09:02.513060+0000","last_clean":"2023-11-28T17:09:02.513060+0000","last_became_active":"2023-11-28T17:09:00.999547+0000","last_became_peered":"2023-11-28T17:09:00.999547+0000","last_unstale":"2023-11-28T17:09:02.513060+0000","last_undegraded":"2023-11-28T17:09:02.513060+0000","last_fullsized":"2023-11-28T17:09:02.513060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:12.314413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.511764+0000","last_change":"2023-11-28T17:09:01.012699+0000","last_active":"2023-11-28T17:09:02.511764+0000","last_peered":"2023-11-28T17:09:02.511764+0000","last_clean":"2023-11-28T17:09:02.511764+0000","last_became_active":"2023-11-28T17:09:01.012442+0000","last_became_peered":"2023-11-28T17:09:01.012442+0000","last_unstale":"2023-11-28T17:09:02.511764+0000","last_undegraded":"2023-11-28T17:09:02.511764+0000","last_fullsized":"2023-11-28T17:09:02.511764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:46:24.200692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223215+0000","last_change":"2023-11-28T17:09:01.016792+0000","last_active":"2023-11-28T17:09:02.223215+0000","last_peered":"2023-11-28T17:09:02.223215+0000","last_clean":"2023-11-28T17:09:02.223215+0000","last_became_active":"2023-11-28T17:09:01.016034+0000","last_became_peered":"2023-11-28T17:09:01.016034+0000","last_unstale":"2023-11-28T17:09:02.223215+0000","last_undegraded":"2023-11-28T17:09:02.223215+0000","last_fullsized":"2023-11-28T17:09:02.223215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:52.224230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515192+0000","last_change":"2023-11-28T17:09:01.007271+0000","last_active":"2023-11-28T17:09:02.515192+0000","last_peered":"2023-11-28T17:09:02.515192+0000","last_clean":"2023-11-28T17:09:02.515192+0000","last_became_active":"2023-11-28T17:09:01.006929+0000","last_became_peered":"2023-11-28T17:09:01.006929+0000","last_unstale":"2023-11-28T17:09:02.515192+0000","last_undegraded":"2023-11-28T17:09:02.515192+0000","last_fullsized":"2023-11-28T17:09:02.515192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:20.109182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515425+0000","last_change":"2023-11-28T17:09:01.012176+0000","last_active":"2023-11-28T17:09:02.515425+0000","last_peered":"2023-11-28T17:09:02.515425+0000","last_clean":"2023-11-28T17:09:02.515425+0000","last_became_active":"2023-11-28T17:09:01.011821+0000","last_became_peered":"2023-11-28T17:09:01.011821+0000","last_unstale":"2023-11-28T17:09:02.515425+0000","last_undegraded":"2023-11-28T17:09:02.515425+0000","last_fullsized":"2023-11-28T17:09:02.515425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:55.626548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513365+0000","last_change":"2023-11-28T17:09:01.008348+0000","last_active":"2023-11-28T17:09:02.513365+0000","last_peered":"2023-11-28T17:09:02.513365+0000","last_clean":"2023-11-28T17:09:02.513365+0000","last_became_active":"2023-11-28T17:09:01.008029+0000","last_became_peered":"2023-11-28T17:09:01.008029+0000","last_unstale":"2023-11-28T17:09:02.513365+0000","last_undegraded":"2023-11-28T17:09:02.513365+0000","last_fullsized":"2023-11-28T17:09:02.513365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:52.224230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512353+0000","last_change":"2023-11-28T17:09:01.013836+0000","last_active":"2023-11-28T17:09:02.512353+0000","last_peered":"2023-11-28T17:09:02.512353+0000","last_clean":"2023-11-28T17:09:02.512353+0000","last_became_active":"2023-11-28T17:09:01.011870+0000","last_became_peered":"2023-11-28T17:09:01.011870+0000","last_unstale":"2023-11-28T17:09:02.512353+0000","last_undegraded":"2023-11-28T17:09:02.512353+0000","last_fullsized":"2023-11-28T17:09:02.512353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:23:27.238395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988425+0000","last_change":"2023-11-28T17:09:01.007854+0000","last_active":"2023-11-28T17:09:01.988425+0000","last_peered":"2023-11-28T17:09:01.988425+0000","last_clean":"2023-11-28T17:09:01.988425+0000","last_became_active":"2023-11-28T17:09:01.007561+0000","last_became_peered":"2023-11-28T17:09:01.007561+0000","last_unstale":"2023-11-28T17:09:01.988425+0000","last_undegraded":"2023-11-28T17:09:01.988425+0000","last_fullsized":"2023-11-28T17:09:01.988425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:53.375742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986992+0000","last_change":"2023-11-28T17:09:01.006142+0000","last_active":"2023-11-28T17:09:01.986992+0000","last_peered":"2023-11-28T17:09:01.986992+0000","last_clean":"2023-11-28T17:09:01.986992+0000","last_became_active":"2023-11-28T17:09:01.005823+0000","last_became_peered":"2023-11-28T17:09:01.005823+0000","last_unstale":"2023-11-28T17:09:01.986992+0000","last_undegraded":"2023-11-28T17:09:01.986992+0000","last_fullsized":"2023-11-28T17:09:01.986992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:32.388198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222442+0000","last_change":"2023-11-28T17:09:01.016320+0000","last_active":"2023-11-28T17:09:02.222442+0000","last_peered":"2023-11-28T17:09:02.222442+0000","last_clean":"2023-11-28T17:09:02.222442+0000","last_became_active":"2023-11-28T17:09:01.016012+0000","last_became_peered":"2023-11-28T17:09:01.016012+0000","last_unstale":"2023-11-28T17:09:02.222442+0000","last_undegraded":"2023-11-28T17:09:02.222442+0000","last_fullsized":"2023-11-28T17:09:02.222442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:44.651188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986526+0000","last_change":"2023-11-28T17:09:00.996901+0000","last_active":"2023-11-28T17:09:01.986526+0000","last_peered":"2023-11-28T17:09:01.986526+0000","last_clean":"2023-11-28T17:09:01.986526+0000","last_became_active":"2023-11-28T17:09:00.996627+0000","last_became_peered":"2023-11-28T17:09:00.996627+0000","last_unstale":"2023-11-28T17:09:01.986526+0000","last_undegraded":"2023-11-28T17:09:01.986526+0000","last_fullsized":"2023-11-28T17:09:01.986526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:45.964344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988654+0000","last_change":"2023-11-28T17:09:01.015527+0000","last_active":"2023-11-28T17:09:01.988654+0000","last_peered":"2023-11-28T17:09:01.988654+0000","last_clean":"2023-11-28T17:09:01.988654+0000","last_became_active":"2023-11-28T17:09:01.015238+0000","last_became_peered":"2023-11-28T17:09:01.015238+0000","last_unstale":"2023-11-28T17:09:01.988654+0000","last_undegraded":"2023-11-28T17:09:01.988654+0000","last_fullsized":"2023-11-28T17:09:01.988654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:44:13.494591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512305+0000","last_change":"2023-11-28T17:09:01.015000+0000","last_active":"2023-11-28T17:09:02.512305+0000","last_peered":"2023-11-28T17:09:02.512305+0000","last_clean":"2023-11-28T17:09:02.512305+0000","last_became_active":"2023-11-28T17:09:01.014649+0000","last_became_peered":"2023-11-28T17:09:01.014649+0000","last_unstale":"2023-11-28T17:09:02.512305+0000","last_undegraded":"2023-11-28T17:09:02.512305+0000","last_fullsized":"2023-11-28T17:09:02.512305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:31.569896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988495+0000","last_change":"2023-11-28T17:09:00.997869+0000","last_active":"2023-11-28T17:09:01.988495+0000","last_peered":"2023-11-28T17:09:01.988495+0000","last_clean":"2023-11-28T17:09:01.988495+0000","last_became_active":"2023-11-28T17:09:00.997614+0000","last_became_peered":"2023-11-28T17:09:00.997614+0000","last_unstale":"2023-11-28T17:09:01.988495+0000","last_undegraded":"2023-11-28T17:09:01.988495+0000","last_fullsized":"2023-11-28T17:09:01.988495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:25:26.955817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513938+0000","last_change":"2023-11-28T17:09:01.011584+0000","last_active":"2023-11-28T17:09:02.513938+0000","last_peered":"2023-11-28T17:09:02.513938+0000","last_clean":"2023-11-28T17:09:02.513938+0000","last_became_active":"2023-11-28T17:09:01.010359+0000","last_became_peered":"2023-11-28T17:09:01.010359+0000","last_unstale":"2023-11-28T17:09:02.513938+0000","last_undegraded":"2023-11-28T17:09:02.513938+0000","last_fullsized":"2023-11-28T17:09:02.513938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:45.964344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.989818+0000","last_change":"2023-11-28T17:09:00.999976+0000","last_active":"2023-11-28T17:09:01.989818+0000","last_peered":"2023-11-28T17:09:01.989818+0000","last_clean":"2023-11-28T17:09:01.989818+0000","last_became_active":"2023-11-28T17:09:00.999647+0000","last_became_peered":"2023-11-28T17:09:00.999647+0000","last_unstale":"2023-11-28T17:09:01.989818+0000","last_undegraded":"2023-11-28T17:09:01.989818+0000","last_fullsized":"2023-11-28T17:09:01.989818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:44.651188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.514610+0000","last_change":"2023-11-28T17:09:01.009510+0000","last_active":"2023-11-28T17:09:02.514610+0000","last_peered":"2023-11-28T17:09:02.514610+0000","last_clean":"2023-11-28T17:09:02.514610+0000","last_became_active":"2023-11-28T17:09:01.009184+0000","last_became_peered":"2023-11-28T17:09:01.009184+0000","last_unstale":"2023-11-28T17:09:02.514610+0000","last_undegraded":"2023-11-28T17:09:02.514610+0000","last_fullsized":"2023-11-28T17:09:02.514610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:12.314413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222577+0000","last_change":"2023-11-28T17:09:01.015602+0000","last_active":"2023-11-28T17:09:02.222577+0000","last_peered":"2023-11-28T17:09:02.222577+0000","last_clean":"2023-11-28T17:09:02.222577+0000","last_became_active":"2023-11-28T17:09:01.012170+0000","last_became_peered":"2023-11-28T17:09:01.012170+0000","last_unstale":"2023-11-28T17:09:02.222577+0000","last_undegraded":"2023-11-28T17:09:02.222577+0000","last_fullsized":"2023-11-28T17:09:02.222577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:07.593383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986512+0000","last_change":"2023-11-28T17:09:01.016062+0000","last_active":"2023-11-28T17:09:01.986512+0000","last_peered":"2023-11-28T17:09:01.986512+0000","last_clean":"2023-11-28T17:09:01.986512+0000","last_became_active":"2023-11-28T17:09:01.015802+0000","last_became_peered":"2023-11-28T17:09:01.015802+0000","last_unstale":"2023-11-28T17:09:01.986512+0000","last_undegraded":"2023-11-28T17:09:01.986512+0000","last_fullsized":"2023-11-28T17:09:01.986512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:34.895213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223456+0000","last_change":"2023-11-28T17:09:01.016389+0000","last_active":"2023-11-28T17:09:02.223456+0000","last_peered":"2023-11-28T17:09:02.223456+0000","last_clean":"2023-11-28T17:09:02.223456+0000","last_became_active":"2023-11-28T17:09:01.011680+0000","last_became_peered":"2023-11-28T17:09:01.011680+0000","last_unstale":"2023-11-28T17:09:02.223456+0000","last_undegraded":"2023-11-28T17:09:02.223456+0000","last_fullsized":"2023-11-28T17:09:02.223456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:22.787341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515917+0000","last_change":"2023-11-28T17:09:01.006281+0000","last_active":"2023-11-28T17:09:02.515917+0000","last_peered":"2023-11-28T17:09:02.515917+0000","last_clean":"2023-11-28T17:09:02.515917+0000","last_became_active":"2023-11-28T17:09:01.000480+0000","last_became_peered":"2023-11-28T17:09:01.000480+0000","last_unstale":"2023-11-28T17:09:02.515917+0000","last_undegraded":"2023-11-28T17:09:02.515917+0000","last_fullsized":"2023-11-28T17:09:02.515917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:25.324961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.989120+0000","last_change":"2023-11-28T17:09:01.002090+0000","last_active":"2023-11-28T17:09:01.989120+0000","last_peered":"2023-11-28T17:09:01.989120+0000","last_clean":"2023-11-28T17:09:01.989120+0000","last_became_active":"2023-11-28T17:09:01.001728+0000","last_became_peered":"2023-11-28T17:09:01.001728+0000","last_unstale":"2023-11-28T17:09:01.989120+0000","last_undegraded":"2023-11-28T17:09:01.989120+0000","last_fullsized":"2023-11-28T17:09:01.989120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:11.188332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988810+0000","last_change":"2023-11-28T17:09:01.013978+0000","last_active":"2023-11-28T17:09:01.988810+0000","last_peered":"2023-11-28T17:09:01.988810+0000","last_clean":"2023-11-28T17:09:01.988810+0000","last_became_active":"2023-11-28T17:09:01.013646+0000","last_became_peered":"2023-11-28T17:09:01.013646+0000","last_unstale":"2023-11-28T17:09:01.988810+0000","last_undegraded":"2023-11-28T17:09:01.988810+0000","last_fullsized":"2023-11-28T17:09:01.988810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:52:00.035340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222920+0000","last_change":"2023-11-28T17:09:01.013635+0000","last_active":"2023-11-28T17:09:02.222920+0000","last_peered":"2023-11-28T17:09:02.222920+0000","last_clean":"2023-11-28T17:09:02.222920+0000","last_became_active":"2023-11-28T17:09:01.013310+0000","last_became_peered":"2023-11-28T17:09:01.013310+0000","last_unstale":"2023-11-28T17:09:02.222920+0000","last_undegraded":"2023-11-28T17:09:02.222920+0000","last_fullsized":"2023-11-28T17:09:02.222920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:23:27.238395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987149+0000","last_change":"2023-11-28T17:09:01.016976+0000","last_active":"2023-11-28T17:09:01.987149+0000","last_peered":"2023-11-28T17:09:01.987149+0000","last_clean":"2023-11-28T17:09:01.987149+0000","last_became_active":"2023-11-28T17:09:01.016690+0000","last_became_peered":"2023-11-28T17:09:01.016690+0000","last_unstale":"2023-11-28T17:09:01.987149+0000","last_undegraded":"2023-11-28T17:09:01.987149+0000","last_fullsized":"2023-11-28T17:09:01.987149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:41:55.626548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.511885+0000","last_change":"2023-11-28T17:09:01.012474+0000","last_active":"2023-11-28T17:09:02.511885+0000","last_peered":"2023-11-28T17:09:02.511885+0000","last_clean":"2023-11-28T17:09:02.511885+0000","last_became_active":"2023-11-28T17:09:01.012226+0000","last_became_peered":"2023-11-28T17:09:01.012226+0000","last_unstale":"2023-11-28T17:09:02.511885+0000","last_undegraded":"2023-11-28T17:09:02.511885+0000","last_fullsized":"2023-11-28T17:09:02.511885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:36.438412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512524+0000","last_change":"2023-11-28T17:09:00.994555+0000","last_active":"2023-11-28T17:09:02.512524+0000","last_peered":"2023-11-28T17:09:02.512524+0000","last_clean":"2023-11-28T17:09:02.512524+0000","last_became_active":"2023-11-28T17:09:00.993870+0000","last_became_peered":"2023-11-28T17:09:00.993870+0000","last_unstale":"2023-11-28T17:09:02.512524+0000","last_undegraded":"2023-11-28T17:09:02.512524+0000","last_fullsized":"2023-11-28T17:09:02.512524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:36.438412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.514893+0000","last_change":"2023-11-28T17:09:01.018035+0000","last_active":"2023-11-28T17:09:02.514893+0000","last_peered":"2023-11-28T17:09:02.514893+0000","last_clean":"2023-11-28T17:09:02.514893+0000","last_became_active":"2023-11-28T17:09:01.017735+0000","last_became_peered":"2023-11-28T17:09:01.017735+0000","last_unstale":"2023-11-28T17:09:02.514893+0000","last_undegraded":"2023-11-28T17:09:02.514893+0000","last_fullsized":"2023-11-28T17:09:02.514893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:32.388198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.271905+0000","last_change":"2023-11-28T17:09:00.266501+0000","last_active":"2023-11-28T17:09:01.271905+0000","last_peered":"2023-11-28T17:09:01.271905+0000","last_clean":"2023-11-28T17:09:01.271905+0000","last_became_active":"2023-11-28T17:09:00.266177+0000","last_became_peered":"2023-11-28T17:09:00.266177+0000","last_unstale":"2023-11-28T17:09:01.271905+0000","last_undegraded":"2023-11-28T17:09:01.271905+0000","last_fullsized":"2023-11-28T17:09:01.271905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:08.376086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.990190+0000","last_change":"2023-11-28T17:09:01.008791+0000","last_active":"2023-11-28T17:09:01.990190+0000","last_peered":"2023-11-28T17:09:01.990190+0000","last_clean":"2023-11-28T17:09:01.990190+0000","last_became_active":"2023-11-28T17:09:01.008266+0000","last_became_peered":"2023-11-28T17:09:01.008266+0000","last_unstale":"2023-11-28T17:09:01.990190+0000","last_undegraded":"2023-11-28T17:09:01.990190+0000","last_fullsized":"2023-11-28T17:09:01.990190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:50.317730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.275936+0000","last_change":"2023-11-28T17:09:00.269760+0000","last_active":"2023-11-28T17:09:01.275936+0000","last_peered":"2023-11-28T17:09:01.275936+0000","last_clean":"2023-11-28T17:09:01.275936+0000","last_became_active":"2023-11-28T17:09:00.269082+0000","last_became_peered":"2023-11-28T17:09:00.269082+0000","last_unstale":"2023-11-28T17:09:01.275936+0000","last_undegraded":"2023-11-28T17:09:01.275936+0000","last_fullsized":"2023-11-28T17:09:01.275936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:14:51.213109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.986115+0000","last_change":"2023-11-28T17:09:00.262912+0000","last_active":"2023-11-28T17:09:00.986115+0000","last_peered":"2023-11-28T17:09:00.986115+0000","last_clean":"2023-11-28T17:09:00.986115+0000","last_became_active":"2023-11-28T17:09:00.260568+0000","last_became_peered":"2023-11-28T17:09:00.260568+0000","last_unstale":"2023-11-28T17:09:00.986115+0000","last_undegraded":"2023-11-28T17:09:00.986115+0000","last_fullsized":"2023-11-28T17:09:00.986115+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:30.558775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223014+0000","last_change":"2023-11-28T17:09:01.012823+0000","last_active":"2023-11-28T17:09:02.223014+0000","last_peered":"2023-11-28T17:09:02.223014+0000","last_clean":"2023-11-28T17:09:02.223014+0000","last_became_active":"2023-11-28T17:09:01.012514+0000","last_became_peered":"2023-11-28T17:09:01.012514+0000","last_unstale":"2023-11-28T17:09:02.223014+0000","last_undegraded":"2023-11-28T17:09:02.223014+0000","last_fullsized":"2023-11-28T17:09:02.223014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:24.354633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.985898+0000","last_change":"2023-11-28T17:09:00.262108+0000","last_active":"2023-11-28T17:09:00.985898+0000","last_peered":"2023-11-28T17:09:00.985898+0000","last_clean":"2023-11-28T17:09:00.985898+0000","last_became_active":"2023-11-28T17:09:00.261621+0000","last_became_peered":"2023-11-28T17:09:00.261621+0000","last_unstale":"2023-11-28T17:09:00.985898+0000","last_undegraded":"2023-11-28T17:09:00.985898+0000","last_fullsized":"2023-11-28T17:09:00.985898+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:14:52.843064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988589+0000","last_change":"2023-11-28T17:09:01.011726+0000","last_active":"2023-11-28T17:09:01.988589+0000","last_peered":"2023-11-28T17:09:01.988589+0000","last_clean":"2023-11-28T17:09:01.988589+0000","last_became_active":"2023-11-28T17:09:01.010904+0000","last_became_peered":"2023-11-28T17:09:01.010904+0000","last_unstale":"2023-11-28T17:09:01.988589+0000","last_undegraded":"2023-11-28T17:09:01.988589+0000","last_fullsized":"2023-11-28T17:09:01.988589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:14:53.854185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.990133+0000","last_change":"2023-11-28T17:09:00.262360+0000","last_active":"2023-11-28T17:09:00.990133+0000","last_peered":"2023-11-28T17:09:00.990133+0000","last_clean":"2023-11-28T17:09:00.990133+0000","last_became_active":"2023-11-28T17:09:00.261943+0000","last_became_peered":"2023-11-28T17:09:00.261943+0000","last_unstale":"2023-11-28T17:09:00.990133+0000","last_undegraded":"2023-11-28T17:09:00.990133+0000","last_fullsized":"2023-11-28T17:09:00.990133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:52.998404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515909+0000","last_change":"2023-11-28T17:09:01.007941+0000","last_active":"2023-11-28T17:09:02.515909+0000","last_peered":"2023-11-28T17:09:02.515909+0000","last_clean":"2023-11-28T17:09:02.515909+0000","last_became_active":"2023-11-28T17:09:01.002743+0000","last_became_peered":"2023-11-28T17:09:01.002743+0000","last_unstale":"2023-11-28T17:09:02.515909+0000","last_undegraded":"2023-11-28T17:09:02.515909+0000","last_fullsized":"2023-11-28T17:09:02.515909+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:20.109182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.997425+0000","last_change":"2023-11-28T17:09:00.259873+0000","last_active":"2023-11-28T17:09:00.997425+0000","last_peered":"2023-11-28T17:09:00.997425+0000","last_clean":"2023-11-28T17:09:00.997425+0000","last_became_active":"2023-11-28T17:09:00.259506+0000","last_became_peered":"2023-11-28T17:09:00.259506+0000","last_unstale":"2023-11-28T17:09:00.997425+0000","last_undegraded":"2023-11-28T17:09:00.997425+0000","last_fullsized":"2023-11-28T17:09:00.997425+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:19.098061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223644+0000","last_change":"2023-11-28T17:09:01.017611+0000","last_active":"2023-11-28T17:09:02.223644+0000","last_peered":"2023-11-28T17:09:02.223644+0000","last_clean":"2023-11-28T17:09:02.223644+0000","last_became_active":"2023-11-28T17:09:01.017269+0000","last_became_peered":"2023-11-28T17:09:01.017269+0000","last_unstale":"2023-11-28T17:09:02.223644+0000","last_undegraded":"2023-11-28T17:09:02.223644+0000","last_fullsized":"2023-11-28T17:09:02.223644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:45.964344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.272853+0000","last_change":"2023-11-28T17:09:00.264904+0000","last_active":"2023-11-28T17:09:01.272853+0000","last_peered":"2023-11-28T17:09:01.272853+0000","last_clean":"2023-11-28T17:09:01.272853+0000","last_became_active":"2023-11-28T17:09:00.264645+0000","last_became_peered":"2023-11-28T17:09:00.264645+0000","last_unstale":"2023-11-28T17:09:01.272853+0000","last_undegraded":"2023-11-28T17:09:01.272853+0000","last_fullsized":"2023-11-28T17:09:01.272853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:25:59.069890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987426+0000","last_change":"2023-11-28T17:09:01.005502+0000","last_active":"2023-11-28T17:09:01.987426+0000","last_peered":"2023-11-28T17:09:01.987426+0000","last_clean":"2023-11-28T17:09:01.987426+0000","last_became_active":"2023-11-28T17:09:01.004783+0000","last_became_peered":"2023-11-28T17:09:01.004783+0000","last_unstale":"2023-11-28T17:09:01.987426+0000","last_undegraded":"2023-11-28T17:09:01.987426+0000","last_fullsized":"2023-11-28T17:09:01.987426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:10:50.736315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276609+0000","last_change":"2023-11-28T17:09:00.263414+0000","last_active":"2023-11-28T17:09:01.276609+0000","last_peered":"2023-11-28T17:09:01.276609+0000","last_clean":"2023-11-28T17:09:01.276609+0000","last_became_active":"2023-11-28T17:09:00.260801+0000","last_became_peered":"2023-11-28T17:09:00.260801+0000","last_unstale":"2023-11-28T17:09:01.276609+0000","last_undegraded":"2023-11-28T17:09:01.276609+0000","last_fullsized":"2023-11-28T17:09:01.276609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:31:18.731288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513670+0000","last_change":"2023-11-28T17:09:01.019149+0000","last_active":"2023-11-28T17:09:02.513670+0000","last_peered":"2023-11-28T17:09:02.513670+0000","last_clean":"2023-11-28T17:09:02.513670+0000","last_became_active":"2023-11-28T17:09:01.018834+0000","last_became_peered":"2023-11-28T17:09:01.018834+0000","last_unstale":"2023-11-28T17:09:02.513670+0000","last_undegraded":"2023-11-28T17:09:02.513670+0000","last_fullsized":"2023-11-28T17:09:02.513670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:34.895213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.275750+0000","last_change":"2023-11-28T17:09:00.271771+0000","last_active":"2023-11-28T17:09:01.275750+0000","last_peered":"2023-11-28T17:09:01.275750+0000","last_clean":"2023-11-28T17:09:01.275750+0000","last_became_active":"2023-11-28T17:09:00.269930+0000","last_became_peered":"2023-11-28T17:09:00.269930+0000","last_unstale":"2023-11-28T17:09:01.275750+0000","last_undegraded":"2023-11-28T17:09:01.275750+0000","last_fullsized":"2023-11-28T17:09:01.275750+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:48:33.502573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987179+0000","last_change":"2023-11-28T17:09:00.999606+0000","last_active":"2023-11-28T17:09:01.987179+0000","last_peered":"2023-11-28T17:09:01.987179+0000","last_clean":"2023-11-28T17:09:01.987179+0000","last_became_active":"2023-11-28T17:09:00.999346+0000","last_became_peered":"2023-11-28T17:09:00.999346+0000","last_unstale":"2023-11-28T17:09:01.987179+0000","last_undegraded":"2023-11-28T17:09:01.987179+0000","last_fullsized":"2023-11-28T17:09:01.987179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:25.324961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.991340+0000","last_change":"2023-11-28T17:09:00.269952+0000","last_active":"2023-11-28T17:09:00.991340+0000","last_peered":"2023-11-28T17:09:00.991340+0000","last_clean":"2023-11-28T17:09:00.991340+0000","last_became_active":"2023-11-28T17:09:00.269539+0000","last_became_peered":"2023-11-28T17:09:00.269539+0000","last_unstale":"2023-11-28T17:09:00.991340+0000","last_undegraded":"2023-11-28T17:09:00.991340+0000","last_fullsized":"2023-11-28T17:09:00.991340+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:58.974841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987385+0000","last_change":"2023-11-28T17:09:01.009043+0000","last_active":"2023-11-28T17:09:01.987385+0000","last_peered":"2023-11-28T17:09:01.987385+0000","last_clean":"2023-11-28T17:09:01.987385+0000","last_became_active":"2023-11-28T17:09:01.008739+0000","last_became_peered":"2023-11-28T17:09:01.008739+0000","last_unstale":"2023-11-28T17:09:01.987385+0000","last_undegraded":"2023-11-28T17:09:01.987385+0000","last_fullsized":"2023-11-28T17:09:01.987385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:27:49.417143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.999207+0000","last_change":"2023-11-28T17:09:00.270474+0000","last_active":"2023-11-28T17:09:00.999207+0000","last_peered":"2023-11-28T17:09:00.999207+0000","last_clean":"2023-11-28T17:09:00.999207+0000","last_became_active":"2023-11-28T17:09:00.270277+0000","last_became_peered":"2023-11-28T17:09:00.270277+0000","last_unstale":"2023-11-28T17:09:00.999207+0000","last_undegraded":"2023-11-28T17:09:00.999207+0000","last_fullsized":"2023-11-28T17:09:00.999207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:52.364620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.511767+0000","last_change":"2023-11-28T17:09:01.016931+0000","last_active":"2023-11-28T17:09:02.511767+0000","last_peered":"2023-11-28T17:09:02.511767+0000","last_clean":"2023-11-28T17:09:02.511767+0000","last_became_active":"2023-11-28T17:09:01.016673+0000","last_became_peered":"2023-11-28T17:09:01.016673+0000","last_unstale":"2023-11-28T17:09:02.511767+0000","last_undegraded":"2023-11-28T17:09:02.511767+0000","last_fullsized":"2023-11-28T17:09:02.511767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:36.438412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.274020+0000","last_change":"2023-11-28T17:09:00.262584+0000","last_active":"2023-11-28T17:09:01.274020+0000","last_peered":"2023-11-28T17:09:01.274020+0000","last_clean":"2023-11-28T17:09:01.274020+0000","last_became_active":"2023-11-28T17:09:00.262259+0000","last_became_peered":"2023-11-28T17:09:00.262259+0000","last_unstale":"2023-11-28T17:09:01.274020+0000","last_undegraded":"2023-11-28T17:09:01.274020+0000","last_fullsized":"2023-11-28T17:09:01.274020+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:31.377077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.986947+0000","last_change":"2023-11-28T17:09:01.003188+0000","last_active":"2023-11-28T17:09:01.986947+0000","last_peered":"2023-11-28T17:09:01.986947+0000","last_clean":"2023-11-28T17:09:01.986947+0000","last_became_active":"2023-11-28T17:09:01.000643+0000","last_became_peered":"2023-11-28T17:09:01.000643+0000","last_unstale":"2023-11-28T17:09:01.986947+0000","last_undegraded":"2023-11-28T17:09:01.986947+0000","last_fullsized":"2023-11-28T17:09:01.986947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:49:27.242284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273771+0000","last_change":"2023-11-28T17:09:00.272398+0000","last_active":"2023-11-28T17:09:01.273771+0000","last_peered":"2023-11-28T17:09:01.273771+0000","last_clean":"2023-11-28T17:09:01.273771+0000","last_became_active":"2023-11-28T17:09:00.271617+0000","last_became_peered":"2023-11-28T17:09:00.271617+0000","last_unstale":"2023-11-28T17:09:01.273771+0000","last_undegraded":"2023-11-28T17:09:01.273771+0000","last_fullsized":"2023-11-28T17:09:01.273771+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:10.177210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.516054+0000","last_change":"2023-11-28T17:09:01.012920+0000","last_active":"2023-11-28T17:09:02.516054+0000","last_peered":"2023-11-28T17:09:02.516054+0000","last_clean":"2023-11-28T17:09:02.516054+0000","last_became_active":"2023-11-28T17:09:01.012651+0000","last_became_peered":"2023-11-28T17:09:01.012651+0000","last_unstale":"2023-11-28T17:09:02.516054+0000","last_undegraded":"2023-11-28T17:09:02.516054+0000","last_fullsized":"2023-11-28T17:09:02.516054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:53.375742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273900+0000","last_change":"2023-11-28T17:09:00.271952+0000","last_active":"2023-11-28T17:09:01.273900+0000","last_peered":"2023-11-28T17:09:01.273900+0000","last_clean":"2023-11-28T17:09:01.273900+0000","last_became_active":"2023-11-28T17:09:00.271641+0000","last_became_peered":"2023-11-28T17:09:00.271641+0000","last_unstale":"2023-11-28T17:09:01.273900+0000","last_undegraded":"2023-11-28T17:09:01.273900+0000","last_fullsized":"2023-11-28T17:09:01.273900+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:52.364620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513991+0000","last_change":"2023-11-28T17:09:01.014412+0000","last_active":"2023-11-28T17:09:02.513991+0000","last_peered":"2023-11-28T17:09:02.513991+0000","last_clean":"2023-11-28T17:09:02.513991+0000","last_became_active":"2023-11-28T17:09:01.014145+0000","last_became_peered":"2023-11-28T17:09:01.014145+0000","last_unstale":"2023-11-28T17:09:02.513991+0000","last_undegraded":"2023-11-28T17:09:02.513991+0000","last_fullsized":"2023-11-28T17:09:02.513991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:49:27.242284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273346+0000","last_change":"2023-11-28T17:09:00.264424+0000","last_active":"2023-11-28T17:09:01.273346+0000","last_peered":"2023-11-28T17:09:01.273346+0000","last_clean":"2023-11-28T17:09:01.273346+0000","last_became_active":"2023-11-28T17:09:00.263195+0000","last_became_peered":"2023-11-28T17:09:00.263195+0000","last_unstale":"2023-11-28T17:09:01.273346+0000","last_undegraded":"2023-11-28T17:09:01.273346+0000","last_fullsized":"2023-11-28T17:09:01.273346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:35.427291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.513525+0000","last_change":"2023-11-28T17:09:01.019072+0000","last_active":"2023-11-28T17:09:02.513525+0000","last_peered":"2023-11-28T17:09:02.513525+0000","last_clean":"2023-11-28T17:09:02.513525+0000","last_became_active":"2023-11-28T17:09:01.018785+0000","last_became_peered":"2023-11-28T17:09:01.018785+0000","last_unstale":"2023-11-28T17:09:02.513525+0000","last_undegraded":"2023-11-28T17:09:02.513525+0000","last_fullsized":"2023-11-28T17:09:02.513525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:59.985963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.984805+0000","last_change":"2023-11-28T17:09:00.268057+0000","last_active":"2023-11-28T17:09:00.984805+0000","last_peered":"2023-11-28T17:09:00.984805+0000","last_clean":"2023-11-28T17:09:00.984805+0000","last_became_active":"2023-11-28T17:09:00.264298+0000","last_became_peered":"2023-11-28T17:09:00.264298+0000","last_unstale":"2023-11-28T17:09:00.984805+0000","last_undegraded":"2023-11-28T17:09:00.984805+0000","last_fullsized":"2023-11-28T17:09:00.984805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:35.427291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988219+0000","last_change":"2023-11-28T17:09:01.009062+0000","last_active":"2023-11-28T17:09:01.988219+0000","last_peered":"2023-11-28T17:09:01.988219+0000","last_clean":"2023-11-28T17:09:01.988219+0000","last_became_active":"2023-11-28T17:09:01.008759+0000","last_became_peered":"2023-11-28T17:09:01.008759+0000","last_unstale":"2023-11-28T17:09:01.988219+0000","last_undegraded":"2023-11-28T17:09:01.988219+0000","last_fullsized":"2023-11-28T17:09:01.988219+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:09:50.534329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.986684+0000","last_change":"2023-11-28T17:08:51.392782+0000","last_active":"2023-11-28T17:09:00.986684+0000","last_peered":"2023-11-28T17:09:00.986684+0000","last_clean":"2023-11-28T17:09:00.986684+0000","last_became_active":"2023-11-28T17:08:51.390415+0000","last_became_peered":"2023-11-28T17:08:51.390415+0000","last_unstale":"2023-11-28T17:09:00.986684+0000","last_undegraded":"2023-11-28T17:09:00.986684+0000","last_fullsized":"2023-11-28T17:09:00.986684+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_clean_scrub_stamp":"2023-11-28T17:08:49.967682+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:03:32.638685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.985042+0000","last_change":"2023-11-28T17:09:00.267795+0000","last_active":"2023-11-28T17:09:00.985042+0000","last_peered":"2023-11-28T17:09:00.985042+0000","last_clean":"2023-11-28T17:09:00.985042+0000","last_became_active":"2023-11-28T17:09:00.264764+0000","last_became_peered":"2023-11-28T17:09:00.264764+0000","last_unstale":"2023-11-28T17:09:00.985042+0000","last_undegraded":"2023-11-28T17:09:00.985042+0000","last_fullsized":"2023-11-28T17:09:00.985042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:59.024219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988457+0000","last_change":"2023-11-28T17:09:01.004564+0000","last_active":"2023-11-28T17:09:01.988457+0000","last_peered":"2023-11-28T17:09:01.988457+0000","last_clean":"2023-11-28T17:09:01.988457+0000","last_became_active":"2023-11-28T17:09:01.004189+0000","last_became_peered":"2023-11-28T17:09:01.004189+0000","last_unstale":"2023-11-28T17:09:01.988457+0000","last_undegraded":"2023-11-28T17:09:01.988457+0000","last_fullsized":"2023-11-28T17:09:01.988457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:09.387207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.986785+0000","last_change":"2023-11-28T17:09:00.269480+0000","last_active":"2023-11-28T17:09:00.986785+0000","last_peered":"2023-11-28T17:09:00.986785+0000","last_clean":"2023-11-28T17:09:00.986785+0000","last_became_active":"2023-11-28T17:09:00.269032+0000","last_became_peered":"2023-11-28T17:09:00.269032+0000","last_unstale":"2023-11-28T17:09:00.986785+0000","last_undegraded":"2023-11-28T17:09:00.986785+0000","last_fullsized":"2023-11-28T17:09:00.986785+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:49.306608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987055+0000","last_change":"2023-11-28T17:09:01.007809+0000","last_active":"2023-11-28T17:09:01.987055+0000","last_peered":"2023-11-28T17:09:01.987055+0000","last_clean":"2023-11-28T17:09:01.987055+0000","last_became_active":"2023-11-28T17:09:01.007508+0000","last_became_peered":"2023-11-28T17:09:01.007508+0000","last_unstale":"2023-11-28T17:09:01.987055+0000","last_undegraded":"2023-11-28T17:09:01.987055+0000","last_fullsized":"2023-11-28T17:09:01.987055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:20.109182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.001078+0000","last_change":"2023-11-28T17:09:00.271320+0000","last_active":"2023-11-28T17:09:01.001078+0000","last_peered":"2023-11-28T17:09:01.001078+0000","last_clean":"2023-11-28T17:09:01.001078+0000","last_became_active":"2023-11-28T17:09:00.271019+0000","last_became_peered":"2023-11-28T17:09:00.271019+0000","last_unstale":"2023-11-28T17:09:01.001078+0000","last_undegraded":"2023-11-28T17:09:01.001078+0000","last_fullsized":"2023-11-28T17:09:01.001078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:41:19.939145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.222456+0000","last_change":"2023-11-28T17:09:01.009687+0000","last_active":"2023-11-28T17:09:02.222456+0000","last_peered":"2023-11-28T17:09:02.222456+0000","last_clean":"2023-11-28T17:09:02.222456+0000","last_became_active":"2023-11-28T17:09:01.009329+0000","last_became_peered":"2023-11-28T17:09:01.009329+0000","last_unstale":"2023-11-28T17:09:02.222456+0000","last_undegraded":"2023-11-28T17:09:02.222456+0000","last_fullsized":"2023-11-28T17:09:02.222456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:50.317730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.991649+0000","last_change":"2023-11-28T17:09:00.269333+0000","last_active":"2023-11-28T17:09:00.991649+0000","last_peered":"2023-11-28T17:09:00.991649+0000","last_clean":"2023-11-28T17:09:00.991649+0000","last_became_active":"2023-11-28T17:09:00.267481+0000","last_became_peered":"2023-11-28T17:09:00.267481+0000","last_unstale":"2023-11-28T17:09:00.991649+0000","last_undegraded":"2023-11-28T17:09:00.991649+0000","last_fullsized":"2023-11-28T17:09:00.991649+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:21.776220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512505+0000","last_change":"2023-11-28T17:09:01.012458+0000","last_active":"2023-11-28T17:09:02.512505+0000","last_peered":"2023-11-28T17:09:02.512505+0000","last_clean":"2023-11-28T17:09:02.512505+0000","last_became_active":"2023-11-28T17:09:01.011811+0000","last_became_peered":"2023-11-28T17:09:01.011811+0000","last_unstale":"2023-11-28T17:09:02.512505+0000","last_undegraded":"2023-11-28T17:09:02.512505+0000","last_fullsized":"2023-11-28T17:09:02.512505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:24.354633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.999956+0000","last_change":"2023-11-28T17:09:00.263517+0000","last_active":"2023-11-28T17:09:00.999956+0000","last_peered":"2023-11-28T17:09:00.999956+0000","last_clean":"2023-11-28T17:09:00.999956+0000","last_became_active":"2023-11-28T17:09:00.263243+0000","last_became_peered":"2023-11-28T17:09:00.263243+0000","last_unstale":"2023-11-28T17:09:00.999956+0000","last_undegraded":"2023-11-28T17:09:00.999956+0000","last_fullsized":"2023-11-28T17:09:00.999956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:10.177210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.515382+0000","last_change":"2023-11-28T17:09:00.997629+0000","last_active":"2023-11-28T17:09:02.515382+0000","last_peered":"2023-11-28T17:09:02.515382+0000","last_clean":"2023-11-28T17:09:02.515382+0000","last_became_active":"2023-11-28T17:09:00.997360+0000","last_became_peered":"2023-11-28T17:09:00.997360+0000","last_unstale":"2023-11-28T17:09:02.515382+0000","last_undegraded":"2023-11-28T17:09:02.515382+0000","last_fullsized":"2023-11-28T17:09:02.515382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:54:32.388198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.988631+0000","last_change":"2023-11-28T17:09:00.268166+0000","last_active":"2023-11-28T17:09:00.988631+0000","last_peered":"2023-11-28T17:09:00.988631+0000","last_clean":"2023-11-28T17:09:00.988631+0000","last_became_active":"2023-11-28T17:09:00.267609+0000","last_became_peered":"2023-11-28T17:09:00.267609+0000","last_unstale":"2023-11-28T17:09:00.988631+0000","last_undegraded":"2023-11-28T17:09:00.988631+0000","last_fullsized":"2023-11-28T17:09:00.988631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:23:26.227274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.989182+0000","last_change":"2023-11-28T17:09:01.017987+0000","last_active":"2023-11-28T17:09:01.989182+0000","last_peered":"2023-11-28T17:09:01.989182+0000","last_clean":"2023-11-28T17:09:01.989182+0000","last_became_active":"2023-11-28T17:09:01.017491+0000","last_became_peered":"2023-11-28T17:09:01.017491+0000","last_unstale":"2023-11-28T17:09:01.989182+0000","last_undegraded":"2023-11-28T17:09:01.989182+0000","last_fullsized":"2023-11-28T17:09:01.989182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:09.387207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.990667+0000","last_change":"2023-11-28T17:09:00.267582+0000","last_active":"2023-11-28T17:09:01.990667+0000","last_peered":"2023-11-28T17:09:01.990667+0000","last_clean":"2023-11-28T17:09:01.990667+0000","last_became_active":"2023-11-28T17:09:00.267245+0000","last_became_peered":"2023-11-28T17:09:00.267245+0000","last_unstale":"2023-11-28T17:09:01.990667+0000","last_undegraded":"2023-11-28T17:09:01.990667+0000","last_fullsized":"2023-11-28T17:09:01.990667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:51:59.024219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.990398+0000","last_change":"2023-11-28T17:09:00.997521+0000","last_active":"2023-11-28T17:09:01.990398+0000","last_peered":"2023-11-28T17:09:01.990398+0000","last_clean":"2023-11-28T17:09:01.990398+0000","last_became_active":"2023-11-28T17:09:00.997041+0000","last_became_peered":"2023-11-28T17:09:00.997041+0000","last_unstale":"2023-11-28T17:09:01.990398+0000","last_undegraded":"2023-11-28T17:09:01.990398+0000","last_fullsized":"2023-11-28T17:09:01.990398+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:07.593383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276430+0000","last_change":"2023-11-28T17:09:00.264837+0000","last_active":"2023-11-28T17:09:01.276430+0000","last_peered":"2023-11-28T17:09:01.276430+0000","last_clean":"2023-11-28T17:09:01.276430+0000","last_became_active":"2023-11-28T17:09:00.264570+0000","last_became_peered":"2023-11-28T17:09:00.264570+0000","last_unstale":"2023-11-28T17:09:01.276430+0000","last_undegraded":"2023-11-28T17:09:01.276430+0000","last_fullsized":"2023-11-28T17:09:01.276430+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:23.343512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.990475+0000","last_change":"2023-11-28T17:09:01.006414+0000","last_active":"2023-11-28T17:09:01.990475+0000","last_peered":"2023-11-28T17:09:01.990475+0000","last_clean":"2023-11-28T17:09:01.990475+0000","last_became_active":"2023-11-28T17:09:01.005570+0000","last_became_peered":"2023-11-28T17:09:01.005570+0000","last_unstale":"2023-11-28T17:09:01.990475+0000","last_undegraded":"2023-11-28T17:09:01.990475+0000","last_fullsized":"2023-11-28T17:09:01.990475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:46:24.200692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.998379+0000","last_change":"2023-11-28T17:09:00.257189+0000","last_active":"2023-11-28T17:09:00.998379+0000","last_peered":"2023-11-28T17:09:00.998379+0000","last_clean":"2023-11-28T17:09:00.998379+0000","last_became_active":"2023-11-28T17:09:00.256862+0000","last_became_peered":"2023-11-28T17:09:00.256862+0000","last_unstale":"2023-11-28T17:09:00.998379+0000","last_undegraded":"2023-11-28T17:09:00.998379+0000","last_fullsized":"2023-11-28T17:09:00.998379+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:24.313840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987313+0000","last_change":"2023-11-28T17:09:01.006232+0000","last_active":"2023-11-28T17:09:01.987313+0000","last_peered":"2023-11-28T17:09:01.987313+0000","last_clean":"2023-11-28T17:09:01.987313+0000","last_became_active":"2023-11-28T17:09:01.005930+0000","last_became_peered":"2023-11-28T17:09:01.005930+0000","last_unstale":"2023-11-28T17:09:01.987313+0000","last_undegraded":"2023-11-28T17:09:01.987313+0000","last_fullsized":"2023-11-28T17:09:01.987313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:06:27.104365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.991766+0000","last_change":"2023-11-28T17:09:00.266555+0000","last_active":"2023-11-28T17:09:00.991766+0000","last_peered":"2023-11-28T17:09:00.991766+0000","last_clean":"2023-11-28T17:09:00.991766+0000","last_became_active":"2023-11-28T17:09:00.266279+0000","last_became_peered":"2023-11-28T17:09:00.266279+0000","last_unstale":"2023-11-28T17:09:00.991766+0000","last_undegraded":"2023-11-28T17:09:00.991766+0000","last_fullsized":"2023-11-28T17:09:00.991766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:34:19.098061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988255+0000","last_change":"2023-11-28T17:09:01.007494+0000","last_active":"2023-11-28T17:09:01.988255+0000","last_peered":"2023-11-28T17:09:01.988255+0000","last_clean":"2023-11-28T17:09:01.988255+0000","last_became_active":"2023-11-28T17:09:01.007121+0000","last_became_peered":"2023-11-28T17:09:01.007121+0000","last_unstale":"2023-11-28T17:09:01.988255+0000","last_undegraded":"2023-11-28T17:09:01.988255+0000","last_fullsized":"2023-11-28T17:09:01.988255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:03:11.188332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:00.984942+0000","last_change":"2023-11-28T17:09:00.267476+0000","last_active":"2023-11-28T17:09:00.984942+0000","last_peered":"2023-11-28T17:09:00.984942+0000","last_clean":"2023-11-28T17:09:00.984942+0000","last_became_active":"2023-11-28T17:09:00.267138+0000","last_became_peered":"2023-11-28T17:09:00.267138+0000","last_unstale":"2023-11-28T17:09:00.984942+0000","last_undegraded":"2023-11-28T17:09:00.984942+0000","last_fullsized":"2023-11-28T17:09:00.984942+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:11.303292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.512236+0000","last_change":"2023-11-28T17:09:01.008773+0000","last_active":"2023-11-28T17:09:02.512236+0000","last_peered":"2023-11-28T17:09:02.512236+0000","last_clean":"2023-11-28T17:09:02.512236+0000","last_became_active":"2023-11-28T17:09:01.008241+0000","last_became_peered":"2023-11-28T17:09:01.008241+0000","last_unstale":"2023-11-28T17:09:02.512236+0000","last_undegraded":"2023-11-28T17:09:02.512236+0000","last_fullsized":"2023-11-28T17:09:02.512236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:07:12.314413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276631+0000","last_change":"2023-11-28T17:09:00.267671+0000","last_active":"2023-11-28T17:09:01.276631+0000","last_peered":"2023-11-28T17:09:01.276631+0000","last_clean":"2023-11-28T17:09:01.276631+0000","last_became_active":"2023-11-28T17:09:00.264445+0000","last_became_peered":"2023-11-28T17:09:00.264445+0000","last_unstale":"2023-11-28T17:09:01.276631+0000","last_undegraded":"2023-11-28T17:09:01.276631+0000","last_fullsized":"2023-11-28T17:09:01.276631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:09:52.364620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.987244+0000","last_change":"2023-11-28T17:09:01.003981+0000","last_active":"2023-11-28T17:09:01.987244+0000","last_peered":"2023-11-28T17:09:01.987244+0000","last_clean":"2023-11-28T17:09:01.987244+0000","last_became_active":"2023-11-28T17:09:01.003508+0000","last_became_peered":"2023-11-28T17:09:01.003508+0000","last_unstale":"2023-11-28T17:09:01.987244+0000","last_undegraded":"2023-11-28T17:09:01.987244+0000","last_fullsized":"2023-11-28T17:09:01.987244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:48:31.212857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.988121+0000","last_change":"2023-11-28T17:09:01.012859+0000","last_active":"2023-11-28T17:09:01.988121+0000","last_peered":"2023-11-28T17:09:01.988121+0000","last_clean":"2023-11-28T17:09:01.988121+0000","last_became_active":"2023-11-28T17:09:01.010930+0000","last_became_peered":"2023-11-28T17:09:01.010930+0000","last_unstale":"2023-11-28T17:09:01.988121+0000","last_undegraded":"2023-11-28T17:09:01.988121+0000","last_fullsized":"2023-11-28T17:09:01.988121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:46:01.399129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.268661+0000","last_change":"2023-11-28T17:09:00.259200+0000","last_active":"2023-11-28T17:09:01.268661+0000","last_peered":"2023-11-28T17:09:01.268661+0000","last_clean":"2023-11-28T17:09:01.268661+0000","last_became_active":"2023-11-28T17:09:00.258886+0000","last_became_peered":"2023-11-28T17:09:00.258886+0000","last_unstale":"2023-11-28T17:09:01.268661+0000","last_undegraded":"2023-11-28T17:09:01.268661+0000","last_fullsized":"2023-11-28T17:09:01.268661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:43.640066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.514743+0000","last_change":"2023-11-28T17:09:01.015028+0000","last_active":"2023-11-28T17:09:02.514743+0000","last_peered":"2023-11-28T17:09:02.514743+0000","last_clean":"2023-11-28T17:09:02.514743+0000","last_became_active":"2023-11-28T17:09:01.014670+0000","last_became_peered":"2023-11-28T17:09:01.014670+0000","last_unstale":"2023-11-28T17:09:02.514743+0000","last_undegraded":"2023-11-28T17:09:02.514743+0000","last_fullsized":"2023-11-28T17:09:02.514743+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:31:31.569896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.268602+0000","last_change":"2023-11-28T17:09:00.263220+0000","last_active":"2023-11-28T17:09:01.268602+0000","last_peered":"2023-11-28T17:09:01.268602+0000","last_clean":"2023-11-28T17:09:01.268602+0000","last_became_active":"2023-11-28T17:09:00.262930+0000","last_became_peered":"2023-11-28T17:09:00.262930+0000","last_unstale":"2023-11-28T17:09:01.268602+0000","last_undegraded":"2023-11-28T17:09:01.268602+0000","last_fullsized":"2023-11-28T17:09:01.268602+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:06.582262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223461+0000","last_change":"2023-11-28T17:09:01.014772+0000","last_active":"2023-11-28T17:09:02.223461+0000","last_peered":"2023-11-28T17:09:02.223461+0000","last_clean":"2023-11-28T17:09:02.223461+0000","last_became_active":"2023-11-28T17:09:01.013249+0000","last_became_peered":"2023-11-28T17:09:01.013249+0000","last_unstale":"2023-11-28T17:09:02.223461+0000","last_undegraded":"2023-11-28T17:09:02.223461+0000","last_fullsized":"2023-11-28T17:09:02.223461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:09:34.895213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.273335+0000","last_change":"2023-11-28T17:09:00.270822+0000","last_active":"2023-11-28T17:09:01.273335+0000","last_peered":"2023-11-28T17:09:01.273335+0000","last_clean":"2023-11-28T17:09:01.273335+0000","last_became_active":"2023-11-28T17:09:00.268995+0000","last_became_peered":"2023-11-28T17:09:00.268995+0000","last_unstale":"2023-11-28T17:09:01.273335+0000","last_undegraded":"2023-11-28T17:09:01.273335+0000","last_fullsized":"2023-11-28T17:09:01.273335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:24.313840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-28T17:09:02.223456+0000","last_change":"2023-11-28T17:09:01.012994+0000","last_active":"2023-11-28T17:09:02.223456+0000","last_peered":"2023-11-28T17:09:02.223456+0000","last_clean":"2023-11-28T17:09:02.223456+0000","last_became_active":"2023-11-28T17:09:01.012496+0000","last_became_peered":"2023-11-28T17:09:01.012496+0000","last_unstale":"2023-11-28T17:09:02.223456+0000","last_undegraded":"2023-11-28T17:09:02.223456+0000","last_fullsized":"2023-11-28T17:09:02.223456+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:09:00.218931+0000","last_clean_scrub_stamp":"2023-11-28T17:09:00.218931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:31:59.985963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-28T17:09:01.276064+0000","last_change":"2023-11-28T17:09:00.265200+0000","last_active":"2023-11-28T17:09:01.276064+0000","last_peered":"2023-11-28T17:09:01.276064+0000","last_clean":"2023-11-28T17:09:01.276064+0000","last_became_active":"2023-11-28T17:09:00.264913+0000","last_became_peered":"2023-11-28T17:09:00.264913+0000","last_unstale":"2023-11-28T17:09:01.276064+0000","last_undegraded":"2023-11-28T17:09:01.276064+0000","last_fullsized":"2023-11-28T17:09:01.276064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:59.207809+0000","last_clean_scrub_stamp":"2023-11-28T17:08:59.207809+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:37:24.313840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7270000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4510000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.409}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.635}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3579999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7809999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6140000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":806912,"data_stored":661370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2569999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7130000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.538}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":73194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.056}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.875}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.944}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2120000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2650000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5379999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3069999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6280000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2879999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3570000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3440000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3740000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5620000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7170000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.788}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3260000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6949999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5720000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.169}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":70966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.456}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2989999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.516}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.421}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T17:09:11.751 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-28T17:09:11.751 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-28T17:09:11.752 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-28T17:09:11.752 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-28T17:09:11.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.981+0000 7f4a62d68640 1 Processor -- start 2023-11-28T17:09:11.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.982+0000 7f4a62d68640 1 -- start start 2023-11-28T17:09:11.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.983+0000 7f4a62d68640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a5c144430 0x7f4a5c15f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.983+0000 7f4a62d68640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4a5c06c250 0x7f4a5c06c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.983+0000 7f4a62d68640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c070bd0 con 0x7f4a5c0703f0 2023-11-28T17:09:11.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.983+0000 7f4a62d68640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a5c144810 con 0x7f4a5c06c250 2023-11-28T17:09:11.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.983+0000 7f4a62d68640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c06cc20 con 0x7f4a5c144430 2023-11-28T17:09:11.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.984+0000 7f4a5bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4a5c0703f0 0x7f4a5c071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54700/0 (socket says 172.21.15.116:54700) 2023-11-28T17:09:11.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.984+0000 7f4a5bfff640 1 -- 172.21.15.116:0/3597700918 learned_addr learned my addr 172.21.15.116:0/3597700918 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.984+0000 7f4a60b04640 1 --2- 172.21.15.116:0/3597700918 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a5c144430 0x7f4a5c15f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.984+0000 7f4a61305640 1 --2- 172.21.15.116:0/3597700918 >> v2:172.21.15.156:3300/0 conn(0x7f4a5c06c250 0x7f4a5c06c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.984+0000 7f4a5b7fe640 1 -- 172.21.15.116:0/3597700918 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135900760 0 0) 0x7f4a5c070bd0 con 0x7f4a5c0703f0 2023-11-28T17:09:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.984+0000 7f4a5b7fe640 1 -- 172.21.15.116:0/3597700918 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a4c003520 con 0x7f4a5c0703f0 2023-11-28T17:09:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.984+0000 7f4a5b7fe640 1 -- 172.21.15.116:0/3597700918 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3555351854 0 0) 0x7f4a4c003520 con 0x7f4a5c0703f0 2023-11-28T17:09:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.984+0000 7f4a5b7fe640 1 -- 172.21.15.116:0/3597700918 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a5c144430 msgr2=0x7f4a5c15f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.984+0000 7f4a5b7fe640 1 --2- 172.21.15.116:0/3597700918 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a5c144430 0x7f4a5c15f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.984+0000 7f4a5b7fe640 1 -- 172.21.15.116:0/3597700918 >> v2:172.21.15.156:3300/0 conn(0x7f4a5c06c250 msgr2=0x7f4a5c06c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.984+0000 7f4a5b7fe640 1 --2- 172.21.15.116:0/3597700918 >> v2:172.21.15.156:3300/0 conn(0x7f4a5c06c250 0x7f4a5c06c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.984+0000 7f4a5b7fe640 1 -- 172.21.15.116:0/3597700918 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c070eb0 con 0x7f4a5c0703f0 2023-11-28T17:09:11.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.984+0000 7f4a5b7fe640 1 -- 172.21.15.116:0/3597700918 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f4a48001670 con 0x7f4a5c0703f0 2023-11-28T17:09:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.984+0000 7f4a5b7fe640 1 -- 172.21.15.116:0/3597700918 >> v1:172.21.15.116:6789/0 conn(0x7f4a5c0703f0 legacy=0x7f4a5c071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.985+0000 7f4a5b7fe640 1 --2- 172.21.15.116:0/3597700918 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a5c144430 0x7f4a4c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.985+0000 7f4a5b7fe640 1 --2- 172.21.15.116:0/3597700918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a4c003ef0 0x7f4a4c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.985+0000 7f4a5b7fe640 1 --2- 172.21.15.116:0/3597700918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a4c004d00 0x7f4a4c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.985+0000 7f4a5b7fe640 1 -- 172.21.15.116:0/3597700918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c144810 con 0x7f4a4c003ef0 2023-11-28T17:09:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.985+0000 7f4a5b7fe640 1 -- 172.21.15.116:0/3597700918 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c06cc20 con 0x7f4a4c004d00 2023-11-28T17:09:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.985+0000 7f4a5b7fe640 1 -- 172.21.15.116:0/3597700918 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c00b980 con 0x7f4a5c144430 2023-11-28T17:09:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.985+0000 7f4a5bfff640 1 --2- 172.21.15.116:0/3597700918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a4c003ef0 0x7f4a4c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.985+0000 7f4a60b04640 1 --2- 172.21.15.116:0/3597700918 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a5c144430 0x7f4a4c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.985+0000 7f4a5bfff640 1 -- 172.21.15.116:0/3597700918 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a5c144430 msgr2=0x7f4a4c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:11.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.985+0000 7f4a5bfff640 1 --2- 172.21.15.116:0/3597700918 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a5c144430 0x7f4a4c003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.985+0000 7f4a5bfff640 1 -- 172.21.15.116:0/3597700918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a4c004d00 msgr2=0x7f4a4c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:11.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.985+0000 7f4a5bfff640 1 --2- 172.21.15.116:0/3597700918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a4c004d00 0x7f4a4c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.985+0000 7f4a5bfff640 1 -- 172.21.15.116:0/3597700918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a4c003520 con 0x7f4a4c003ef0 2023-11-28T17:09:11.990 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.986+0000 7f4a5bfff640 1 --2- 172.21.15.116:0/3597700918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a4c003ef0 0x7f4a4c0047c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800c240 tx=0x7f4a4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fcc0d55bbd0563 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:11.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.986+0000 7f4a5b7fe640 1 -- 172.21.15.116:0/3597700918 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4802cae0 con 0x7f4a4c003ef0 2023-11-28T17:09:11.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.987+0000 7f4a5b7fe640 1 -- 172.21.15.116:0/3597700918 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4802cc40 con 0x7f4a4c003ef0 2023-11-28T17:09:11.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.987+0000 7f4a5b7fe640 1 -- 172.21.15.116:0/3597700918 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48005b10 con 0x7f4a4c003ef0 2023-11-28T17:09:11.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.987+0000 7f4a62d68640 1 -- 172.21.15.116:0/3597700918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a4c003ef0 msgr2=0x7f4a4c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.987+0000 7f4a62d68640 1 --2- 172.21.15.116:0/3597700918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a4c003ef0 0x7f4a4c0047c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800c240 tx=0x7f4a4c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.987+0000 7f4a61305640 1 -- 172.21.15.116:0/3597700918 reap_dead start 2023-11-28T17:09:11.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.987+0000 7f4a62d68640 1 -- 172.21.15.116:0/3597700918 shutdown_connections 2023-11-28T17:09:11.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.987+0000 7f4a62d68640 1 -- 172.21.15.116:0/3597700918 >> 172.21.15.116:0/3597700918 conn(0x7f4a5c06a4c0 msgr2=0x7f4a5c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:11.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.987+0000 7f4a62d68640 1 -- 172.21.15.116:0/3597700918 shutdown_connections 2023-11-28T17:09:11.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.987+0000 7f4a62d68640 1 -- 172.21.15.116:0/3597700918 wait complete. 2023-11-28T17:09:11.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.989+0000 7f4a62d68640 1 Processor -- start 2023-11-28T17:09:11.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.989+0000 7f4a62d68640 1 -- start start 2023-11-28T17:09:11.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.990+0000 7f4a62d68640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a4c003ef0 0x7f4a5c0affc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.990+0000 7f4a62d68640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a5c137c20 0x7f4a5c0ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.990+0000 7f4a62d68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a5c13ad60 0x7f4a5c15a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.990+0000 7f4a62d68640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c0adab0 con 0x7f4a5c13ad60 2023-11-28T17:09:11.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.990+0000 7f4a62d68640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c0aef60 con 0x7f4a5c137c20 2023-11-28T17:09:11.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.990+0000 7f4a62d68640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c0af200 con 0x7f4a4c003ef0 2023-11-28T17:09:11.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.990+0000 7f4a61305640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a5c13ad60 0x7f4a5c15a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.990+0000 7f4a61305640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a5c13ad60 0x7f4a5c15a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52160/0 (socket says 172.21.15.116:52160) 2023-11-28T17:09:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.990+0000 7f4a61305640 1 -- 172.21.15.116:0/1660273059 learned_addr learned my addr 172.21.15.116:0/1660273059 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.991+0000 7f4a60b04640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a4c003ef0 0x7f4a5c0affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.991+0000 7f4a61305640 1 -- 172.21.15.116:0/1660273059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a4c003ef0 msgr2=0x7f4a5c0affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.991+0000 7f4a5bfff640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a5c137c20 0x7f4a5c0ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.991+0000 7f4a61305640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a4c003ef0 0x7f4a5c0affc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.991+0000 7f4a61305640 1 -- 172.21.15.116:0/1660273059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a5c137c20 msgr2=0x7f4a5c0ad280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.991+0000 7f4a61305640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a5c137c20 0x7f4a5c0ad280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.991+0000 7f4a61305640 1 -- 172.21.15.116:0/1660273059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c0af4e0 con 0x7f4a5c13ad60 2023-11-28T17:09:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.991+0000 7f4a60b04640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a4c003ef0 0x7f4a5c0affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.991+0000 7f4a61305640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a5c13ad60 0x7f4a5c15a540 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c144e70 tx=0x7f4a5400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:11.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.991+0000 7f4a597fa640 1 -- 172.21.15.116:0/1660273059 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54010040 con 0x7f4a5c13ad60 2023-11-28T17:09:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.991+0000 7f4a597fa640 1 -- 172.21.15.116:0/1660273059 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5400bce0 con 0x7f4a5c13ad60 2023-11-28T17:09:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.991+0000 7f4a62d68640 1 -- 172.21.15.116:0/1660273059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5c0af7c0 con 0x7f4a5c13ad60 2023-11-28T17:09:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.991+0000 7f4a597fa640 1 -- 172.21.15.116:0/1660273059 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54014640 con 0x7f4a5c13ad60 2023-11-28T17:09:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.991+0000 7f4a62d68640 1 -- 172.21.15.116:0/1660273059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5c0ad7c0 con 0x7f4a5c13ad60 2023-11-28T17:09:11.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.993+0000 7f4a62d68640 1 -- 172.21.15.116:0/1660273059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5c0693d0 con 0x7f4a5c13ad60 2023-11-28T17:09:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.993+0000 7f4a597fa640 1 -- 172.21.15.116:0/1660273059 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4a540147a0 con 0x7f4a5c13ad60 2023-11-28T17:09:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.995+0000 7f4a597fa640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4a4006b2d0 0x7f4a4006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.995+0000 7f4a597fa640 1 -- 172.21.15.116:0/1660273059 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a5408a310 con 0x7f4a5c13ad60 2023-11-28T17:09:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.995+0000 7f4a60b04640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4a4006b2d0 0x7f4a4006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:11.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.995+0000 7f4a60b04640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4a4006b2d0 0x7f4a4006d790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009910 tx=0x7f4a50008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:12.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:11.999+0000 7f4a597fa640 1 -- 172.21.15.116:0/1660273059 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a54055210 con 0x7f4a5c13ad60 2023-11-28T17:09:12.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.232+0000 7f4a62d68640 1 -- 172.21.15.116:0/1660273059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4a5c06d100 con 0x7f4a5c13ad60 2023-11-28T17:09:12.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.232+0000 7f4a597fa640 1 -- 172.21.15.116:0/1660273059 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f4a54058830 con 0x7f4a5c13ad60 2023-11-28T17:09:12.234 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:12.235 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-28T17:09:12.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.235+0000 7f4a62d68640 1 -- 172.21.15.116:0/1660273059 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4a4006b2d0 msgr2=0x7f4a4006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:12.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.235+0000 7f4a62d68640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4a4006b2d0 0x7f4a4006d790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009910 tx=0x7f4a50008040 comp rx=0 tx=0).stop 2023-11-28T17:09:12.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.235+0000 7f4a62d68640 1 -- 172.21.15.116:0/1660273059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a5c13ad60 msgr2=0x7f4a5c15a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:12.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.235+0000 7f4a62d68640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a5c13ad60 0x7f4a5c15a540 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c144e70 tx=0x7f4a5400b480 comp rx=0 tx=0).stop 2023-11-28T17:09:12.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.235+0000 7f4a62d68640 1 -- 172.21.15.116:0/1660273059 shutdown_connections 2023-11-28T17:09:12.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.235+0000 7f4a62d68640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4a4006b2d0 0x7f4a4006d790 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.235+0000 7f4a62d68640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a5c13ad60 0x7f4a5c15a540 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.235+0000 7f4a62d68640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a5c137c20 0x7f4a5c0ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.235+0000 7f4a62d68640 1 --2- 172.21.15.116:0/1660273059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a4c003ef0 0x7f4a5c0affc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.235+0000 7f4a62d68640 1 -- 172.21.15.116:0/1660273059 >> 172.21.15.116:0/1660273059 conn(0x7f4a5c06a4c0 msgr2=0x7f4a5c143a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:12.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.235+0000 7f4a62d68640 1 -- 172.21.15.116:0/1660273059 shutdown_connections 2023-11-28T17:09:12.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.235+0000 7f4a62d68640 1 -- 172.21.15.116:0/1660273059 wait complete. 2023-11-28T17:09:12.255 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-28T17:09:12.255 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:12.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.462+0000 7f7391a6b640 1 Processor -- start 2023-11-28T17:09:12.464 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.462+0000 7f7391a6b640 1 -- start start 2023-11-28T17:09:12.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.464+0000 7f7391a6b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f738c144430 0x7f738c15f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:12.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.464+0000 7f7391a6b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f738c06c250 0x7f738c06c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:12.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.464+0000 7f7391a6b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c070bd0 con 0x7f738c0703f0 2023-11-28T17:09:12.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.464+0000 7f7391a6b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738c144810 con 0x7f738c06c250 2023-11-28T17:09:12.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.464+0000 7f7391a6b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738c06cc20 con 0x7f738c144430 2023-11-28T17:09:12.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.464+0000 7f738b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f738c144430 0x7f738c15f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:12.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.464+0000 7f738b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f738c144430 0x7f738c15f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:49792/0 (socket says 172.21.15.116:49792) 2023-11-28T17:09:12.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.464+0000 7f738b7fe640 1 -- 172.21.15.116:0/672515643 learned_addr learned my addr 172.21.15.116:0/672515643 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:12.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.465+0000 7f738bfff640 1 --2- 172.21.15.116:0/672515643 >> v2:172.21.15.156:3300/0 conn(0x7f738c06c250 0x7f738c06c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:12.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.465+0000 7f738affd640 1 -- 172.21.15.116:0/672515643 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142685424 0 0) 0x7f738c070bd0 con 0x7f738c0703f0 2023-11-28T17:09:12.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.465+0000 7f738affd640 1 -- 172.21.15.116:0/672515643 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7378003520 con 0x7f738c0703f0 2023-11-28T17:09:12.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.465+0000 7f738b7fe640 1 -- 172.21.15.116:0/672515643 >> v2:172.21.15.156:3300/0 conn(0x7f738c06c250 msgr2=0x7f738c06c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:12.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.465+0000 7f738b7fe640 1 --2- 172.21.15.116:0/672515643 >> v2:172.21.15.156:3300/0 conn(0x7f738c06c250 0x7f738c06c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.465+0000 7f738b7fe640 1 -- 172.21.15.116:0/672515643 >> v1:172.21.15.156:6789/0 conn(0x7f738c0703f0 legacy=0x7f738c071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:12.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.465+0000 7f738b7fe640 1 -- 172.21.15.116:0/672515643 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c070eb0 con 0x7f738c144430 2023-11-28T17:09:12.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.467+0000 7f738b7fe640 1 --2- 172.21.15.116:0/672515643 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f738c144430 0x7f738c15f7b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f737c009870 tx=0x7f737c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95d165ad75910ea server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.467+0000 7f738affd640 1 -- 172.21.15.116:0/672515643 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c030840 con 0x7f738c144430 2023-11-28T17:09:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.467+0000 7f738affd640 1 -- 172.21.15.116:0/672515643 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737c0309a0 con 0x7f738c144430 2023-11-28T17:09:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.467+0000 7f738affd640 1 -- 172.21.15.116:0/672515643 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c030d30 con 0x7f738c144430 2023-11-28T17:09:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.468+0000 7f7391a6b640 1 -- 172.21.15.116:0/672515643 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f738c144430 msgr2=0x7f738c15f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.468+0000 7f7391a6b640 1 --2- 172.21.15.116:0/672515643 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f738c144430 0x7f738c15f7b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f737c009870 tx=0x7f737c02b980 comp rx=0 tx=0).stop 2023-11-28T17:09:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.468+0000 7f7391a6b640 1 -- 172.21.15.116:0/672515643 shutdown_connections 2023-11-28T17:09:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.468+0000 7f7391a6b640 1 --2- 172.21.15.116:0/672515643 >> v2:172.21.15.156:3300/0 conn(0x7f738c06c250 0x7f738c06c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.468+0000 7f7391a6b640 1 --2- 172.21.15.116:0/672515643 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f738c144430 0x7f738c15f7b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.468+0000 7f7391a6b640 1 -- 172.21.15.116:0/672515643 >> 172.21.15.116:0/672515643 conn(0x7f738c06a4c0 msgr2=0x7f738c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.468+0000 7f7391a6b640 1 -- 172.21.15.116:0/672515643 shutdown_connections 2023-11-28T17:09:12.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.468+0000 7f7391a6b640 1 -- 172.21.15.116:0/672515643 wait complete. 2023-11-28T17:09:12.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.470+0000 7f7391a6b640 1 Processor -- start 2023-11-28T17:09:12.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.470+0000 7f7391a6b640 1 -- start start 2023-11-28T17:09:12.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.471+0000 7f7391a6b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f738c06c250 0x7f738c15a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:12.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.471+0000 7f7391a6b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c0703f0 0x7f738c13a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:12.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.471+0000 7f7391a6b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f738c144430 0x7f738c13bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:12.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.471+0000 7f7391a6b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738c13ecb0 con 0x7f738c0703f0 2023-11-28T17:09:12.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.471+0000 7f7391a6b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738c13e0a0 con 0x7f738c144430 2023-11-28T17:09:12.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.471+0000 7f7391a6b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738c13e380 con 0x7f738c06c250 2023-11-28T17:09:12.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.471+0000 7f7383fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c0703f0 0x7f738c13a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:12.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.471+0000 7f7383fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c0703f0 0x7f738c13a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52176/0 (socket says 172.21.15.116:52176) 2023-11-28T17:09:12.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.471+0000 7f7383fff640 1 -- 172.21.15.116:0/1954008827 learned_addr learned my addr 172.21.15.116:0/1954008827 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:12.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.472+0000 7f738bfff640 1 --2- 172.21.15.116:0/1954008827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f738c144430 0x7f738c13bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.472+0000 7f738b7fe640 1 --2- 172.21.15.116:0/1954008827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f738c06c250 0x7f738c15a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.472+0000 7f7383fff640 1 -- 172.21.15.116:0/1954008827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f738c06c250 msgr2=0x7f738c15a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.472+0000 7f7383fff640 1 --2- 172.21.15.116:0/1954008827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f738c06c250 0x7f738c15a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.472+0000 7f7383fff640 1 -- 172.21.15.116:0/1954008827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f738c144430 msgr2=0x7f738c13bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.472+0000 7f7383fff640 1 --2- 172.21.15.116:0/1954008827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f738c144430 0x7f738c13bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.472+0000 7f7383fff640 1 -- 172.21.15.116:0/1954008827 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737c0094d0 con 0x7f738c0703f0 2023-11-28T17:09:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.472+0000 7f7383fff640 1 --2- 172.21.15.116:0/1954008827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c0703f0 0x7f738c13a0d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f737400c490 tx=0x7f737400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.472+0000 7f7388ff9640 1 -- 172.21.15.116:0/1954008827 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73740052d0 con 0x7f738c0703f0 2023-11-28T17:09:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.473+0000 7f7391a6b640 1 -- 172.21.15.116:0/1954008827 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738c13e660 con 0x7f738c0703f0 2023-11-28T17:09:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.472+0000 7f7388ff9640 1 -- 172.21.15.116:0/1954008827 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7374005430 con 0x7f738c0703f0 2023-11-28T17:09:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.473+0000 7f7388ff9640 1 -- 172.21.15.116:0/1954008827 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374012870 con 0x7f738c0703f0 2023-11-28T17:09:12.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.473+0000 7f7391a6b640 1 -- 172.21.15.116:0/1954008827 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738c15ac70 con 0x7f738c0703f0 2023-11-28T17:09:12.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.474+0000 7f7391a6b640 1 -- 172.21.15.116:0/1954008827 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738c13ee60 con 0x7f738c0703f0 2023-11-28T17:09:12.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.475+0000 7f7388ff9640 1 -- 172.21.15.116:0/1954008827 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f73740129d0 con 0x7f738c0703f0 2023-11-28T17:09:12.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.475+0000 7f7388ff9640 1 --2- 172.21.15.116:0/1954008827 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f736c06b2a0 0x7f736c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:12.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.476+0000 7f7388ff9640 1 -- 172.21.15.116:0/1954008827 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7374089740 con 0x7f738c0703f0 2023-11-28T17:09:12.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.476+0000 7f738b7fe640 1 --2- 172.21.15.116:0/1954008827 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f736c06b2a0 0x7f736c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:12.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.477+0000 7f738b7fe640 1 --2- 172.21.15.116:0/1954008827 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f736c06b2a0 0x7f736c06d760 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f737c036000 tx=0x7f737c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:12.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.480+0000 7f7388ff9640 1 -- 172.21.15.116:0/1954008827 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73740545b0 con 0x7f738c0703f0 2023-11-28T17:09:12.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.671+0000 7f7391a6b640 1 -- 172.21.15.116:0/1954008827 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f738c0693d0 con 0x7f738c0703f0 2023-11-28T17:09:12.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.674+0000 7f7388ff9640 1 -- 172.21.15.116:0/1954008827 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16521 (secure 0 0 0) 0x7f7374057bd0 con 0x7f738c0703f0 2023-11-28T17:09:12.676 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:12.676 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:01.970957+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:12.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.677+0000 7f7391a6b640 1 -- 172.21.15.116:0/1954008827 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f736c06b2a0 msgr2=0x7f736c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:12.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.677+0000 7f7391a6b640 1 --2- 172.21.15.116:0/1954008827 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f736c06b2a0 0x7f736c06d760 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f737c036000 tx=0x7f737c008030 comp rx=0 tx=0).stop 2023-11-28T17:09:12.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.677+0000 7f7391a6b640 1 -- 172.21.15.116:0/1954008827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c0703f0 msgr2=0x7f738c13a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.677+0000 7f7391a6b640 1 --2- 172.21.15.116:0/1954008827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c0703f0 0x7f738c13a0d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f737400c490 tx=0x7f737400c950 comp rx=0 tx=0).stop 2023-11-28T17:09:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.677+0000 7f7391a6b640 1 -- 172.21.15.116:0/1954008827 shutdown_connections 2023-11-28T17:09:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.677+0000 7f7391a6b640 1 --2- 172.21.15.116:0/1954008827 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f736c06b2a0 0x7f736c06d760 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.677+0000 7f7391a6b640 1 --2- 172.21.15.116:0/1954008827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f738c144430 0x7f738c13bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.677+0000 7f7391a6b640 1 --2- 172.21.15.116:0/1954008827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f738c0703f0 0x7f738c13a0d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.677+0000 7f7391a6b640 1 --2- 172.21.15.116:0/1954008827 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f738c06c250 0x7f738c15a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.677+0000 7f7391a6b640 1 -- 172.21.15.116:0/1954008827 >> 172.21.15.116:0/1954008827 conn(0x7f738c06a4c0 msgr2=0x7f738c15b330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.677+0000 7f7391a6b640 1 -- 172.21.15.116:0/1954008827 shutdown_connections 2023-11-28T17:09:12.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.677+0000 7f7391a6b640 1 -- 172.21.15.116:0/1954008827 wait complete. 2023-11-28T17:09:12.699 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:09:12.699 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:09:12.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.909+0000 7f2abbbbe640 1 Processor -- start 2023-11-28T17:09:12.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.909+0000 7f2abbbbe640 1 -- start start 2023-11-28T17:09:12.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.911+0000 7f2abbbbe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2ab40d1620 0x7f2ab4161a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.911+0000 7f2abbbbe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab406eaf0 con 0x7f2ab4146b10 2023-11-28T17:09:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.911+0000 7f2abbbbe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab415e4e0 con 0x7f2ab415d8a0 2023-11-28T17:09:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.911+0000 7f2abbbbe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab415e7c0 con 0x7f2ab40d1620 2023-11-28T17:09:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.911+0000 7f2ab9159640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2ab40d1620 0x7f2ab4161a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:12.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.911+0000 7f2ab9159640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2ab40d1620 0x7f2ab4161a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52184/0 (socket says 172.21.15.116:52184) 2023-11-28T17:09:12.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.911+0000 7f2ab9159640 1 -- 172.21.15.116:0/1469082329 learned_addr learned my addr 172.21.15.116:0/1469082329 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:12.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.911+0000 7f2ab995a640 1 --1- 172.21.15.116:0/1469082329 >> v1:172.21.15.116:6789/0 conn(0x7f2ab4146b10 0x7f2ab40d0f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54708/0 (socket says 172.21.15.116:54708) 2023-11-28T17:09:12.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.912+0000 7f2ab9159640 1 -- 172.21.15.116:0/1469082329 >> v1:172.21.15.156:6789/0 conn(0x7f2ab415d8a0 legacy=0x7f2ab415dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:12.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.912+0000 7f2ab9159640 1 -- 172.21.15.116:0/1469082329 >> v1:172.21.15.116:6789/0 conn(0x7f2ab4146b10 legacy=0x7f2ab40d0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:12.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.912+0000 7f2ab8958640 1 -- 172.21.15.116:0/1469082329 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072873475 0 0) 0x7f2ab406eaf0 con 0x7f2ab4146b10 2023-11-28T17:09:12.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.912+0000 7f2ab9159640 1 -- 172.21.15.116:0/1469082329 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab4160f50 con 0x7f2ab40d1620 2023-11-28T17:09:12.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.913+0000 7f2ab9159640 1 --2- 172.21.15.116:0/1469082329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab40d1620 0x7f2ab4161a30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009870 tx=0x7f2aa402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc184d7f8eb2179b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:12.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.913+0000 7f2ab8958640 1 -- 172.21.15.116:0/1469082329 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4030860 con 0x7f2ab40d1620 2023-11-28T17:09:12.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.913+0000 7f2ab8958640 1 -- 172.21.15.116:0/1469082329 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa40309c0 con 0x7f2ab40d1620 2023-11-28T17:09:12.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.913+0000 7f2ab8958640 1 -- 172.21.15.116:0/1469082329 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4030d50 con 0x7f2ab40d1620 2023-11-28T17:09:12.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.914+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1469082329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab40d1620 msgr2=0x7f2ab4161a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:12.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.914+0000 7f2abbbbe640 1 --2- 172.21.15.116:0/1469082329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab40d1620 0x7f2ab4161a30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009870 tx=0x7f2aa402b9a0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.914+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1469082329 shutdown_connections 2023-11-28T17:09:12.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.914+0000 7f2abbbbe640 1 --2- 172.21.15.116:0/1469082329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab40d1620 0x7f2ab4161a30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.914+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1469082329 >> 172.21.15.116:0/1469082329 conn(0x7f2ab4070af0 msgr2=0x7f2ab40662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:12.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.914+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1469082329 shutdown_connections 2023-11-28T17:09:12.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.914+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1469082329 wait complete. 2023-11-28T17:09:12.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.915+0000 7f2abbbbe640 1 Processor -- start 2023-11-28T17:09:12.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.915+0000 7f2abbbbe640 1 -- start start 2023-11-28T17:09:12.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.916+0000 7f2abbbbe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab40d1620 0x7f2ab413ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:12.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.916+0000 7f2abbbbe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab4146b10 0x7f2ab40d0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:12.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.916+0000 7f2abbbbe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab415d8a0 0x7f2ab40b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:12.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.916+0000 7f2abbbbe640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab413f390 con 0x7f2ab4146b10 2023-11-28T17:09:12.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.916+0000 7f2abbbbe640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab40d0da0 con 0x7f2ab40d1620 2023-11-28T17:09:12.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.916+0000 7f2abbbbe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab415d090 con 0x7f2ab415d8a0 2023-11-28T17:09:12.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.917+0000 7f2ab9159640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab4146b10 0x7f2ab40d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:12.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.917+0000 7f2ab9159640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab4146b10 0x7f2ab40d0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52196/0 (socket says 172.21.15.116:52196) 2023-11-28T17:09:12.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.917+0000 7f2ab9159640 1 -- 172.21.15.116:0/1625748621 learned_addr learned my addr 172.21.15.116:0/1625748621 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:12.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.917+0000 7f2ab9159640 1 -- 172.21.15.116:0/1625748621 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab415d8a0 msgr2=0x7f2ab40b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:12.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.917+0000 7f2ab9159640 1 --2- 172.21.15.116:0/1625748621 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab415d8a0 0x7f2ab40b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.917+0000 7f2ab9159640 1 -- 172.21.15.116:0/1625748621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab40d1620 msgr2=0x7f2ab413ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:12.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.917+0000 7f2ab9159640 1 --2- 172.21.15.116:0/1625748621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab40d1620 0x7f2ab413ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:12.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.917+0000 7f2ab9159640 1 -- 172.21.15.116:0/1625748621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa40094d0 con 0x7f2ab4146b10 2023-11-28T17:09:12.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.917+0000 7f2ab9159640 1 --2- 172.21.15.116:0/1625748621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab4146b10 0x7f2ab40d0860 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4000c00 tx=0x7f2aa4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:12.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.917+0000 7f2aaa7fc640 1 -- 172.21.15.116:0/1625748621 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4039d60 con 0x7f2ab4146b10 2023-11-28T17:09:12.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.917+0000 7f2aaa7fc640 1 -- 172.21.15.116:0/1625748621 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa4043400 con 0x7f2ab4146b10 2023-11-28T17:09:12.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.917+0000 7f2aaa7fc640 1 -- 172.21.15.116:0/1625748621 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa40437b0 con 0x7f2ab4146b10 2023-11-28T17:09:12.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.917+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1625748621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab40cf7c0 con 0x7f2ab4146b10 2023-11-28T17:09:12.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.918+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1625748621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab40cfd80 con 0x7f2ab4146b10 2023-11-28T17:09:12.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.918+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1625748621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab4069360 con 0x7f2ab4146b10 2023-11-28T17:09:12.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.919+0000 7f2aaa7fc640 1 -- 172.21.15.116:0/1625748621 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2aa4043910 con 0x7f2ab4146b10 2023-11-28T17:09:12.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.920+0000 7f2aaa7fc640 1 --2- 172.21.15.116:0/1625748621 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2a9806b2a0 0x7f2a9806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:12.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.920+0000 7f2aaa7fc640 1 -- 172.21.15.116:0/1625748621 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2aa403e070 con 0x7f2ab4146b10 2023-11-28T17:09:12.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.921+0000 7f2ab995a640 1 --2- 172.21.15.116:0/1625748621 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2a9806b2a0 0x7f2a9806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:12.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.921+0000 7f2ab995a640 1 --2- 172.21.15.116:0/1625748621 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2a9806b2a0 0x7f2a9806d760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0007fd0 tx=0x7f2ab0007920 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:12.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:12.924+0000 7f2aaa7fc640 1 -- 172.21.15.116:0/1625748621 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2aa4077e20 con 0x7f2ab4146b10 2023-11-28T17:09:13.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.117+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1625748621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2ab4141fc0 con 0x7f2ab4146b10 2023-11-28T17:09:13.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.117+0000 7f2aaa7fc640 1 -- 172.21.15.116:0/1625748621 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2aa407b440 con 0x7f2ab4146b10 2023-11-28T17:09:13.119 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:09:13.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.120+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1625748621 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2a9806b2a0 msgr2=0x7f2a9806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:13.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.120+0000 7f2abbbbe640 1 --2- 172.21.15.116:0/1625748621 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2a9806b2a0 0x7f2a9806d760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0007fd0 tx=0x7f2ab0007920 comp rx=0 tx=0).stop 2023-11-28T17:09:13.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.120+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1625748621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab4146b10 msgr2=0x7f2ab40d0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:13.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.120+0000 7f2abbbbe640 1 --2- 172.21.15.116:0/1625748621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab4146b10 0x7f2ab40d0860 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4000c00 tx=0x7f2aa4004730 comp rx=0 tx=0).stop 2023-11-28T17:09:13.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.120+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1625748621 shutdown_connections 2023-11-28T17:09:13.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.120+0000 7f2abbbbe640 1 --2- 172.21.15.116:0/1625748621 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2a9806b2a0 0x7f2a9806d760 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.120+0000 7f2abbbbe640 1 --2- 172.21.15.116:0/1625748621 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab415d8a0 0x7f2ab40b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.120+0000 7f2abbbbe640 1 --2- 172.21.15.116:0/1625748621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab4146b10 0x7f2ab40d0860 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.120+0000 7f2abbbbe640 1 --2- 172.21.15.116:0/1625748621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab40d1620 0x7f2ab413ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.120+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1625748621 >> 172.21.15.116:0/1625748621 conn(0x7f2ab4070af0 msgr2=0x7f2ab415fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:13.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.120+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1625748621 shutdown_connections 2023-11-28T17:09:13.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.120+0000 7f2abbbbe640 1 -- 172.21.15.116:0/1625748621 wait complete. 2023-11-28T17:09:13.140 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:09:13.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.346+0000 7fb75759e640 1 Processor -- start 2023-11-28T17:09:13.348 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.346+0000 7fb75759e640 1 -- start start 2023-11-28T17:09:13.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.348+0000 7fb75759e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb7580b2fe0 0x7fb7580ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:13.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.348+0000 7fb75759e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb7580abd30 0x7fb7580ade70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:13.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.348+0000 7fb75759e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7580b2b50 con 0x7fb758152050 2023-11-28T17:09:13.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.348+0000 7fb75759e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7580ae3b0 con 0x7fb7580b2fe0 2023-11-28T17:09:13.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.348+0000 7fb75759e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7580ac2d0 con 0x7fb7580abd30 2023-11-28T17:09:13.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.348+0000 7fb755d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb7580b2fe0 0x7fb7580ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:13.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.348+0000 7fb755d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb7580b2fe0 0x7fb7580ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52210/0 (socket says 172.21.15.116:52210) 2023-11-28T17:09:13.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.348+0000 7fb75659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb758152050 0x7fb75815bfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54722/0 (socket says 172.21.15.116:54722) 2023-11-28T17:09:13.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.348+0000 7fb755d9b640 1 -- 172.21.15.116:0/3429291611 learned_addr learned my addr 172.21.15.116:0/3429291611 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:13.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.349+0000 7fb756d9d640 1 --2- 172.21.15.116:0/3429291611 >> v2:172.21.15.156:3300/0 conn(0x7fb7580abd30 0x7fb7580ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:13.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.349+0000 7fb755d9b640 1 -- 172.21.15.116:0/3429291611 >> v2:172.21.15.156:3300/0 conn(0x7fb7580abd30 msgr2=0x7fb7580ade70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:13.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.349+0000 7fb755d9b640 1 --2- 172.21.15.116:0/3429291611 >> v2:172.21.15.156:3300/0 conn(0x7fb7580abd30 0x7fb7580ade70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.349+0000 7fb755d9b640 1 -- 172.21.15.116:0/3429291611 >> v1:172.21.15.116:6789/0 conn(0x7fb758152050 legacy=0x7fb75815bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:13.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.349+0000 7fb755d9b640 1 -- 172.21.15.116:0/3429291611 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7580b2220 con 0x7fb7580b2fe0 2023-11-28T17:09:13.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.350+0000 7fb755d9b640 1 --2- 172.21.15.116:0/3429291611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7580b2fe0 0x7fb7580ad930 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb7580b1930 tx=0x7fb74c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24acc11f00f4957 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:13.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.350+0000 7fb75559a640 1 -- 172.21.15.116:0/3429291611 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c030860 con 0x7fb7580b2fe0 2023-11-28T17:09:13.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.350+0000 7fb75559a640 1 -- 172.21.15.116:0/3429291611 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74c0309c0 con 0x7fb7580b2fe0 2023-11-28T17:09:13.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.350+0000 7fb75559a640 1 -- 172.21.15.116:0/3429291611 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c030d50 con 0x7fb7580b2fe0 2023-11-28T17:09:13.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.351+0000 7fb75759e640 1 -- 172.21.15.116:0/3429291611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7580b2fe0 msgr2=0x7fb7580ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:13.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.351+0000 7fb75759e640 1 --2- 172.21.15.116:0/3429291611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7580b2fe0 0x7fb7580ad930 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb7580b1930 tx=0x7fb74c02b9a0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.351+0000 7fb75759e640 1 -- 172.21.15.116:0/3429291611 shutdown_connections 2023-11-28T17:09:13.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.351+0000 7fb75759e640 1 --2- 172.21.15.116:0/3429291611 >> v2:172.21.15.156:3300/0 conn(0x7fb7580abd30 0x7fb7580ade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.351+0000 7fb75759e640 1 --2- 172.21.15.116:0/3429291611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7580b2fe0 0x7fb7580ad930 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.351+0000 7fb75759e640 1 -- 172.21.15.116:0/3429291611 >> 172.21.15.116:0/3429291611 conn(0x7fb7580b0950 msgr2=0x7fb7580b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:13.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.351+0000 7fb75759e640 1 -- 172.21.15.116:0/3429291611 shutdown_connections 2023-11-28T17:09:13.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.351+0000 7fb75759e640 1 -- 172.21.15.116:0/3429291611 wait complete. 2023-11-28T17:09:13.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.353+0000 7fb75759e640 1 Processor -- start 2023-11-28T17:09:13.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.353+0000 7fb75759e640 1 -- start start 2023-11-28T17:09:13.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.354+0000 7fb75759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7580abd30 0x7fb758147690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.354+0000 7fb75759e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb7580b2fe0 0x7fb758070e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.354+0000 7fb75759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb758152050 0x7fb75815a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.354+0000 7fb75759e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75809a6f0 con 0x7fb7580abd30 2023-11-28T17:09:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.354+0000 7fb75759e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75809a9d0 con 0x7fb758152050 2023-11-28T17:09:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.354+0000 7fb75759e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75809acb0 con 0x7fb7580b2fe0 2023-11-28T17:09:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.354+0000 7fb75659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7580abd30 0x7fb758147690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.354+0000 7fb75659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7580abd30 0x7fb758147690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52222/0 (socket says 172.21.15.116:52222) 2023-11-28T17:09:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.354+0000 7fb75659c640 1 -- 172.21.15.116:0/753111617 learned_addr learned my addr 172.21.15.116:0/753111617 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.354+0000 7fb75659c640 1 -- 172.21.15.116:0/753111617 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb7580b2fe0 msgr2=0x7fb758070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.354+0000 7fb75659c640 1 --2- 172.21.15.116:0/753111617 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb7580b2fe0 0x7fb758070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.354+0000 7fb75659c640 1 -- 172.21.15.116:0/753111617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb758152050 msgr2=0x7fb75815a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:13.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.354+0000 7fb75659c640 1 --2- 172.21.15.116:0/753111617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb758152050 0x7fb75815a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.354+0000 7fb75659c640 1 -- 172.21.15.116:0/753111617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0094d0 con 0x7fb7580abd30 2023-11-28T17:09:13.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.355+0000 7fb75659c640 1 --2- 172.21.15.116:0/753111617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7580abd30 0x7fb758147690 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb74800c3e0 tx=0x7fb74800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.355+0000 7fb746ffd640 1 -- 172.21.15.116:0/753111617 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7480074e0 con 0x7fb7580abd30 2023-11-28T17:09:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.355+0000 7fb746ffd640 1 -- 172.21.15.116:0/753111617 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb748007640 con 0x7fb7580abd30 2023-11-28T17:09:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.355+0000 7fb75759e640 1 -- 172.21.15.116:0/753111617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75809af90 con 0x7fb7580abd30 2023-11-28T17:09:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.355+0000 7fb746ffd640 1 -- 172.21.15.116:0/753111617 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7480049e0 con 0x7fb7580abd30 2023-11-28T17:09:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.355+0000 7fb75759e640 1 -- 172.21.15.116:0/753111617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7580713a0 con 0x7fb7580abd30 2023-11-28T17:09:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.356+0000 7fb75759e640 1 -- 172.21.15.116:0/753111617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb758071590 con 0x7fb7580abd30 2023-11-28T17:09:13.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.357+0000 7fb746ffd640 1 -- 172.21.15.116:0/753111617 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb748004b40 con 0x7fb7580abd30 2023-11-28T17:09:13.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.358+0000 7fb746ffd640 1 --2- 172.21.15.116:0/753111617 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb73806b2d0 0x7fb73806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:13.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.359+0000 7fb746ffd640 1 -- 172.21.15.116:0/753111617 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb748089500 con 0x7fb7580abd30 2023-11-28T17:09:13.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.359+0000 7fb755d9b640 1 --2- 172.21.15.116:0/753111617 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb73806b2d0 0x7fb73806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:13.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.359+0000 7fb755d9b640 1 --2- 172.21.15.116:0/753111617 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb73806b2d0 0x7fb73806d790 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb74c036000 tx=0x7fb74c03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:13.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.363+0000 7fb746ffd640 1 -- 172.21.15.116:0/753111617 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb758071590 con 0x7fb7580abd30 2023-11-28T17:09:13.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.551+0000 7fb75759e640 1 -- 172.21.15.116:0/753111617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb758071590 con 0x7fb7580abd30 2023-11-28T17:09:13.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.552+0000 7fb746ffd640 1 -- 172.21.15.116:0/753111617 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb758071590 con 0x7fb7580abd30 2023-11-28T17:09:13.553 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:09:13.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.554+0000 7fb75759e640 1 -- 172.21.15.116:0/753111617 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb73806b2d0 msgr2=0x7fb73806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.554+0000 7fb75759e640 1 --2- 172.21.15.116:0/753111617 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb73806b2d0 0x7fb73806d790 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb74c036000 tx=0x7fb74c03b040 comp rx=0 tx=0).stop 2023-11-28T17:09:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.554+0000 7fb75759e640 1 -- 172.21.15.116:0/753111617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7580abd30 msgr2=0x7fb758147690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.554+0000 7fb75759e640 1 --2- 172.21.15.116:0/753111617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7580abd30 0x7fb758147690 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb74800c3e0 tx=0x7fb74800c8a0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.555+0000 7fb75759e640 1 -- 172.21.15.116:0/753111617 shutdown_connections 2023-11-28T17:09:13.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.555+0000 7fb75759e640 1 --2- 172.21.15.116:0/753111617 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb73806b2d0 0x7fb73806d790 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.555+0000 7fb75759e640 1 --2- 172.21.15.116:0/753111617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb758152050 0x7fb75815a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.555+0000 7fb75759e640 1 --2- 172.21.15.116:0/753111617 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb7580b2fe0 0x7fb758070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.555+0000 7fb75759e640 1 --2- 172.21.15.116:0/753111617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7580abd30 0x7fb758147690 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.555+0000 7fb75759e640 1 -- 172.21.15.116:0/753111617 >> 172.21.15.116:0/753111617 conn(0x7fb7580b0950 msgr2=0x7fb7580ad8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:13.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.555+0000 7fb75759e640 1 -- 172.21.15.116:0/753111617 shutdown_connections 2023-11-28T17:09:13.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.555+0000 7fb75759e640 1 -- 172.21.15.116:0/753111617 wait complete. 2023-11-28T17:09:13.574 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:09:13.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.782+0000 7fa4b0d83640 1 Processor -- start 2023-11-28T17:09:13.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.782+0000 7fa4b0d83640 1 -- start start 2023-11-28T17:09:13.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.784+0000 7fa4b0d83640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa4ac15b0c0 0x7fa4ac15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:13.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.784+0000 7fa4b0d83640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa4ac0ce940 0x7fa4ac0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.784+0000 7fa4b0d83640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ac071b40 0x7fa4ac071fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.784+0000 7fa4b0d83640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ac0724f0 con 0x7fa4ac0ce940 2023-11-28T17:09:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.784+0000 7fa4b0d83640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ac0cd320 con 0x7fa4ac15b0c0 2023-11-28T17:09:13.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.784+0000 7fa4b0d83640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac0ced80 con 0x7fa4ac071b40 2023-11-28T17:09:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.784+0000 7fa4a9d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa4ac0ce940 0x7fa4ac0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.784+0000 7fa4a9d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa4ac0ce940 0x7fa4ac0cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52228/0 (socket says 172.21.15.116:52228) 2023-11-28T17:09:13.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.784+0000 7fa4a9d9b640 1 -- 172.21.15.116:0/144829426 learned_addr learned my addr 172.21.15.116:0/144829426 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.784+0000 7fa4aad9d640 1 --2- 172.21.15.116:0/144829426 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ac071b40 0x7fa4ac071fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:13.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.784+0000 7fa4a9d9b640 1 -- 172.21.15.116:0/144829426 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ac071b40 msgr2=0x7fa4ac071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.784+0000 7fa4a9d9b640 1 --2- 172.21.15.116:0/144829426 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ac071b40 0x7fa4ac071fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.785+0000 7fa4a9d9b640 1 -- 172.21.15.116:0/144829426 >> v2:172.21.15.156:3300/0 conn(0x7fa4ac15b0c0 msgr2=0x7fa4ac15ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.785+0000 7fa4aa59c640 1 --2- 172.21.15.116:0/144829426 >> v2:172.21.15.156:3300/0 conn(0x7fa4ac15b0c0 0x7fa4ac15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.785+0000 7fa4a9d9b640 1 --2- 172.21.15.116:0/144829426 >> v2:172.21.15.156:3300/0 conn(0x7fa4ac15b0c0 0x7fa4ac15ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.785+0000 7fa4a9d9b640 1 -- 172.21.15.116:0/144829426 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac0cbee0 con 0x7fa4ac0ce940 2023-11-28T17:09:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.785+0000 7fa4aad9d640 1 --2- 172.21.15.116:0/144829426 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ac071b40 0x7fa4ac071fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.785+0000 7fa4aa59c640 1 --2- 172.21.15.116:0/144829426 >> v2:172.21.15.156:3300/0 conn(0x7fa4ac15b0c0 0x7fa4ac15ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:13.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.786+0000 7fa4a9d9b640 1 --2- 172.21.15.116:0/144829426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ac0ce940 0x7fa4ac0cb9a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac06b470 tx=0x7fa49402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161f024fc325f378 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.786+0000 7fa4a959a640 1 -- 172.21.15.116:0/144829426 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4940317e0 con 0x7fa4ac0ce940 2023-11-28T17:09:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.786+0000 7fa4a959a640 1 -- 172.21.15.116:0/144829426 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa494031940 con 0x7fa4ac0ce940 2023-11-28T17:09:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.786+0000 7fa4a959a640 1 -- 172.21.15.116:0/144829426 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa494031cd0 con 0x7fa4ac0ce940 2023-11-28T17:09:13.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.787+0000 7fa4b0d83640 1 -- 172.21.15.116:0/144829426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ac0ce940 msgr2=0x7fa4ac0cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.787+0000 7fa4b0d83640 1 --2- 172.21.15.116:0/144829426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ac0ce940 0x7fa4ac0cb9a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac06b470 tx=0x7fa49402b880 comp rx=0 tx=0).stop 2023-11-28T17:09:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.787+0000 7fa4b0d83640 1 -- 172.21.15.116:0/144829426 shutdown_connections 2023-11-28T17:09:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.787+0000 7fa4b0d83640 1 --2- 172.21.15.116:0/144829426 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ac071b40 0x7fa4ac071fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.787+0000 7fa4b0d83640 1 --2- 172.21.15.116:0/144829426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ac0ce940 0x7fa4ac0cb9a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.787+0000 7fa4b0d83640 1 --2- 172.21.15.116:0/144829426 >> v2:172.21.15.156:3300/0 conn(0x7fa4ac15b0c0 0x7fa4ac15ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.787+0000 7fa4b0d83640 1 -- 172.21.15.116:0/144829426 >> 172.21.15.116:0/144829426 conn(0x7fa4ac0b0cf0 msgr2=0x7fa4ac0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.787+0000 7fa4b0d83640 1 -- 172.21.15.116:0/144829426 shutdown_connections 2023-11-28T17:09:13.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.787+0000 7fa4b0d83640 1 -- 172.21.15.116:0/144829426 wait complete. 2023-11-28T17:09:13.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.789+0000 7fa4b0d83640 1 Processor -- start 2023-11-28T17:09:13.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.789+0000 7fa4b0d83640 1 -- start start 2023-11-28T17:09:13.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.790+0000 7fa4b0d83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ac071b40 0x7fa4ac0b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:13.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.790+0000 7fa4b0d83640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ac0ce940 0x7fa4ac150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:13.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.790+0000 7fa4b0d83640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ac15b0c0 0x7fa4ac1506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:13.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.790+0000 7fa4b0d83640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac158ca0 con 0x7fa4ac071b40 2023-11-28T17:09:13.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.790+0000 7fa4b0d83640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac158ef0 con 0x7fa4ac15b0c0 2023-11-28T17:09:13.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.790+0000 7fa4b0d83640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac1591d0 con 0x7fa4ac0ce940 2023-11-28T17:09:13.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.790+0000 7fa4aa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ac071b40 0x7fa4ac0b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:13.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4aa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ac071b40 0x7fa4ac0b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52242/0 (socket says 172.21.15.116:52242) 2023-11-28T17:09:13.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4aa59c640 1 -- 172.21.15.116:0/682953731 learned_addr learned my addr 172.21.15.116:0/682953731 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:13.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4a9d9b640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ac0ce940 0x7fa4ac150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:13.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4aad9d640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ac15b0c0 0x7fa4ac1506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:13.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4aa59c640 1 -- 172.21.15.116:0/682953731 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ac0ce940 msgr2=0x7fa4ac150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:13.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4aa59c640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ac0ce940 0x7fa4ac150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4aa59c640 1 -- 172.21.15.116:0/682953731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ac15b0c0 msgr2=0x7fa4ac1506a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:13.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4aa59c640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ac15b0c0 0x7fa4ac1506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4aa59c640 1 -- 172.21.15.116:0/682953731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4940094d0 con 0x7fa4ac071b40 2023-11-28T17:09:13.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4a9d9b640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ac0ce940 0x7fa4ac150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:13.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4aa59c640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ac071b40 0x7fa4ac0b03f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00ce00 tx=0x7fa49c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:13.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4a2ffd640 1 -- 172.21.15.116:0/682953731 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c00f890 con 0x7fa4ac071b40 2023-11-28T17:09:13.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4a2ffd640 1 -- 172.21.15.116:0/682953731 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c00f9f0 con 0x7fa4ac071b40 2023-11-28T17:09:13.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4a2ffd640 1 -- 172.21.15.116:0/682953731 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c00fdc0 con 0x7fa4ac071b40 2023-11-28T17:09:13.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.791+0000 7fa4b0d83640 1 -- 172.21.15.116:0/682953731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ac1594b0 con 0x7fa4ac071b40 2023-11-28T17:09:13.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.792+0000 7fa4b0d83640 1 -- 172.21.15.116:0/682953731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ac159cb0 con 0x7fa4ac071b40 2023-11-28T17:09:13.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.793+0000 7fa4b0d83640 1 -- 172.21.15.116:0/682953731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ac159ea0 con 0x7fa4ac071b40 2023-11-28T17:09:13.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.794+0000 7fa4a2ffd640 1 -- 172.21.15.116:0/682953731 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa49c00b040 con 0x7fa4ac071b40 2023-11-28T17:09:13.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.794+0000 7fa4a2ffd640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa48c06b2a0 0x7fa48c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:13.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.794+0000 7fa4a2ffd640 1 -- 172.21.15.116:0/682953731 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa49c08a480 con 0x7fa4ac071b40 2023-11-28T17:09:13.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.795+0000 7fa4a9d9b640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa48c06b2a0 0x7fa48c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:13.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.795+0000 7fa4a9d9b640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa48c06b2a0 0x7fa48c06d760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa494036000 tx=0x7fa49403d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:13.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.799+0000 7fa4a2ffd640 1 -- 172.21.15.116:0/682953731 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa49c055380 con 0x7fa4ac071b40 2023-11-28T17:09:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.987+0000 7fa4b0d83640 1 -- 172.21.15.116:0/682953731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa4ac069360 con 0x7fa4ac071b40 2023-11-28T17:09:13.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.988+0000 7fa4a2ffd640 1 -- 172.21.15.116:0/682953731 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa49c01d020 con 0x7fa4ac071b40 2023-11-28T17:09:13.990 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:09:13.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.990+0000 7fa4b0d83640 1 -- 172.21.15.116:0/682953731 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa48c06b2a0 msgr2=0x7fa48c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:13.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.990+0000 7fa4b0d83640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa48c06b2a0 0x7fa48c06d760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa494036000 tx=0x7fa49403d040 comp rx=0 tx=0).stop 2023-11-28T17:09:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.990+0000 7fa4b0d83640 1 -- 172.21.15.116:0/682953731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ac071b40 msgr2=0x7fa4ac0b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.990+0000 7fa4b0d83640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ac071b40 0x7fa4ac0b03f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00ce00 tx=0x7fa49c00a650 comp rx=0 tx=0).stop 2023-11-28T17:09:13.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.991+0000 7fa4b0d83640 1 -- 172.21.15.116:0/682953731 shutdown_connections 2023-11-28T17:09:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.991+0000 7fa4b0d83640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa48c06b2a0 0x7fa48c06d760 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.991+0000 7fa4b0d83640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4ac15b0c0 0x7fa4ac1506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.991+0000 7fa4b0d83640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ac0ce940 0x7fa4ac150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.991+0000 7fa4b0d83640 1 --2- 172.21.15.116:0/682953731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ac071b40 0x7fa4ac0b03f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.991+0000 7fa4b0d83640 1 -- 172.21.15.116:0/682953731 >> 172.21.15.116:0/682953731 conn(0x7fa4ac0b0cf0 msgr2=0x7fa4ac06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.991+0000 7fa4b0d83640 1 -- 172.21.15.116:0/682953731 shutdown_connections 2023-11-28T17:09:13.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:13.991+0000 7fa4b0d83640 1 -- 172.21.15.116:0/682953731 wait complete. 2023-11-28T17:09:14.009 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:09:14.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.210+0000 7f84a608c640 1 Processor -- start 2023-11-28T17:09:14.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.210+0000 7f84a608c640 1 -- start start 2023-11-28T17:09:14.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.212+0000 7f84a608c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f84a0153bf0 0x7f84a015a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:14.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.212+0000 7f84a608c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f84a00c9ab0 0x7f84a00cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:14.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.212+0000 7f84a608c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a00c9870 con 0x7f84a00cdd30 2023-11-28T17:09:14.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.212+0000 7f84a608c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a006a4c0 con 0x7f84a00c9ab0 2023-11-28T17:09:14.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.212+0000 7f84a608c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a0153fd0 con 0x7f84a0153bf0 2023-11-28T17:09:14.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.212+0000 7f84a4889640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f84a00c9ab0 0x7f84a00cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:14.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.212+0000 7f84a4889640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f84a00c9ab0 0x7f84a00cd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52258/0 (socket says 172.21.15.116:52258) 2023-11-28T17:09:14.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.212+0000 7f84a4889640 1 -- 172.21.15.116:0/3146545332 learned_addr learned my addr 172.21.15.116:0/3146545332 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:14.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.213+0000 7f84a4889640 1 -- 172.21.15.116:0/3146545332 >> v2:172.21.15.156:3300/0 conn(0x7f84a0153bf0 msgr2=0x7f84a015a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:14.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.213+0000 7f84a4889640 1 --2- 172.21.15.116:0/3146545332 >> v2:172.21.15.156:3300/0 conn(0x7f84a0153bf0 0x7f84a015a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.213+0000 7f84a4889640 1 -- 172.21.15.116:0/3146545332 >> v1:172.21.15.156:6789/0 conn(0x7f84a00cdd30 legacy=0x7f84a006b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:14.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.213+0000 7f84a4889640 1 -- 172.21.15.116:0/3146545332 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a006a6d0 con 0x7f84a00c9ab0 2023-11-28T17:09:14.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.214+0000 7f84a4889640 1 --2- 172.21.15.116:0/3146545332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a00c9ab0 0x7f84a00cd7f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f84a00cd150 tx=0x7f849002c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e84c89b7509c3450 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:14.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.214+0000 7f8497fff640 1 -- 172.21.15.116:0/3146545332 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490004230 con 0x7f84a00c9ab0 2023-11-28T17:09:14.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.214+0000 7f8497fff640 1 -- 172.21.15.116:0/3146545332 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8490004390 con 0x7f84a00c9ab0 2023-11-28T17:09:14.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.214+0000 7f8497fff640 1 -- 172.21.15.116:0/3146545332 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490031830 con 0x7f84a00c9ab0 2023-11-28T17:09:14.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.215+0000 7f84a608c640 1 -- 172.21.15.116:0/3146545332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a00c9ab0 msgr2=0x7f84a00cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:14.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.215+0000 7f84a608c640 1 --2- 172.21.15.116:0/3146545332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a00c9ab0 0x7f84a00cd7f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f84a00cd150 tx=0x7f849002c970 comp rx=0 tx=0).stop 2023-11-28T17:09:14.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.215+0000 7f84a608c640 1 -- 172.21.15.116:0/3146545332 shutdown_connections 2023-11-28T17:09:14.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.215+0000 7f84a608c640 1 --2- 172.21.15.116:0/3146545332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a00c9ab0 0x7f84a00cd7f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.215+0000 7f84a608c640 1 --2- 172.21.15.116:0/3146545332 >> v2:172.21.15.156:3300/0 conn(0x7f84a0153bf0 0x7f84a015a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.215+0000 7f84a608c640 1 -- 172.21.15.116:0/3146545332 >> 172.21.15.116:0/3146545332 conn(0x7f84a00cbee0 msgr2=0x7f84a00cc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:14.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.215+0000 7f84a608c640 1 -- 172.21.15.116:0/3146545332 shutdown_connections 2023-11-28T17:09:14.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.215+0000 7f84a608c640 1 -- 172.21.15.116:0/3146545332 wait complete. 2023-11-28T17:09:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.217+0000 7f84a608c640 1 Processor -- start 2023-11-28T17:09:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.217+0000 7f84a608c640 1 -- start start 2023-11-28T17:09:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a608c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84a00c9ab0 0x7f84a0149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a608c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84a00cdd30 0x7f84a015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a608c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a0153bf0 0x7f84a014ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a608c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a015c350 con 0x7f84a0153bf0 2023-11-28T17:09:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a608c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a015cb60 con 0x7f84a00cdd30 2023-11-28T17:09:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a608c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a01495d0 con 0x7f84a00c9ab0 2023-11-28T17:09:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a588b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a0153bf0 0x7f84a014ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a588b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a0153bf0 0x7f84a014ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52270/0 (socket says 172.21.15.116:52270) 2023-11-28T17:09:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a588b640 1 -- 172.21.15.116:0/2786258336 learned_addr learned my addr 172.21.15.116:0/2786258336 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a4889640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84a00cdd30 0x7f84a015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a588b640 1 -- 172.21.15.116:0/2786258336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84a00c9ab0 msgr2=0x7f84a0149090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a508a640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84a00c9ab0 0x7f84a0149090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a588b640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84a00c9ab0 0x7f84a0149090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a588b640 1 -- 172.21.15.116:0/2786258336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84a00cdd30 msgr2=0x7f84a015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a588b640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84a00cdd30 0x7f84a015d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a588b640 1 -- 172.21.15.116:0/2786258336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8490009460 con 0x7f84a0153bf0 2023-11-28T17:09:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.218+0000 7f84a508a640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84a00c9ab0 0x7f84a0149090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.219+0000 7f84a588b640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a0153bf0 0x7f84a014ecb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f849800c780 tx=0x7f849800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.219+0000 7f8495ffb640 1 -- 172.21.15.116:0/2786258336 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498003d80 con 0x7f84a0153bf0 2023-11-28T17:09:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.219+0000 7f8495ffb640 1 -- 172.21.15.116:0/2786258336 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849800ce90 con 0x7f84a0153bf0 2023-11-28T17:09:14.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.219+0000 7f8495ffb640 1 -- 172.21.15.116:0/2786258336 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84980052b0 con 0x7f84a0153bf0 2023-11-28T17:09:14.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.219+0000 7f84a608c640 1 -- 172.21.15.116:0/2786258336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a015da30 con 0x7f84a0153bf0 2023-11-28T17:09:14.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.219+0000 7f84a608c640 1 -- 172.21.15.116:0/2786258336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a015c880 con 0x7f84a0153bf0 2023-11-28T17:09:14.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.220+0000 7f84a608c640 1 -- 172.21.15.116:0/2786258336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a0072350 con 0x7f84a0153bf0 2023-11-28T17:09:14.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.221+0000 7f8495ffb640 1 -- 172.21.15.116:0/2786258336 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f8498005410 con 0x7f84a0153bf0 2023-11-28T17:09:14.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.222+0000 7f8495ffb640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f848406b2d0 0x7f848406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:14.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.222+0000 7f8495ffb640 1 -- 172.21.15.116:0/2786258336 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8498088f90 con 0x7f84a0153bf0 2023-11-28T17:09:14.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.222+0000 7f84a508a640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f848406b2d0 0x7f848406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:14.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.223+0000 7f84a508a640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f848406b2d0 0x7f848406d790 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f84a01f6940 tx=0x7f849c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:14.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.226+0000 7f8495ffb640 1 -- 172.21.15.116:0/2786258336 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8498053df0 con 0x7f84a0153bf0 2023-11-28T17:09:14.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.455+0000 7f84a608c640 1 -- 172.21.15.116:0/2786258336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84a006bb50 con 0x7f84a0153bf0 2023-11-28T17:09:14.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.456+0000 7f8495ffb640 1 -- 172.21.15.116:0/2786258336 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f8498017090 con 0x7f84a0153bf0 2023-11-28T17:09:14.457 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:14.457 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3138968863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":3138968863},{"type":"v1","addr":"172.21.15.156:6835","nonce":3138968863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/733981412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":733981412},{"type":"v1","addr":"172.21.15.116:6836","nonce":733981412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2751219705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2751219705},{"type":"v1","addr":"172.21.15.116:6837","nonce":2751219705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:09:00.673159+0000","modified":"2023-11-28T17:09:03.771842+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6834/3268354971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3268354971},{"type":"v1","addr":"172.21.15.156:6834","nonce":3268354971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T17:09:14.457 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2023-11-28T17:09:14.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.458+0000 7f84a608c640 1 -- 172.21.15.116:0/2786258336 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f848406b2d0 msgr2=0x7f848406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:14.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.458+0000 7f84a608c640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f848406b2d0 0x7f848406d790 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f84a01f6940 tx=0x7f849c008040 comp rx=0 tx=0).stop 2023-11-28T17:09:14.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.458+0000 7f84a608c640 1 -- 172.21.15.116:0/2786258336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a0153bf0 msgr2=0x7f84a014ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:14.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.458+0000 7f84a608c640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a0153bf0 0x7f84a014ecb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f849800c780 tx=0x7f849800cc40 comp rx=0 tx=0).stop 2023-11-28T17:09:14.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.458+0000 7f84a608c640 1 -- 172.21.15.116:0/2786258336 shutdown_connections 2023-11-28T17:09:14.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.458+0000 7f84a608c640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f848406b2d0 0x7f848406d790 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.458+0000 7f84a608c640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a0153bf0 0x7f84a014ecb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.458+0000 7f84a608c640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84a00cdd30 0x7f84a015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.458+0000 7f84a608c640 1 --2- 172.21.15.116:0/2786258336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84a00c9ab0 0x7f84a0149090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.458+0000 7f84a608c640 1 -- 172.21.15.116:0/2786258336 >> 172.21.15.116:0/2786258336 conn(0x7f84a00cbee0 msgr2=0x7f84a006d160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:14.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.459+0000 7f84a608c640 1 -- 172.21.15.116:0/2786258336 shutdown_connections 2023-11-28T17:09:14.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.459+0000 7f84a608c640 1 -- 172.21.15.116:0/2786258336 wait complete. 2023-11-28T17:09:14.482 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:14.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.689+0000 7fde39609640 1 Processor -- start 2023-11-28T17:09:14.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.689+0000 7fde39609640 1 -- start start 2023-11-28T17:09:14.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.691+0000 7fde39609640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde34144430 0x7fde3415f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:14.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.691+0000 7fde39609640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde3406c250 0x7fde3406c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:14.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.691+0000 7fde39609640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde34070bd0 con 0x7fde340703f0 2023-11-28T17:09:14.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.691+0000 7fde39609640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde34144810 con 0x7fde34144430 2023-11-28T17:09:14.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.691+0000 7fde39609640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde3406cc20 con 0x7fde3406c250 2023-11-28T17:09:14.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.692+0000 7fde32ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde34144430 0x7fde3415f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:14.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.692+0000 7fde32ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde34144430 0x7fde3415f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52276/0 (socket says 172.21.15.116:52276) 2023-11-28T17:09:14.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.692+0000 7fde32ffd640 1 -- 172.21.15.116:0/2783418785 learned_addr learned my addr 172.21.15.116:0/2783418785 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:14.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.692+0000 7fde337fe640 1 --2- 172.21.15.116:0/2783418785 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde3406c250 0x7fde3406c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:14.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.692+0000 7fde32ffd640 1 -- 172.21.15.116:0/2783418785 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde3406c250 msgr2=0x7fde3406c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:14.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.692+0000 7fde32ffd640 1 --2- 172.21.15.116:0/2783418785 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde3406c250 0x7fde3406c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.692+0000 7fde32ffd640 1 -- 172.21.15.116:0/2783418785 >> v1:172.21.15.116:6789/0 conn(0x7fde340703f0 legacy=0x7fde34071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:14.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.692+0000 7fde32ffd640 1 -- 172.21.15.116:0/2783418785 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde34070eb0 con 0x7fde34144430 2023-11-28T17:09:14.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.692+0000 7fde327fc640 1 -- 172.21.15.116:0/2783418785 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252389794 0 0) 0x7fde34070bd0 con 0x7fde340703f0 2023-11-28T17:09:14.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.692+0000 7fde337fe640 1 --2- 172.21.15.116:0/2783418785 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde3406c250 0x7fde3406c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:14.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.693+0000 7fde32ffd640 1 --2- 172.21.15.116:0/2783418785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde34144430 0x7fde3415f7b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fde24009870 tx=0x7fde2402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d3a16efacbd27f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:14.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.694+0000 7fde327fc640 1 -- 172.21.15.116:0/2783418785 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde24030840 con 0x7fde34144430 2023-11-28T17:09:14.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.694+0000 7fde327fc640 1 -- 172.21.15.116:0/2783418785 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde240309a0 con 0x7fde34144430 2023-11-28T17:09:14.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.694+0000 7fde327fc640 1 -- 172.21.15.116:0/2783418785 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde24030d30 con 0x7fde34144430 2023-11-28T17:09:14.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.694+0000 7fde39609640 1 -- 172.21.15.116:0/2783418785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde34144430 msgr2=0x7fde3415f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:14.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.694+0000 7fde39609640 1 --2- 172.21.15.116:0/2783418785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde34144430 0x7fde3415f7b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fde24009870 tx=0x7fde2402b980 comp rx=0 tx=0).stop 2023-11-28T17:09:14.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.694+0000 7fde39609640 1 -- 172.21.15.116:0/2783418785 shutdown_connections 2023-11-28T17:09:14.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.694+0000 7fde39609640 1 --2- 172.21.15.116:0/2783418785 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde3406c250 0x7fde3406c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.694+0000 7fde39609640 1 --2- 172.21.15.116:0/2783418785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde34144430 0x7fde3415f7b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.694+0000 7fde39609640 1 -- 172.21.15.116:0/2783418785 >> 172.21.15.116:0/2783418785 conn(0x7fde3406a4c0 msgr2=0x7fde3406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:14.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.694+0000 7fde39609640 1 -- 172.21.15.116:0/2783418785 shutdown_connections 2023-11-28T17:09:14.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.694+0000 7fde39609640 1 -- 172.21.15.116:0/2783418785 wait complete. 2023-11-28T17:09:14.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.696+0000 7fde39609640 1 Processor -- start 2023-11-28T17:09:14.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.696+0000 7fde39609640 1 -- start start 2023-11-28T17:09:14.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.697+0000 7fde39609640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde3406c250 0x7fde341e8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:14.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.697+0000 7fde39609640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde340703f0 0x7fde340aa640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:14.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.697+0000 7fde39609640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde34144430 0x7fde341e6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:14.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.697+0000 7fde39609640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde341e5d00 con 0x7fde34144430 2023-11-28T17:09:14.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.697+0000 7fde39609640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde341e5fe0 con 0x7fde3406c250 2023-11-28T17:09:14.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.697+0000 7fde39609640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde341e62c0 con 0x7fde340703f0 2023-11-28T17:09:14.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.697+0000 7fde337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde34144430 0x7fde341e6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:14.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.697+0000 7fde337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde34144430 0x7fde341e6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52288/0 (socket says 172.21.15.116:52288) 2023-11-28T17:09:14.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.697+0000 7fde337fe640 1 -- 172.21.15.116:0/3552105543 learned_addr learned my addr 172.21.15.116:0/3552105543 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:14.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.698+0000 7fde337fe640 1 -- 172.21.15.116:0/3552105543 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde340703f0 msgr2=0x7fde340aa640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:14.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.698+0000 7fde337fe640 1 --2- 172.21.15.116:0/3552105543 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde340703f0 0x7fde340aa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.698+0000 7fde337fe640 1 -- 172.21.15.116:0/3552105543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde3406c250 msgr2=0x7fde341e8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:14.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.698+0000 7fde337fe640 1 --2- 172.21.15.116:0/3552105543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde3406c250 0x7fde341e8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.698+0000 7fde337fe640 1 -- 172.21.15.116:0/3552105543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde240094d0 con 0x7fde34144430 2023-11-28T17:09:14.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.698+0000 7fde337fe640 1 --2- 172.21.15.116:0/3552105543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde34144430 0x7fde341e6da0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fde2c00cd50 tx=0x7fde2c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:14.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.698+0000 7fde2bfff640 1 -- 172.21.15.116:0/3552105543 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c00f680 con 0x7fde34144430 2023-11-28T17:09:14.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.698+0000 7fde2bfff640 1 -- 172.21.15.116:0/3552105543 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde2c00f800 con 0x7fde34144430 2023-11-28T17:09:14.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.698+0000 7fde2bfff640 1 -- 172.21.15.116:0/3552105543 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c00fbb0 con 0x7fde34144430 2023-11-28T17:09:14.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.698+0000 7fde39609640 1 -- 172.21.15.116:0/3552105543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde341e65a0 con 0x7fde34144430 2023-11-28T17:09:14.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.698+0000 7fde39609640 1 -- 172.21.15.116:0/3552105543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde341e90a0 con 0x7fde34144430 2023-11-28T17:09:14.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.699+0000 7fde39609640 1 -- 172.21.15.116:0/3552105543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde341e6ac0 con 0x7fde34144430 2023-11-28T17:09:14.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.700+0000 7fde2bfff640 1 -- 172.21.15.116:0/3552105543 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fde2c00b040 con 0x7fde34144430 2023-11-28T17:09:14.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.701+0000 7fde2bfff640 1 --2- 172.21.15.116:0/3552105543 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fde1406b2a0 0x7fde1406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:14.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.701+0000 7fde2bfff640 1 -- 172.21.15.116:0/3552105543 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde2c08a580 con 0x7fde34144430 2023-11-28T17:09:14.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.702+0000 7fde32ffd640 1 --2- 172.21.15.116:0/3552105543 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fde1406b2a0 0x7fde1406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:14.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.702+0000 7fde32ffd640 1 --2- 172.21.15.116:0/3552105543 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fde1406b2a0 0x7fde1406d760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fde24036000 tx=0x7fde2403b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:14.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.706+0000 7fde2bfff640 1 -- 172.21.15.116:0/3552105543 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde2c0553e0 con 0x7fde34144430 2023-11-28T17:09:14.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.900+0000 7fde39609640 1 -- 172.21.15.116:0/3552105543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde340693d0 con 0x7fde34144430 2023-11-28T17:09:14.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.901+0000 7fde2bfff640 1 -- 172.21.15.116:0/3552105543 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16521 (secure 0 0 0) 0x7fde2c058a00 con 0x7fde34144430 2023-11-28T17:09:14.903 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:14.903 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:01.970957+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:14.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.904+0000 7fde39609640 1 -- 172.21.15.116:0/3552105543 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fde1406b2a0 msgr2=0x7fde1406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:14.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.904+0000 7fde39609640 1 --2- 172.21.15.116:0/3552105543 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fde1406b2a0 0x7fde1406d760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fde24036000 tx=0x7fde2403b040 comp rx=0 tx=0).stop 2023-11-28T17:09:14.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.904+0000 7fde39609640 1 -- 172.21.15.116:0/3552105543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde34144430 msgr2=0x7fde341e6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:14.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.904+0000 7fde39609640 1 --2- 172.21.15.116:0/3552105543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde34144430 0x7fde341e6da0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fde2c00cd50 tx=0x7fde2c00a650 comp rx=0 tx=0).stop 2023-11-28T17:09:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.904+0000 7fde39609640 1 -- 172.21.15.116:0/3552105543 shutdown_connections 2023-11-28T17:09:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.904+0000 7fde39609640 1 --2- 172.21.15.116:0/3552105543 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fde1406b2a0 0x7fde1406d760 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.904+0000 7fde39609640 1 --2- 172.21.15.116:0/3552105543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde34144430 0x7fde341e6da0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.904+0000 7fde39609640 1 --2- 172.21.15.116:0/3552105543 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde340703f0 0x7fde340aa640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.904+0000 7fde39609640 1 --2- 172.21.15.116:0/3552105543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde3406c250 0x7fde341e8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.904+0000 7fde39609640 1 -- 172.21.15.116:0/3552105543 >> 172.21.15.116:0/3552105543 conn(0x7fde3406a4c0 msgr2=0x7fde34143660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.904+0000 7fde39609640 1 -- 172.21.15.116:0/3552105543 shutdown_connections 2023-11-28T17:09:14.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:14.904+0000 7fde39609640 1 -- 172.21.15.116:0/3552105543 wait complete. 2023-11-28T17:09:14.925 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:09:14.926 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:09:15.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.140+0000 7fdfc989b640 1 Processor -- start 2023-11-28T17:09:15.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.140+0000 7fdfc989b640 1 -- start start 2023-11-28T17:09:15.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.142+0000 7fdfc989b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfc4072430 0x7fdfc413fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.142+0000 7fdfc989b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc413dd10 con 0x7fdfc413cee0 2023-11-28T17:09:15.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.142+0000 7fdfc989b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc413d2c0 con 0x7fdfc413f640 2023-11-28T17:09:15.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.142+0000 7fdfc989b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc41400e0 con 0x7fdfc4072430 2023-11-28T17:09:15.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.143+0000 7fdfc2ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdfc413cee0 0x7fdfc413d600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54734/0 (socket says 172.21.15.116:54734) 2023-11-28T17:09:15.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.143+0000 7fdfc2ffd640 1 -- 172.21.15.116:0/1275489924 learned_addr learned my addr 172.21.15.116:0/1275489924 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:15.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.143+0000 7fdfc37fe640 1 --2- 172.21.15.116:0/1275489924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfc4072430 0x7fdfc413fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:15.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.143+0000 7fdfc1ffb640 1 -- 172.21.15.116:0/1275489924 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054642272 0 0) 0x7fdfc413dd10 con 0x7fdfc413cee0 2023-11-28T17:09:15.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.143+0000 7fdfc1ffb640 1 -- 172.21.15.116:0/1275489924 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb0003520 con 0x7fdfc413cee0 2023-11-28T17:09:15.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.143+0000 7fdfc1ffb640 1 -- 172.21.15.116:0/1275489924 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1227778138 0 0) 0x7fdfb0003520 con 0x7fdfc413cee0 2023-11-28T17:09:15.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.143+0000 7fdfc1ffb640 1 -- 172.21.15.116:0/1275489924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfc4072430 msgr2=0x7fdfc413fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:15.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.143+0000 7fdfc1ffb640 1 --2- 172.21.15.116:0/1275489924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfc4072430 0x7fdfc413fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.143+0000 7fdfc1ffb640 1 -- 172.21.15.116:0/1275489924 >> v1:172.21.15.156:6789/0 conn(0x7fdfc413f640 legacy=0x7fdfc4071d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:15.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.143+0000 7fdfc1ffb640 1 -- 172.21.15.116:0/1275489924 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc41403c0 con 0x7fdfc413cee0 2023-11-28T17:09:15.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.143+0000 7fdfc1ffb640 1 -- 172.21.15.116:0/1275489924 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fdfb40019f0 con 0x7fdfc413cee0 2023-11-28T17:09:15.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.143+0000 7fdfc1ffb640 1 -- 172.21.15.116:0/1275489924 >> v1:172.21.15.116:6789/0 conn(0x7fdfc413cee0 legacy=0x7fdfc413d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:15.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.143+0000 7fdfc1ffb640 1 --2- 172.21.15.116:0/1275489924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfb00039a0 0x7fdfb0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.144+0000 7fdfc1ffb640 1 --2- 172.21.15.116:0/1275489924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfc4072430 0x7fdfb00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.144+0000 7fdfc1ffb640 1 --2- 172.21.15.116:0/1275489924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb0004d50 0x7fdfb00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.144+0000 7fdfc1ffb640 1 -- 172.21.15.116:0/1275489924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc41400e0 con 0x7fdfb0004d50 2023-11-28T17:09:15.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.144+0000 7fdfc1ffb640 1 -- 172.21.15.116:0/1275489924 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb000de10 con 0x7fdfb00039a0 2023-11-28T17:09:15.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.144+0000 7fdfc1ffb640 1 -- 172.21.15.116:0/1275489924 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb000dfd0 con 0x7fdfc4072430 2023-11-28T17:09:15.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.144+0000 7fdfc37fe640 1 --2- 172.21.15.116:0/1275489924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb0004d50 0x7fdfb00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:15.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.144+0000 7fdfc27fc640 1 --2- 172.21.15.116:0/1275489924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfc4072430 0x7fdfb00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:15.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.144+0000 7fdfc2ffd640 1 --2- 172.21.15.116:0/1275489924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfb00039a0 0x7fdfb0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:15.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.144+0000 7fdfc37fe640 1 -- 172.21.15.116:0/1275489924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfc4072430 msgr2=0x7fdfb00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:15.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.144+0000 7fdfc37fe640 1 --2- 172.21.15.116:0/1275489924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfc4072430 0x7fdfb00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.144+0000 7fdfc37fe640 1 -- 172.21.15.116:0/1275489924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfb00039a0 msgr2=0x7fdfb0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:15.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.144+0000 7fdfc37fe640 1 --2- 172.21.15.116:0/1275489924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfb00039a0 0x7fdfb0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.144+0000 7fdfc37fe640 1 -- 172.21.15.116:0/1275489924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb0003520 con 0x7fdfb0004d50 2023-11-28T17:09:15.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.144+0000 7fdfc27fc640 1 --2- 172.21.15.116:0/1275489924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfc4072430 0x7fdfb00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:15.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.145+0000 7fdfc37fe640 1 --2- 172.21.15.116:0/1275489924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb0004d50 0x7fdfb00051d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00ba00 tx=0x7fdfb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff435d92425edbd2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:15.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.145+0000 7fdfc1ffb640 1 -- 172.21.15.116:0/1275489924 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc0325f0 con 0x7fdfb0004d50 2023-11-28T17:09:15.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.145+0000 7fdfc1ffb640 1 -- 172.21.15.116:0/1275489924 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfbc032750 con 0x7fdfb0004d50 2023-11-28T17:09:15.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.145+0000 7fdfc1ffb640 1 -- 172.21.15.116:0/1275489924 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc032ae0 con 0x7fdfb0004d50 2023-11-28T17:09:15.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.146+0000 7fdfc989b640 1 -- 172.21.15.116:0/1275489924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb0004d50 msgr2=0x7fdfb00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:15.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.146+0000 7fdfc989b640 1 --2- 172.21.15.116:0/1275489924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb0004d50 0x7fdfb00051d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00ba00 tx=0x7fdfb00057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.146+0000 7fdfc37fe640 1 -- 172.21.15.116:0/1275489924 reap_dead start 2023-11-28T17:09:15.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.146+0000 7fdfc989b640 1 -- 172.21.15.116:0/1275489924 shutdown_connections 2023-11-28T17:09:15.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.146+0000 7fdfc989b640 1 -- 172.21.15.116:0/1275489924 >> 172.21.15.116:0/1275489924 conn(0x7fdfc40b0cf0 msgr2=0x7fdfc40b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:15.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.146+0000 7fdfc989b640 1 -- 172.21.15.116:0/1275489924 shutdown_connections 2023-11-28T17:09:15.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.146+0000 7fdfc989b640 1 -- 172.21.15.116:0/1275489924 wait complete. 2023-11-28T17:09:15.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.148+0000 7fdfc989b640 1 Processor -- start 2023-11-28T17:09:15.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.148+0000 7fdfc989b640 1 -- start start 2023-11-28T17:09:15.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.149+0000 7fdfc989b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb0004d50 0x7fdfc41f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.149+0000 7fdfc989b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfc40a8900 0x7fdfc41edf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.149+0000 7fdfc989b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfc413b650 0x7fdfc413bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.149+0000 7fdfc989b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc413c010 con 0x7fdfb0004d50 2023-11-28T17:09:15.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.149+0000 7fdfc989b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc415aab0 con 0x7fdfc413b650 2023-11-28T17:09:15.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.149+0000 7fdfc989b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40a90e0 con 0x7fdfc40a8900 2023-11-28T17:09:15.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.149+0000 7fdfc2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb0004d50 0x7fdfc41f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:15.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.149+0000 7fdfc2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb0004d50 0x7fdfc41f27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52292/0 (socket says 172.21.15.116:52292) 2023-11-28T17:09:15.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.149+0000 7fdfc2ffd640 1 -- 172.21.15.116:0/2534225153 learned_addr learned my addr 172.21.15.116:0/2534225153 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:15.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.150+0000 7fdfc2ffd640 1 -- 172.21.15.116:0/2534225153 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfc40a8900 msgr2=0x7fdfc41edf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:15.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.150+0000 7fdfc2ffd640 1 --2- 172.21.15.116:0/2534225153 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfc40a8900 0x7fdfc41edf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.150+0000 7fdfc2ffd640 1 -- 172.21.15.116:0/2534225153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfc413b650 msgr2=0x7fdfc413bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:15.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.150+0000 7fdfc2ffd640 1 --2- 172.21.15.116:0/2534225153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfc413b650 0x7fdfc413bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.150+0000 7fdfc2ffd640 1 -- 172.21.15.116:0/2534225153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc40a93c0 con 0x7fdfb0004d50 2023-11-28T17:09:15.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.150+0000 7fdfc2ffd640 1 --2- 172.21.15.116:0/2534225153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb0004d50 0x7fdfc41f27b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdfb40079e0 tx=0x7fdfb400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:15.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.150+0000 7fdfbbfff640 1 -- 172.21.15.116:0/2534225153 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb400edd0 con 0x7fdfb0004d50 2023-11-28T17:09:15.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.150+0000 7fdfbbfff640 1 -- 172.21.15.116:0/2534225153 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb4013420 con 0x7fdfb0004d50 2023-11-28T17:09:15.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.150+0000 7fdfbbfff640 1 -- 172.21.15.116:0/2534225153 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfb40137d0 con 0x7fdfb0004d50 2023-11-28T17:09:15.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.150+0000 7fdfc989b640 1 -- 172.21.15.116:0/2534225153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc415ad00 con 0x7fdfb0004d50 2023-11-28T17:09:15.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.150+0000 7fdfc989b640 1 -- 172.21.15.116:0/2534225153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc41ee490 con 0x7fdfb0004d50 2023-11-28T17:09:15.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.151+0000 7fdfc989b640 1 -- 172.21.15.116:0/2534225153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc415b220 con 0x7fdfb0004d50 2023-11-28T17:09:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.152+0000 7fdfbbfff640 1 -- 172.21.15.116:0/2534225153 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fdfb4013930 con 0x7fdfb0004d50 2023-11-28T17:09:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.153+0000 7fdfbbfff640 1 --2- 172.21.15.116:0/2534225153 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdfa406b2d0 0x7fdfa406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.153+0000 7fdfbbfff640 1 -- 172.21.15.116:0/2534225153 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdfb408a390 con 0x7fdfb0004d50 2023-11-28T17:09:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.153+0000 7fdfc27fc640 1 --2- 172.21.15.116:0/2534225153 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdfa406b2d0 0x7fdfa406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:15.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.154+0000 7fdfc27fc640 1 --2- 172.21.15.116:0/2534225153 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdfa406b2d0 0x7fdfa406d790 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdfc406d080 tx=0x7fdfac009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:15.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.157+0000 7fdfbbfff640 1 -- 172.21.15.116:0/2534225153 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdfc415b220 con 0x7fdfb0004d50 2023-11-28T17:09:15.348 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.347+0000 7fdfc989b640 1 -- 172.21.15.116:0/2534225153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdfc4069360 con 0x7fdfb0004d50 2023-11-28T17:09:15.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.348+0000 7fdfbbfff640 1 -- 172.21.15.116:0/2534225153 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdfb4058bf0 con 0x7fdfb0004d50 2023-11-28T17:09:15.349 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:09:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.350+0000 7fdfc989b640 1 -- 172.21.15.116:0/2534225153 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdfa406b2d0 msgr2=0x7fdfa406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.350+0000 7fdfc989b640 1 --2- 172.21.15.116:0/2534225153 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdfa406b2d0 0x7fdfa406d790 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdfc406d080 tx=0x7fdfac009040 comp rx=0 tx=0).stop 2023-11-28T17:09:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.351+0000 7fdfc989b640 1 -- 172.21.15.116:0/2534225153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb0004d50 msgr2=0x7fdfc41f27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.351+0000 7fdfc989b640 1 --2- 172.21.15.116:0/2534225153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb0004d50 0x7fdfc41f27b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdfb40079e0 tx=0x7fdfb400e530 comp rx=0 tx=0).stop 2023-11-28T17:09:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.351+0000 7fdfc989b640 1 -- 172.21.15.116:0/2534225153 shutdown_connections 2023-11-28T17:09:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.351+0000 7fdfc989b640 1 --2- 172.21.15.116:0/2534225153 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdfa406b2d0 0x7fdfa406d790 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.351+0000 7fdfc989b640 1 --2- 172.21.15.116:0/2534225153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfc413b650 0x7fdfc413bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.351+0000 7fdfc989b640 1 --2- 172.21.15.116:0/2534225153 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfc40a8900 0x7fdfc41edf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.351+0000 7fdfc989b640 1 --2- 172.21.15.116:0/2534225153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfb0004d50 0x7fdfc41f27b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.351+0000 7fdfc989b640 1 -- 172.21.15.116:0/2534225153 >> 172.21.15.116:0/2534225153 conn(0x7fdfc40b0cf0 msgr2=0x7fdfc415cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.351+0000 7fdfc989b640 1 -- 172.21.15.116:0/2534225153 shutdown_connections 2023-11-28T17:09:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.351+0000 7fdfc989b640 1 -- 172.21.15.116:0/2534225153 wait complete. 2023-11-28T17:09:15.372 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:09:15.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.588+0000 7f89d49b0640 1 Processor -- start 2023-11-28T17:09:15.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.588+0000 7f89d49b0640 1 -- start start 2023-11-28T17:09:15.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.590+0000 7f89d49b0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d0072250 0x7f89d015ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.590+0000 7f89d49b0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89d00cd320 con 0x7f89d015b0c0 2023-11-28T17:09:15.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.590+0000 7f89d49b0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89d00cc710 con 0x7f89d00ce940 2023-11-28T17:09:15.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.590+0000 7f89d49b0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d00cedf0 con 0x7f89d0072250 2023-11-28T17:09:15.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.590+0000 7f89ce59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f89d015b0c0 0x7f89d0071b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54746/0 (socket says 172.21.15.116:54746) 2023-11-28T17:09:15.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.590+0000 7f89ce59c640 1 -- 172.21.15.116:0/1657641656 learned_addr learned my addr 172.21.15.116:0/1657641656 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:15.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.590+0000 7f89cdd9b640 1 --2- 172.21.15.116:0/1657641656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d0072250 0x7f89d015ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:15.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 -- 172.21.15.116:0/1657641656 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157818118 0 0) 0x7f89d00cd320 con 0x7f89d015b0c0 2023-11-28T17:09:15.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 -- 172.21.15.116:0/1657641656 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b8003520 con 0x7f89d015b0c0 2023-11-28T17:09:15.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 -- 172.21.15.116:0/1657641656 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3384596232 0 0) 0x7f89b8003520 con 0x7f89d015b0c0 2023-11-28T17:09:15.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 -- 172.21.15.116:0/1657641656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d0072250 msgr2=0x7f89d015ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:15.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 --2- 172.21.15.116:0/1657641656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d0072250 0x7f89d015ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 -- 172.21.15.116:0/1657641656 >> v1:172.21.15.156:6789/0 conn(0x7f89d00ce940 legacy=0x7f89d015b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:15.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 -- 172.21.15.116:0/1657641656 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d00cc9f0 con 0x7f89d015b0c0 2023-11-28T17:09:15.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 -- 172.21.15.116:0/1657641656 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f89c0001670 con 0x7f89d015b0c0 2023-11-28T17:09:15.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 -- 172.21.15.116:0/1657641656 >> v1:172.21.15.116:6789/0 conn(0x7f89d015b0c0 legacy=0x7f89d0071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:15.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 --2- 172.21.15.116:0/1657641656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d0072250 0x7f89b80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 --2- 172.21.15.116:0/1657641656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89b8003f20 0x7f89b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 --2- 172.21.15.116:0/1657641656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89b8004d00 0x7f89b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 -- 172.21.15.116:0/1657641656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b800bb10 con 0x7f89b8003f20 2023-11-28T17:09:15.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 -- 172.21.15.116:0/1657641656 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b800bc50 con 0x7f89b8004d00 2023-11-28T17:09:15.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cd59a640 1 -- 172.21.15.116:0/1657641656 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b800be10 con 0x7f89d0072250 2023-11-28T17:09:15.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.591+0000 7f89cdd9b640 1 --2- 172.21.15.116:0/1657641656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89b8003f20 0x7f89b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:15.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.592+0000 7f89ce59c640 1 --2- 172.21.15.116:0/1657641656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d0072250 0x7f89b80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:15.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.592+0000 7f89cdd9b640 1 -- 172.21.15.116:0/1657641656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d0072250 msgr2=0x7f89b80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:15.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.592+0000 7f89ced9d640 1 --2- 172.21.15.116:0/1657641656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89b8004d00 0x7f89b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:15.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.592+0000 7f89cdd9b640 1 --2- 172.21.15.116:0/1657641656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d0072250 0x7f89b80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.592+0000 7f89cdd9b640 1 -- 172.21.15.116:0/1657641656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89b8004d00 msgr2=0x7f89b8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:15.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.592+0000 7f89cdd9b640 1 --2- 172.21.15.116:0/1657641656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89b8004d00 0x7f89b8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.592+0000 7f89cdd9b640 1 -- 172.21.15.116:0/1657641656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b8003520 con 0x7f89b8003f20 2023-11-28T17:09:15.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.592+0000 7f89ced9d640 1 --2- 172.21.15.116:0/1657641656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89b8004d00 0x7f89b8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:15.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.592+0000 7f89ce59c640 1 --2- 172.21.15.116:0/1657641656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d0072250 0x7f89b80039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:15.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.593+0000 7f89cdd9b640 1 --2- 172.21.15.116:0/1657641656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89b8003f20 0x7f89b80047c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f89c4009ac0 tx=0x7f89b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e9f309ed00b3cc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:15.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.593+0000 7f89cd59a640 1 -- 172.21.15.116:0/1657641656 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c402c6b0 con 0x7f89b8003f20 2023-11-28T17:09:15.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.593+0000 7f89cd59a640 1 -- 172.21.15.116:0/1657641656 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c402c810 con 0x7f89b8003f20 2023-11-28T17:09:15.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.593+0000 7f89cd59a640 1 -- 172.21.15.116:0/1657641656 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c402cba0 con 0x7f89b8003f20 2023-11-28T17:09:15.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.594+0000 7f89d49b0640 1 -- 172.21.15.116:0/1657641656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89b8003f20 msgr2=0x7f89b80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:15.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.594+0000 7f89d49b0640 1 --2- 172.21.15.116:0/1657641656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89b8003f20 0x7f89b80047c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f89c4009ac0 tx=0x7f89b80057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.594+0000 7f89ced9d640 1 -- 172.21.15.116:0/1657641656 reap_dead start 2023-11-28T17:09:15.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.594+0000 7f89d49b0640 1 -- 172.21.15.116:0/1657641656 shutdown_connections 2023-11-28T17:09:15.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.594+0000 7f89d49b0640 1 -- 172.21.15.116:0/1657641656 >> 172.21.15.116:0/1657641656 conn(0x7f89d00b0cf0 msgr2=0x7f89d00b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:15.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.594+0000 7f89d49b0640 1 -- 172.21.15.116:0/1657641656 shutdown_connections 2023-11-28T17:09:15.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.594+0000 7f89d49b0640 1 -- 172.21.15.116:0/1657641656 wait complete. 2023-11-28T17:09:15.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.596+0000 7f89d49b0640 1 Processor -- start 2023-11-28T17:09:15.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.596+0000 7f89d49b0640 1 -- start start 2023-11-28T17:09:15.607 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89d49b0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89b8003f20 0x7f89d0145c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89d49b0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d01512b0 0x7f89d0151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89d49b0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d014f2d0 0x7f89d014f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89d49b0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d0151c70 con 0x7f89d01512b0 2023-11-28T17:09:15.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89d49b0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d015b7b0 con 0x7f89b8003f20 2023-11-28T17:09:15.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89d49b0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89d014ffe0 con 0x7f89d014f2d0 2023-11-28T17:09:15.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89cdd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d01512b0 0x7f89d0151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:15.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89cdd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d01512b0 0x7f89d0151730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52312/0 (socket says 172.21.15.116:52312) 2023-11-28T17:09:15.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89cdd9b640 1 -- 172.21.15.116:0/2168389857 learned_addr learned my addr 172.21.15.116:0/2168389857 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:15.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89ced9d640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d014f2d0 0x7f89d014f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:15.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89ce59c640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89b8003f20 0x7f89d0145c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:15.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89cdd9b640 1 -- 172.21.15.116:0/2168389857 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d014f2d0 msgr2=0x7f89d014f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:15.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89cdd9b640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d014f2d0 0x7f89d014f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89cdd9b640 1 -- 172.21.15.116:0/2168389857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89b8003f20 msgr2=0x7f89d0145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:15.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89cdd9b640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89b8003f20 0x7f89d0145c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89cdd9b640 1 -- 172.21.15.116:0/2168389857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d015b620 con 0x7f89d01512b0 2023-11-28T17:09:15.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.597+0000 7f89ced9d640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d014f2d0 0x7f89d014f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:15.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.598+0000 7f89cdd9b640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d01512b0 0x7f89d0151730 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f89c4037040 tx=0x7f89c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:15.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.598+0000 7f89beffd640 1 -- 172.21.15.116:0/2168389857 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c4003810 con 0x7f89d01512b0 2023-11-28T17:09:15.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.598+0000 7f89beffd640 1 -- 172.21.15.116:0/2168389857 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c4003970 con 0x7f89d01512b0 2023-11-28T17:09:15.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.598+0000 7f89d49b0640 1 -- 172.21.15.116:0/2168389857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89d01502c0 con 0x7f89d01512b0 2023-11-28T17:09:15.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.598+0000 7f89beffd640 1 -- 172.21.15.116:0/2168389857 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c4003d00 con 0x7f89d01512b0 2023-11-28T17:09:15.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.598+0000 7f89d49b0640 1 -- 172.21.15.116:0/2168389857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89d015ba50 con 0x7f89d01512b0 2023-11-28T17:09:15.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.599+0000 7f89d49b0640 1 -- 172.21.15.116:0/2168389857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d0069360 con 0x7f89d01512b0 2023-11-28T17:09:15.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.600+0000 7f89beffd640 1 -- 172.21.15.116:0/2168389857 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f89c4034440 con 0x7f89d01512b0 2023-11-28T17:09:15.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.601+0000 7f89beffd640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f89b006b2d0 0x7f89b006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:15.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.601+0000 7f89beffd640 1 -- 172.21.15.116:0/2168389857 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89c40abf10 con 0x7f89d01512b0 2023-11-28T17:09:15.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.601+0000 7f89ce59c640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f89b006b2d0 0x7f89b006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:15.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.602+0000 7f89ce59c640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f89b006b2d0 0x7f89b006d790 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f89c0001eb0 tx=0x7f89c0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:15.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.605+0000 7f89beffd640 1 -- 172.21.15.116:0/2168389857 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f89c4076d90 con 0x7f89d01512b0 2023-11-28T17:09:15.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.794+0000 7f89d49b0640 1 -- 172.21.15.116:0/2168389857 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f89d00b1990 con 0x7f89d01512b0 2023-11-28T17:09:15.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.795+0000 7f89beffd640 1 -- 172.21.15.116:0/2168389857 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f89c404c020 con 0x7f89d01512b0 2023-11-28T17:09:15.796 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:09:15.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.797+0000 7f89d49b0640 1 -- 172.21.15.116:0/2168389857 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f89b006b2d0 msgr2=0x7f89b006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:15.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.797+0000 7f89d49b0640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f89b006b2d0 0x7f89b006d790 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f89c0001eb0 tx=0x7f89c0009040 comp rx=0 tx=0).stop 2023-11-28T17:09:15.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.797+0000 7f89d49b0640 1 -- 172.21.15.116:0/2168389857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d01512b0 msgr2=0x7f89d0151730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:15.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.797+0000 7f89d49b0640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d01512b0 0x7f89d0151730 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f89c4037040 tx=0x7f89c4004730 comp rx=0 tx=0).stop 2023-11-28T17:09:15.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.797+0000 7f89d49b0640 1 -- 172.21.15.116:0/2168389857 shutdown_connections 2023-11-28T17:09:15.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.797+0000 7f89d49b0640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f89b006b2d0 0x7f89b006d790 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.797+0000 7f89d49b0640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89d014f2d0 0x7f89d014f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.797+0000 7f89d49b0640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89d01512b0 0x7f89d0151730 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.797+0000 7f89d49b0640 1 --2- 172.21.15.116:0/2168389857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89b8003f20 0x7f89d0145c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:15.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.797+0000 7f89d49b0640 1 -- 172.21.15.116:0/2168389857 >> 172.21.15.116:0/2168389857 conn(0x7f89d00b0cf0 msgr2=0x7f89d015f300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:15.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.797+0000 7f89d49b0640 1 -- 172.21.15.116:0/2168389857 shutdown_connections 2023-11-28T17:09:15.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:15.798+0000 7f89d49b0640 1 -- 172.21.15.116:0/2168389857 wait complete. 2023-11-28T17:09:15.818 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:09:16.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.036+0000 7f329baa6640 1 Processor -- start 2023-11-28T17:09:16.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.036+0000 7f329baa6640 1 -- start start 2023-11-28T17:09:16.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f329baa6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f329413cee0 0x7f329413d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f329baa6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f329413f640 0x7f329413fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f329baa6640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3294071d20 0x7f3294072170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f329baa6640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32940726b0 con 0x7f329413cee0 2023-11-28T17:09:16.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f329baa6640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f329413d2c0 con 0x7f3294071d20 2023-11-28T17:09:16.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f329baa6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32941400e0 con 0x7f329413f640 2023-11-28T17:09:16.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f3299842640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f329413cee0 0x7f329413d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f3299842640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f329413cee0 0x7f329413d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52314/0 (socket says 172.21.15.116:52314) 2023-11-28T17:09:16.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f3299842640 1 -- 172.21.15.116:0/1044037638 learned_addr learned my addr 172.21.15.116:0/1044037638 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:16.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f3299842640 1 -- 172.21.15.116:0/1044037638 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f329413f640 msgr2=0x7f329413fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:16.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f3299041640 1 --2- 172.21.15.116:0/1044037638 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f329413f640 0x7f329413fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f329a043640 1 --2- 172.21.15.116:0/1044037638 >> v2:172.21.15.156:3300/0 conn(0x7f3294071d20 0x7f3294072170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f3299842640 1 --2- 172.21.15.116:0/1044037638 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f329413f640 0x7f329413fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f3299842640 1 -- 172.21.15.116:0/1044037638 >> v2:172.21.15.156:3300/0 conn(0x7f3294071d20 msgr2=0x7f3294072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:16.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f3299842640 1 --2- 172.21.15.116:0/1044037638 >> v2:172.21.15.156:3300/0 conn(0x7f3294071d20 0x7f3294072170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.038+0000 7f3299842640 1 -- 172.21.15.116:0/1044037638 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32941403c0 con 0x7f329413cee0 2023-11-28T17:09:16.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.039+0000 7f3299842640 1 --2- 172.21.15.116:0/1044037638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f329413cee0 0x7f329413d600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f32940cc910 tx=0x7f329002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be9522b167e71c5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:16.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.040+0000 7f3298840640 1 -- 172.21.15.116:0/1044037638 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3290031740 con 0x7f329413cee0 2023-11-28T17:09:16.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.040+0000 7f3298840640 1 -- 172.21.15.116:0/1044037638 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32900318a0 con 0x7f329413cee0 2023-11-28T17:09:16.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.040+0000 7f3298840640 1 -- 172.21.15.116:0/1044037638 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3290031c30 con 0x7f329413cee0 2023-11-28T17:09:16.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.040+0000 7f329baa6640 1 -- 172.21.15.116:0/1044037638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f329413cee0 msgr2=0x7f329413d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.040+0000 7f329baa6640 1 --2- 172.21.15.116:0/1044037638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f329413cee0 0x7f329413d600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f32940cc910 tx=0x7f329002b7e0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.041+0000 7f329baa6640 1 -- 172.21.15.116:0/1044037638 shutdown_connections 2023-11-28T17:09:16.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.041+0000 7f329baa6640 1 --2- 172.21.15.116:0/1044037638 >> v2:172.21.15.156:3300/0 conn(0x7f3294071d20 0x7f3294072170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.041+0000 7f329baa6640 1 --2- 172.21.15.116:0/1044037638 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f329413f640 0x7f329413fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.041+0000 7f329baa6640 1 --2- 172.21.15.116:0/1044037638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f329413cee0 0x7f329413d600 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.041+0000 7f329baa6640 1 -- 172.21.15.116:0/1044037638 >> 172.21.15.116:0/1044037638 conn(0x7f32940b0cf0 msgr2=0x7f32940b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:16.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.041+0000 7f329baa6640 1 -- 172.21.15.116:0/1044037638 shutdown_connections 2023-11-28T17:09:16.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.041+0000 7f329baa6640 1 -- 172.21.15.116:0/1044037638 wait complete. 2023-11-28T17:09:16.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.043+0000 7f329baa6640 1 Processor -- start 2023-11-28T17:09:16.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.043+0000 7f329baa6640 1 -- start start 2023-11-28T17:09:16.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.044+0000 7f329baa6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3294071d20 0x7f329415c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.044+0000 7f329baa6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f329413cee0 0x7f3294137bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.044+0000 7f329baa6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f329413f640 0x7f32941f0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.044+0000 7f329baa6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32941414b0 con 0x7f329413f640 2023-11-28T17:09:16.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.044+0000 7f329baa6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3294141790 con 0x7f329413cee0 2023-11-28T17:09:16.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.044+0000 7f329baa6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3294141a70 con 0x7f3294071d20 2023-11-28T17:09:16.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.045+0000 7f329a043640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f329413f640 0x7f32941f0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.045+0000 7f329a043640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f329413f640 0x7f32941f0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52316/0 (socket says 172.21.15.116:52316) 2023-11-28T17:09:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.045+0000 7f329a043640 1 -- 172.21.15.116:0/46948884 learned_addr learned my addr 172.21.15.116:0/46948884 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.045+0000 7f3299041640 1 --2- 172.21.15.116:0/46948884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f329413cee0 0x7f3294137bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.045+0000 7f3299842640 1 --2- 172.21.15.116:0/46948884 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3294071d20 0x7f329415c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.045+0000 7f329a043640 1 -- 172.21.15.116:0/46948884 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3294071d20 msgr2=0x7f329415c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.045+0000 7f329a043640 1 --2- 172.21.15.116:0/46948884 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3294071d20 0x7f329415c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.045+0000 7f329a043640 1 -- 172.21.15.116:0/46948884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f329413cee0 msgr2=0x7f3294137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.045+0000 7f329a043640 1 --2- 172.21.15.116:0/46948884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f329413cee0 0x7f3294137bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.045+0000 7f329a043640 1 -- 172.21.15.116:0/46948884 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32900094d0 con 0x7f329413f640 2023-11-28T17:09:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.045+0000 7f329a043640 1 --2- 172.21.15.116:0/46948884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f329413f640 0x7f32941f0fd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f328c00cc30 tx=0x7f328c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.046+0000 7f328a7fc640 1 -- 172.21.15.116:0/46948884 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c00ce40 con 0x7f329413f640 2023-11-28T17:09:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.046+0000 7f328a7fc640 1 -- 172.21.15.116:0/46948884 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328c003990 con 0x7f329413f640 2023-11-28T17:09:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.046+0000 7f329baa6640 1 -- 172.21.15.116:0/46948884 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3294141d50 con 0x7f329413f640 2023-11-28T17:09:16.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.046+0000 7f328a7fc640 1 -- 172.21.15.116:0/46948884 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c003d20 con 0x7f329413f640 2023-11-28T17:09:16.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.046+0000 7f329baa6640 1 -- 172.21.15.116:0/46948884 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32941410f0 con 0x7f329413f640 2023-11-28T17:09:16.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.047+0000 7f329baa6640 1 -- 172.21.15.116:0/46948884 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3294142270 con 0x7f329413f640 2023-11-28T17:09:16.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.048+0000 7f328a7fc640 1 -- 172.21.15.116:0/46948884 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f328c003e80 con 0x7f329413f640 2023-11-28T17:09:16.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.049+0000 7f328a7fc640 1 --2- 172.21.15.116:0/46948884 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f327806b2a0 0x7f327806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.049+0000 7f328a7fc640 1 -- 172.21.15.116:0/46948884 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f328c089fa0 con 0x7f329413f640 2023-11-28T17:09:16.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.049+0000 7f3299842640 1 --2- 172.21.15.116:0/46948884 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f327806b2a0 0x7f327806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.050+0000 7f3299842640 1 --2- 172.21.15.116:0/46948884 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f327806b2a0 0x7f327806d760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3290035000 tx=0x7f3290008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.053+0000 7f328a7fc640 1 -- 172.21.15.116:0/46948884 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f328c0bce90 con 0x7f329413f640 2023-11-28T17:09:16.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.244+0000 7f329baa6640 1 -- 172.21.15.116:0/46948884 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3294069360 con 0x7f329413f640 2023-11-28T17:09:16.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.245+0000 7f328a7fc640 1 -- 172.21.15.116:0/46948884 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f328c01d070 con 0x7f329413f640 2023-11-28T17:09:16.247 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:09:16.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.248+0000 7f329baa6640 1 -- 172.21.15.116:0/46948884 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f327806b2a0 msgr2=0x7f327806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.248+0000 7f329baa6640 1 --2- 172.21.15.116:0/46948884 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f327806b2a0 0x7f327806d760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3290035000 tx=0x7f3290008030 comp rx=0 tx=0).stop 2023-11-28T17:09:16.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.248+0000 7f329baa6640 1 -- 172.21.15.116:0/46948884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f329413f640 msgr2=0x7f32941f0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.248+0000 7f329baa6640 1 --2- 172.21.15.116:0/46948884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f329413f640 0x7f32941f0fd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f328c00cc30 tx=0x7f328c00a5c0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.248+0000 7f329baa6640 1 -- 172.21.15.116:0/46948884 shutdown_connections 2023-11-28T17:09:16.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.248+0000 7f329baa6640 1 --2- 172.21.15.116:0/46948884 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f327806b2a0 0x7f327806d760 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.248+0000 7f329baa6640 1 --2- 172.21.15.116:0/46948884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f329413f640 0x7f32941f0fd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.248+0000 7f329baa6640 1 --2- 172.21.15.116:0/46948884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f329413cee0 0x7f3294137bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.248+0000 7f329baa6640 1 --2- 172.21.15.116:0/46948884 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3294071d20 0x7f329415c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.248+0000 7f329baa6640 1 -- 172.21.15.116:0/46948884 >> 172.21.15.116:0/46948884 conn(0x7f32940b0cf0 msgr2=0x7f32940af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:16.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.248+0000 7f329baa6640 1 -- 172.21.15.116:0/46948884 shutdown_connections 2023-11-28T17:09:16.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.248+0000 7f329baa6640 1 -- 172.21.15.116:0/46948884 wait complete. 2023-11-28T17:09:16.268 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:09:16.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.464+0000 7fc6a5f7c640 1 Processor -- start 2023-11-28T17:09:16.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.464+0000 7fc6a5f7c640 1 -- start start 2023-11-28T17:09:16.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.466+0000 7fc6a5f7c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc6a015f7b0 0x7fc6a0071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.466+0000 7fc6a5f7c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a015fc10 con 0x7fc6a00721b0 2023-11-28T17:09:16.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.466+0000 7fc6a5f7c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a0070bd0 con 0x7fc6a0144430 2023-11-28T17:09:16.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.466+0000 7fc6a5f7c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a0144810 con 0x7fc6a015f7b0 2023-11-28T17:09:16.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.466+0000 7fc69ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc6a00721b0 0x7fc6a006c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54748/0 (socket says 172.21.15.116:54748) 2023-11-28T17:09:16.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.466+0000 7fc69ffff640 1 -- 172.21.15.116:0/1371586782 learned_addr learned my addr 172.21.15.116:0/1371586782 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:16.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.466+0000 7fc69effd640 1 --2- 172.21.15.116:0/1371586782 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc6a015f7b0 0x7fc6a0071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.466+0000 7fc69e7fc640 1 -- 172.21.15.116:0/1371586782 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951364462 0 0) 0x7fc6a015fc10 con 0x7fc6a00721b0 2023-11-28T17:09:16.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.466+0000 7fc69e7fc640 1 -- 172.21.15.116:0/1371586782 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc688003520 con 0x7fc6a00721b0 2023-11-28T17:09:16.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69e7fc640 1 -- 172.21.15.116:0/1371586782 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1055696012 0 0) 0x7fc688003520 con 0x7fc6a00721b0 2023-11-28T17:09:16.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69e7fc640 1 -- 172.21.15.116:0/1371586782 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc6a015f7b0 msgr2=0x7fc6a0071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:16.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69e7fc640 1 --2- 172.21.15.116:0/1371586782 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc6a015f7b0 0x7fc6a0071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69e7fc640 1 -- 172.21.15.116:0/1371586782 >> v1:172.21.15.156:6789/0 conn(0x7fc6a0144430 legacy=0x7fc6a00703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69e7fc640 1 -- 172.21.15.116:0/1371586782 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a0070eb0 con 0x7fc6a00721b0 2023-11-28T17:09:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69e7fc640 1 -- 172.21.15.116:0/1371586782 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fc6980019f0 con 0x7fc6a00721b0 2023-11-28T17:09:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69e7fc640 1 -- 172.21.15.116:0/1371586782 >> v1:172.21.15.116:6789/0 conn(0x7fc6a00721b0 legacy=0x7fc6a006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69e7fc640 1 --2- 172.21.15.116:0/1371586782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6880039a0 0x7fc688003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69e7fc640 1 --2- 172.21.15.116:0/1371586782 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc6a015f7b0 0x7fc6880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69e7fc640 1 --2- 172.21.15.116:0/1371586782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc688004d00 0x7fc688005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69e7fc640 1 -- 172.21.15.116:0/1371586782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a0144810 con 0x7fc688004d00 2023-11-28T17:09:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69e7fc640 1 -- 172.21.15.116:0/1371586782 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68800ba60 con 0x7fc6880039a0 2023-11-28T17:09:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69e7fc640 1 -- 172.21.15.116:0/1371586782 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc68800bc20 con 0x7fc6a015f7b0 2023-11-28T17:09:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69ffff640 1 --2- 172.21.15.116:0/1371586782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc688004d00 0x7fc688005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69f7fe640 1 --2- 172.21.15.116:0/1371586782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6880039a0 0x7fc688003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69effd640 1 --2- 172.21.15.116:0/1371586782 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc6a015f7b0 0x7fc6880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69ffff640 1 -- 172.21.15.116:0/1371586782 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc6a015f7b0 msgr2=0x7fc6880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69ffff640 1 --2- 172.21.15.116:0/1371586782 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc6a015f7b0 0x7fc6880047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69ffff640 1 -- 172.21.15.116:0/1371586782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6880039a0 msgr2=0x7fc688003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.467+0000 7fc69ffff640 1 --2- 172.21.15.116:0/1371586782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6880039a0 0x7fc688003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.468+0000 7fc69ffff640 1 -- 172.21.15.116:0/1371586782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc688003520 con 0x7fc688004d00 2023-11-28T17:09:16.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.468+0000 7fc69ffff640 1 --2- 172.21.15.116:0/1371586782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc688004d00 0x7fc688005180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc698002230 tx=0x7fc6880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=645b096398205072 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:16.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.469+0000 7fc69e7fc640 1 -- 172.21.15.116:0/1371586782 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc698034630 con 0x7fc688004d00 2023-11-28T17:09:16.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.469+0000 7fc69e7fc640 1 -- 172.21.15.116:0/1371586782 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69802f070 con 0x7fc688004d00 2023-11-28T17:09:16.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.469+0000 7fc69e7fc640 1 -- 172.21.15.116:0/1371586782 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6980349f0 con 0x7fc688004d00 2023-11-28T17:09:16.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.469+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/1371586782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc688004d00 msgr2=0x7fc688005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.469+0000 7fc6a5f7c640 1 --2- 172.21.15.116:0/1371586782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc688004d00 0x7fc688005180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc698002230 tx=0x7fc6880057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.469+0000 7fc69ffff640 1 -- 172.21.15.116:0/1371586782 reap_dead start 2023-11-28T17:09:16.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.469+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/1371586782 shutdown_connections 2023-11-28T17:09:16.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.469+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/1371586782 >> 172.21.15.116:0/1371586782 conn(0x7fc6a006a4c0 msgr2=0x7fc6a006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.469+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/1371586782 shutdown_connections 2023-11-28T17:09:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.469+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/1371586782 wait complete. 2023-11-28T17:09:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.471+0000 7fc6a5f7c640 1 Processor -- start 2023-11-28T17:09:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.471+0000 7fc6a5f7c640 1 -- start start 2023-11-28T17:09:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.472+0000 7fc6a5f7c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc688004d00 0x7fc6a014af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.472+0000 7fc6a5f7c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6a013aeb0 0x7fc6a0137920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.472+0000 7fc6a5f7c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6a01394c0 0x7fc6a014a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.472+0000 7fc6a5f7c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a0137e60 con 0x7fc6a01394c0 2023-11-28T17:09:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.472+0000 7fc6a5f7c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a013b330 con 0x7fc6a013aeb0 2023-11-28T17:09:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.472+0000 7fc6a5f7c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a0139e10 con 0x7fc688004d00 2023-11-28T17:09:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.472+0000 7fc69ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6a01394c0 0x7fc6a014a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.472+0000 7fc69ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6a01394c0 0x7fc6a014a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52334/0 (socket says 172.21.15.116:52334) 2023-11-28T17:09:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.472+0000 7fc69ffff640 1 -- 172.21.15.116:0/2676969950 learned_addr learned my addr 172.21.15.116:0/2676969950 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.473+0000 7fc69f7fe640 1 --2- 172.21.15.116:0/2676969950 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc688004d00 0x7fc6a014af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.473+0000 7fc69effd640 1 --2- 172.21.15.116:0/2676969950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6a013aeb0 0x7fc6a0137920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.473+0000 7fc69ffff640 1 -- 172.21.15.116:0/2676969950 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc688004d00 msgr2=0x7fc6a014af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.473+0000 7fc69ffff640 1 --2- 172.21.15.116:0/2676969950 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc688004d00 0x7fc6a014af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.473+0000 7fc69ffff640 1 -- 172.21.15.116:0/2676969950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6a013aeb0 msgr2=0x7fc6a0137920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.473+0000 7fc69ffff640 1 --2- 172.21.15.116:0/2676969950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6a013aeb0 0x7fc6a0137920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.473+0000 7fc69ffff640 1 -- 172.21.15.116:0/2676969950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a013a0f0 con 0x7fc6a01394c0 2023-11-28T17:09:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.473+0000 7fc69ffff640 1 --2- 172.21.15.116:0/2676969950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6a01394c0 0x7fc6a014a370 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc69802ca70 tx=0x7fc698001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.473+0000 7fc687fff640 1 -- 172.21.15.116:0/2676969950 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc698034840 con 0x7fc6a01394c0 2023-11-28T17:09:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.473+0000 7fc687fff640 1 -- 172.21.15.116:0/2676969950 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6980349a0 con 0x7fc6a01394c0 2023-11-28T17:09:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.473+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/2676969950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a013a3d0 con 0x7fc6a01394c0 2023-11-28T17:09:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.473+0000 7fc687fff640 1 -- 172.21.15.116:0/2676969950 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc698034d10 con 0x7fc6a01394c0 2023-11-28T17:09:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.474+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/2676969950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a013a990 con 0x7fc6a01394c0 2023-11-28T17:09:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.475+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/2676969950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a00693d0 con 0x7fc6a01394c0 2023-11-28T17:09:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.476+0000 7fc687fff640 1 -- 172.21.15.116:0/2676969950 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fc698031030 con 0x7fc6a01394c0 2023-11-28T17:09:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.476+0000 7fc687fff640 1 --2- 172.21.15.116:0/2676969950 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc68c06b2a0 0x7fc68c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.477+0000 7fc687fff640 1 -- 172.21.15.116:0/2676969950 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6980ac260 con 0x7fc6a01394c0 2023-11-28T17:09:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.477+0000 7fc69f7fe640 1 --2- 172.21.15.116:0/2676969950 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc68c06b2a0 0x7fc68c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.477+0000 7fc69f7fe640 1 --2- 172.21.15.116:0/2676969950 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc68c06b2a0 0x7fc68c06d760 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc6a01435b0 tx=0x7fc690009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:16.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.481+0000 7fc687fff640 1 -- 172.21.15.116:0/2676969950 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6980771d0 con 0x7fc6a01394c0 2023-11-28T17:09:16.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.708+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/2676969950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6a00693d0 con 0x7fc6a01394c0 2023-11-28T17:09:16.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.709+0000 7fc687fff640 1 -- 172.21.15.116:0/2676969950 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7fc69807a7f0 con 0x7fc6a01394c0 2023-11-28T17:09:16.711 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:16.711 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3138968863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":3138968863},{"type":"v1","addr":"172.21.15.156:6835","nonce":3138968863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/733981412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":733981412},{"type":"v1","addr":"172.21.15.116:6836","nonce":733981412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2751219705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2751219705},{"type":"v1","addr":"172.21.15.116:6837","nonce":2751219705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:09:00.673159+0000","modified":"2023-11-28T17:09:03.771842+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6834/3268354971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3268354971},{"type":"v1","addr":"172.21.15.156:6834","nonce":3268354971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T17:09:16.711 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2023-11-28T17:09:16.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.712+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/2676969950 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc68c06b2a0 msgr2=0x7fc68c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.712+0000 7fc6a5f7c640 1 --2- 172.21.15.116:0/2676969950 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc68c06b2a0 0x7fc68c06d760 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc6a01435b0 tx=0x7fc690009040 comp rx=0 tx=0).stop 2023-11-28T17:09:16.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.712+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/2676969950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6a01394c0 msgr2=0x7fc6a014a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.712+0000 7fc6a5f7c640 1 --2- 172.21.15.116:0/2676969950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6a01394c0 0x7fc6a014a370 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc69802ca70 tx=0x7fc698001cb0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.712+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/2676969950 shutdown_connections 2023-11-28T17:09:16.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.712+0000 7fc6a5f7c640 1 --2- 172.21.15.116:0/2676969950 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc68c06b2a0 0x7fc68c06d760 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.712+0000 7fc6a5f7c640 1 --2- 172.21.15.116:0/2676969950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6a01394c0 0x7fc6a014a370 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.712+0000 7fc6a5f7c640 1 --2- 172.21.15.116:0/2676969950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6a013aeb0 0x7fc6a0137920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.712+0000 7fc6a5f7c640 1 --2- 172.21.15.116:0/2676969950 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc688004d00 0x7fc6a014af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.712+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/2676969950 >> 172.21.15.116:0/2676969950 conn(0x7fc6a006a4c0 msgr2=0x7fc6a015b370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:16.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.712+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/2676969950 shutdown_connections 2023-11-28T17:09:16.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.712+0000 7fc6a5f7c640 1 -- 172.21.15.116:0/2676969950 wait complete. 2023-11-28T17:09:16.732 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:16.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.928+0000 7f40f5abb640 1 Processor -- start 2023-11-28T17:09:16.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.928+0000 7f40f5abb640 1 -- start start 2023-11-28T17:09:16.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.930+0000 7f40f5abb640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f40f015f7b0 0x7f40f0071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.930+0000 7f40f5abb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f015fc10 con 0x7f40f00721b0 2023-11-28T17:09:16.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.930+0000 7f40f5abb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f0070bd0 con 0x7f40f0144430 2023-11-28T17:09:16.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.930+0000 7f40f5abb640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40f0144810 con 0x7f40f015f7b0 2023-11-28T17:09:16.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.930+0000 7f40eeffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f40f015f7b0 0x7f40f0071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.930+0000 7f40eeffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f40f015f7b0 0x7f40f0071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52348/0 (socket says 172.21.15.116:52348) 2023-11-28T17:09:16.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.930+0000 7f40eeffd640 1 -- 172.21.15.116:0/1994239224 learned_addr learned my addr 172.21.15.116:0/1994239224 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:16.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.930+0000 7f40effff640 1 --1- 172.21.15.116:0/1994239224 >> v1:172.21.15.116:6789/0 conn(0x7f40f00721b0 0x7f40f006c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54750/0 (socket says 172.21.15.116:54750) 2023-11-28T17:09:16.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.930+0000 7f40eeffd640 1 -- 172.21.15.116:0/1994239224 >> v1:172.21.15.156:6789/0 conn(0x7f40f0144430 legacy=0x7f40f00703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:16.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.930+0000 7f40eeffd640 1 -- 172.21.15.116:0/1994239224 >> v1:172.21.15.116:6789/0 conn(0x7f40f00721b0 legacy=0x7f40f006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.930+0000 7f40eeffd640 1 -- 172.21.15.116:0/1994239224 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f0070eb0 con 0x7f40f015f7b0 2023-11-28T17:09:16.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.931+0000 7f40eeffd640 1 --2- 172.21.15.116:0/1994239224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f015f7b0 0x7f40f0071c70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f40e4009870 tx=0x7f40e402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214d81ef49e5818b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:16.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.932+0000 7f40ee7fc640 1 -- 172.21.15.116:0/1994239224 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e4030850 con 0x7f40f015f7b0 2023-11-28T17:09:16.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.932+0000 7f40ee7fc640 1 -- 172.21.15.116:0/1994239224 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e40309b0 con 0x7f40f015f7b0 2023-11-28T17:09:16.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.932+0000 7f40ee7fc640 1 -- 172.21.15.116:0/1994239224 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e4030d40 con 0x7f40f015f7b0 2023-11-28T17:09:16.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.932+0000 7f40f5abb640 1 -- 172.21.15.116:0/1994239224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f015f7b0 msgr2=0x7f40f0071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.932+0000 7f40f5abb640 1 --2- 172.21.15.116:0/1994239224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f015f7b0 0x7f40f0071c70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f40e4009870 tx=0x7f40e402b990 comp rx=0 tx=0).stop 2023-11-28T17:09:16.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.932+0000 7f40f5abb640 1 -- 172.21.15.116:0/1994239224 shutdown_connections 2023-11-28T17:09:16.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.933+0000 7f40f5abb640 1 --2- 172.21.15.116:0/1994239224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f015f7b0 0x7f40f0071c70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.933+0000 7f40f5abb640 1 -- 172.21.15.116:0/1994239224 >> 172.21.15.116:0/1994239224 conn(0x7f40f006a4c0 msgr2=0x7f40f006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:16.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.933+0000 7f40f5abb640 1 -- 172.21.15.116:0/1994239224 shutdown_connections 2023-11-28T17:09:16.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.933+0000 7f40f5abb640 1 -- 172.21.15.116:0/1994239224 wait complete. 2023-11-28T17:09:16.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.934+0000 7f40f5abb640 1 Processor -- start 2023-11-28T17:09:16.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.934+0000 7f40f5abb640 1 -- start start 2023-11-28T17:09:16.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.935+0000 7f40f5abb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f40f00721b0 0x7f40f015dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.935+0000 7f40f5abb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40f0144430 0x7f40f00a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.935+0000 7f40f5abb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f015f7b0 0x7f40f013ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.935+0000 7f40f5abb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f0130910 con 0x7f40f015f7b0 2023-11-28T17:09:16.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.935+0000 7f40f5abb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f012fd00 con 0x7f40f0144430 2023-11-28T17:09:16.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.935+0000 7f40f5abb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f012ffe0 con 0x7f40f00721b0 2023-11-28T17:09:16.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.935+0000 7f40effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f015f7b0 0x7f40f013ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.935+0000 7f40effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f015f7b0 0x7f40f013ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52362/0 (socket says 172.21.15.116:52362) 2023-11-28T17:09:16.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.935+0000 7f40effff640 1 -- 172.21.15.116:0/2352043897 learned_addr learned my addr 172.21.15.116:0/2352043897 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:16.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.936+0000 7f40effff640 1 -- 172.21.15.116:0/2352043897 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f40f00721b0 msgr2=0x7f40f015dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.936+0000 7f40effff640 1 --2- 172.21.15.116:0/2352043897 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f40f00721b0 0x7f40f015dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.936+0000 7f40eeffd640 1 --2- 172.21.15.116:0/2352043897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40f0144430 0x7f40f00a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.936+0000 7f40effff640 1 -- 172.21.15.116:0/2352043897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40f0144430 msgr2=0x7f40f00a8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:16.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.936+0000 7f40effff640 1 --2- 172.21.15.116:0/2352043897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40f0144430 0x7f40f00a8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:16.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.936+0000 7f40effff640 1 -- 172.21.15.116:0/2352043897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e40094d0 con 0x7f40f015f7b0 2023-11-28T17:09:16.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.936+0000 7f40eeffd640 1 --2- 172.21.15.116:0/2352043897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40f0144430 0x7f40f00a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:16.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.936+0000 7f40effff640 1 --2- 172.21.15.116:0/2352043897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f015f7b0 0x7f40f013ea90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f40e800c730 tx=0x7f40e800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:16.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.936+0000 7f40d7fff640 1 -- 172.21.15.116:0/2352043897 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e800ce40 con 0x7f40f015f7b0 2023-11-28T17:09:16.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.936+0000 7f40d7fff640 1 -- 172.21.15.116:0/2352043897 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e800a640 con 0x7f40f015f7b0 2023-11-28T17:09:16.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.936+0000 7f40d7fff640 1 -- 172.21.15.116:0/2352043897 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e800a9f0 con 0x7f40f015f7b0 2023-11-28T17:09:16.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.936+0000 7f40f5abb640 1 -- 172.21.15.116:0/2352043897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f01302c0 con 0x7f40f015f7b0 2023-11-28T17:09:16.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.936+0000 7f40f5abb640 1 -- 172.21.15.116:0/2352043897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f015e1a0 con 0x7f40f015f7b0 2023-11-28T17:09:16.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.937+0000 7f40f5abb640 1 -- 172.21.15.116:0/2352043897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f00693d0 con 0x7f40f015f7b0 2023-11-28T17:09:16.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.938+0000 7f40d7fff640 1 -- 172.21.15.116:0/2352043897 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f40e800ab50 con 0x7f40f015f7b0 2023-11-28T17:09:16.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.939+0000 7f40d7fff640 1 --2- 172.21.15.116:0/2352043897 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f40dc06b2a0 0x7f40dc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:16.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.939+0000 7f40d7fff640 1 -- 172.21.15.116:0/2352043897 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40e8089370 con 0x7f40f015f7b0 2023-11-28T17:09:16.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.939+0000 7f40ef7fe640 1 --2- 172.21.15.116:0/2352043897 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f40dc06b2a0 0x7f40dc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:16.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.940+0000 7f40ef7fe640 1 --2- 172.21.15.116:0/2352043897 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f40dc06b2a0 0x7f40dc06d760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f40e0003cc0 tx=0x7f40e00079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:16.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:16.943+0000 7f40d7fff640 1 -- 172.21.15.116:0/2352043897 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f40e80541f0 con 0x7f40f015f7b0 2023-11-28T17:09:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.140+0000 7f40f5abb640 1 -- 172.21.15.116:0/2352043897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40f015ee20 con 0x7f40f015f7b0 2023-11-28T17:09:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.142+0000 7f40d7fff640 1 -- 172.21.15.116:0/2352043897 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16521 (secure 0 0 0) 0x7f40e8057810 con 0x7f40f015f7b0 2023-11-28T17:09:17.143 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:17.144 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:01.970957+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.144+0000 7f40f5abb640 1 -- 172.21.15.116:0/2352043897 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f40dc06b2a0 msgr2=0x7f40dc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.144+0000 7f40f5abb640 1 --2- 172.21.15.116:0/2352043897 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f40dc06b2a0 0x7f40dc06d760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f40e0003cc0 tx=0x7f40e00079b0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.144+0000 7f40f5abb640 1 -- 172.21.15.116:0/2352043897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f015f7b0 msgr2=0x7f40f013ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.144+0000 7f40f5abb640 1 --2- 172.21.15.116:0/2352043897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f015f7b0 0x7f40f013ea90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f40e800c730 tx=0x7f40e800cbf0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.144+0000 7f40f5abb640 1 -- 172.21.15.116:0/2352043897 shutdown_connections 2023-11-28T17:09:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.144+0000 7f40f5abb640 1 --2- 172.21.15.116:0/2352043897 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f40dc06b2a0 0x7f40dc06d760 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.144+0000 7f40f5abb640 1 --2- 172.21.15.116:0/2352043897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40f015f7b0 0x7f40f013ea90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.144+0000 7f40f5abb640 1 --2- 172.21.15.116:0/2352043897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40f0144430 0x7f40f00a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.144+0000 7f40f5abb640 1 --2- 172.21.15.116:0/2352043897 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f40f00721b0 0x7f40f015dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.144+0000 7f40f5abb640 1 -- 172.21.15.116:0/2352043897 >> 172.21.15.116:0/2352043897 conn(0x7f40f006a4c0 msgr2=0x7f40f00c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.145+0000 7f40f5abb640 1 -- 172.21.15.116:0/2352043897 shutdown_connections 2023-11-28T17:09:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.145+0000 7f40f5abb640 1 -- 172.21.15.116:0/2352043897 wait complete. 2023-11-28T17:09:17.165 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:09:17.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.372+0000 7f52c94d3640 1 Processor -- start 2023-11-28T17:09:17.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.373+0000 7f52c94d3640 1 -- start start 2023-11-28T17:09:17.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.375+0000 7f52c94d3640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f52c40ce140 0x7f52c40d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:17.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.375+0000 7f52c94d3640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c40b6640 con 0x7f52c415d2c0 2023-11-28T17:09:17.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.375+0000 7f52c94d3640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c40d31c0 con 0x7f52c40d2d70 2023-11-28T17:09:17.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.375+0000 7f52c94d3640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c40d3480 con 0x7f52c40ce140 2023-11-28T17:09:17.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.375+0000 7f52c2ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f52c415d2c0 0x7f52c40cda30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54760/0 (socket says 172.21.15.116:54760) 2023-11-28T17:09:17.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.375+0000 7f52c2ffd640 1 -- 172.21.15.116:0/2703025148 learned_addr learned my addr 172.21.15.116:0/2703025148 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:17.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52ba7fc640 1 --2- 172.21.15.116:0/2703025148 >> v2:172.21.15.156:3300/0 conn(0x7f52c40ce140 0x7f52c40d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:17.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 -- 172.21.15.116:0/2703025148 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955804511 0 0) 0x7f52c40b6640 con 0x7f52c415d2c0 2023-11-28T17:09:17.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 -- 172.21.15.116:0/2703025148 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b0003520 con 0x7f52c415d2c0 2023-11-28T17:09:17.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 -- 172.21.15.116:0/2703025148 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1174861231 0 0) 0x7f52b0003520 con 0x7f52c415d2c0 2023-11-28T17:09:17.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 -- 172.21.15.116:0/2703025148 >> v2:172.21.15.156:3300/0 conn(0x7f52c40ce140 msgr2=0x7f52c40d2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:17.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 --2- 172.21.15.116:0/2703025148 >> v2:172.21.15.156:3300/0 conn(0x7f52c40ce140 0x7f52c40d2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 -- 172.21.15.116:0/2703025148 >> v1:172.21.15.156:6789/0 conn(0x7f52c40d2d70 legacy=0x7f52c40b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:17.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 -- 172.21.15.116:0/2703025148 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c40d3760 con 0x7f52c415d2c0 2023-11-28T17:09:17.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 -- 172.21.15.116:0/2703025148 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f52b4001670 con 0x7f52c415d2c0 2023-11-28T17:09:17.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 -- 172.21.15.116:0/2703025148 >> v1:172.21.15.116:6789/0 conn(0x7f52c415d2c0 legacy=0x7f52c40cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:17.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 --2- 172.21.15.116:0/2703025148 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f52b0003970 0x7f52b0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:17.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 --2- 172.21.15.116:0/2703025148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b00047c0 0x7f52b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:17.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 --2- 172.21.15.116:0/2703025148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52b0005180 0x7f52b000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:17.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 -- 172.21.15.116:0/2703025148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40d3480 con 0x7f52b00047c0 2023-11-28T17:09:17.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 -- 172.21.15.116:0/2703025148 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b000c120 con 0x7f52b0005180 2023-11-28T17:09:17.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52c27fc640 1 -- 172.21.15.116:0/2703025148 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b000c2e0 con 0x7f52b0003970 2023-11-28T17:09:17.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.376+0000 7f52ba7fc640 1 --2- 172.21.15.116:0/2703025148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b00047c0 0x7f52b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:17.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.377+0000 7f52ba7fc640 1 -- 172.21.15.116:0/2703025148 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f52b0003970 msgr2=0x7f52b0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:17.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.377+0000 7f52ba7fc640 1 --2- 172.21.15.116:0/2703025148 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f52b0003970 0x7f52b0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.377+0000 7f52ba7fc640 1 -- 172.21.15.116:0/2703025148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52b0005180 msgr2=0x7f52b000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:17.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.377+0000 7f52ba7fc640 1 --2- 172.21.15.116:0/2703025148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52b0005180 0x7f52b000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.377+0000 7f52ba7fc640 1 -- 172.21.15.116:0/2703025148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b0003520 con 0x7f52b00047c0 2023-11-28T17:09:17.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.377+0000 7f52c37fe640 1 -- 172.21.15.116:0/2703025148 reap_dead start 2023-11-28T17:09:17.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.378+0000 7f52ba7fc640 1 --2- 172.21.15.116:0/2703025148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b00047c0 0x7f52b0004c40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f52ac009a90 tx=0x7f52b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9bd040ca9cdc68 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:17.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.378+0000 7f52c27fc640 1 -- 172.21.15.116:0/2703025148 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac02c6b0 con 0x7f52b00047c0 2023-11-28T17:09:17.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.378+0000 7f52c27fc640 1 -- 172.21.15.116:0/2703025148 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ac02c810 con 0x7f52b00047c0 2023-11-28T17:09:17.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.378+0000 7f52c27fc640 1 -- 172.21.15.116:0/2703025148 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ac02cbc0 con 0x7f52b00047c0 2023-11-28T17:09:17.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.378+0000 7f52c94d3640 1 -- 172.21.15.116:0/2703025148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b00047c0 msgr2=0x7f52b0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:17.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.378+0000 7f52c94d3640 1 --2- 172.21.15.116:0/2703025148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b00047c0 0x7f52b0004c40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f52ac009a90 tx=0x7f52b00057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.379+0000 7f52c94d3640 1 -- 172.21.15.116:0/2703025148 shutdown_connections 2023-11-28T17:09:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.379+0000 7f52c94d3640 1 --2- 172.21.15.116:0/2703025148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52b00047c0 0x7f52b0004c40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.379+0000 7f52c94d3640 1 -- 172.21.15.116:0/2703025148 >> 172.21.15.116:0/2703025148 conn(0x7f52c4072010 msgr2=0x7f52c4072420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.379+0000 7f52c94d3640 1 -- 172.21.15.116:0/2703025148 shutdown_connections 2023-11-28T17:09:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.379+0000 7f52c94d3640 1 -- 172.21.15.116:0/2703025148 wait complete. 2023-11-28T17:09:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.380+0000 7f52c94d3640 1 Processor -- start 2023-11-28T17:09:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.381+0000 7f52c94d3640 1 -- start start 2023-11-28T17:09:17.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.381+0000 7f52c94d3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52b00047c0 0x7f52c415a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:17.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.381+0000 7f52c94d3640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f52c41445c0 0x7f52c415aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:17.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.381+0000 7f52c94d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c415b1e0 0x7f52c40c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:17.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.381+0000 7f52c94d3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c41599e0 con 0x7f52c415b1e0 2023-11-28T17:09:17.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.381+0000 7f52c94d3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c40cd0f0 con 0x7f52b00047c0 2023-11-28T17:09:17.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.381+0000 7f52c94d3640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c4159c80 con 0x7f52c41445c0 2023-11-28T17:09:17.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.382+0000 7f52c37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c415b1e0 0x7f52c40c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.382+0000 7f52c37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c415b1e0 0x7f52c40c7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52370/0 (socket says 172.21.15.116:52370) 2023-11-28T17:09:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.382+0000 7f52c37fe640 1 -- 172.21.15.116:0/643738623 learned_addr learned my addr 172.21.15.116:0/643738623 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.382+0000 7f52c2ffd640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52b00047c0 0x7f52c415a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.382+0000 7f52ba7fc640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f52c41445c0 0x7f52c415aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.382+0000 7f52c37fe640 1 -- 172.21.15.116:0/643738623 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f52c41445c0 msgr2=0x7f52c415aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.382+0000 7f52c37fe640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f52c41445c0 0x7f52c415aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.382+0000 7f52c37fe640 1 -- 172.21.15.116:0/643738623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52b00047c0 msgr2=0x7f52c415a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.382+0000 7f52c37fe640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52b00047c0 0x7f52c415a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.382+0000 7f52c37fe640 1 -- 172.21.15.116:0/643738623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c4159f60 con 0x7f52c415b1e0 2023-11-28T17:09:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.382+0000 7f52ba7fc640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f52c41445c0 0x7f52c415aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.382+0000 7f52c37fe640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c415b1e0 0x7f52c40c7d50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f52c40cef70 tx=0x7f52bc00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:17.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.383+0000 7f52bbfff640 1 -- 172.21.15.116:0/643738623 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc007070 con 0x7f52c415b1e0 2023-11-28T17:09:17.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.383+0000 7f52bbfff640 1 -- 172.21.15.116:0/643738623 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52bc0071d0 con 0x7f52c415b1e0 2023-11-28T17:09:17.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.383+0000 7f52bbfff640 1 -- 172.21.15.116:0/643738623 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc007580 con 0x7f52c415b1e0 2023-11-28T17:09:17.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.383+0000 7f52c94d3640 1 -- 172.21.15.116:0/643738623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c415a240 con 0x7f52c415b1e0 2023-11-28T17:09:17.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.383+0000 7f52c94d3640 1 -- 172.21.15.116:0/643738623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c40ccca0 con 0x7f52c415b1e0 2023-11-28T17:09:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.384+0000 7f52c94d3640 1 -- 172.21.15.116:0/643738623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c40ccec0 con 0x7f52c415b1e0 2023-11-28T17:09:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.385+0000 7f52bbfff640 1 -- 172.21.15.116:0/643738623 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f52bc0076e0 con 0x7f52c415b1e0 2023-11-28T17:09:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.386+0000 7f52bbfff640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f52a406b2a0 0x7f52a406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.386+0000 7f52bbfff640 1 -- 172.21.15.116:0/643738623 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52bc089020 con 0x7f52c415b1e0 2023-11-28T17:09:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.386+0000 7f52c2ffd640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f52a406b2a0 0x7f52a406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:17.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.387+0000 7f52c2ffd640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f52a406b2a0 0x7f52a406d760 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f52b4000f60 tx=0x7f52b4001700 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:17.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.390+0000 7f52bbfff640 1 -- 172.21.15.116:0/643738623 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52bc053e90 con 0x7f52c415b1e0 2023-11-28T17:09:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.618+0000 7f52c94d3640 1 -- 172.21.15.116:0/643738623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52c40c6b70 con 0x7f52c415b1e0 2023-11-28T17:09:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.619+0000 7f52bbfff640 1 -- 172.21.15.116:0/643738623 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f52bc017020 con 0x7f52c415b1e0 2023-11-28T17:09:17.620 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:17.620 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3138968863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":3138968863},{"type":"v1","addr":"172.21.15.156:6835","nonce":3138968863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/733981412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":733981412},{"type":"v1","addr":"172.21.15.116:6836","nonce":733981412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2751219705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2751219705},{"type":"v1","addr":"172.21.15.116:6837","nonce":2751219705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:09:00.673159+0000","modified":"2023-11-28T17:09:03.771842+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6834/3268354971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3268354971},{"type":"v1","addr":"172.21.15.156:6834","nonce":3268354971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T17:09:17.621 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2023-11-28T17:09:17.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.621+0000 7f52c94d3640 1 -- 172.21.15.116:0/643738623 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f52a406b2a0 msgr2=0x7f52a406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:17.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.621+0000 7f52c94d3640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f52a406b2a0 0x7f52a406d760 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f52b4000f60 tx=0x7f52b4001700 comp rx=0 tx=0).stop 2023-11-28T17:09:17.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.621+0000 7f52c94d3640 1 -- 172.21.15.116:0/643738623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c415b1e0 msgr2=0x7f52c40c7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:17.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.621+0000 7f52c94d3640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c415b1e0 0x7f52c40c7d50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f52c40cef70 tx=0x7f52bc00c980 comp rx=0 tx=0).stop 2023-11-28T17:09:17.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.621+0000 7f52c94d3640 1 -- 172.21.15.116:0/643738623 shutdown_connections 2023-11-28T17:09:17.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.622+0000 7f52c94d3640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f52a406b2a0 0x7f52a406d760 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.622+0000 7f52c94d3640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52c415b1e0 0x7f52c40c7d50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.622+0000 7f52c94d3640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f52c41445c0 0x7f52c415aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.622+0000 7f52c94d3640 1 --2- 172.21.15.116:0/643738623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52b00047c0 0x7f52c415a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.622+0000 7f52c94d3640 1 -- 172.21.15.116:0/643738623 >> 172.21.15.116:0/643738623 conn(0x7f52c4072010 msgr2=0x7f52c41600a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.622+0000 7f52c94d3640 1 -- 172.21.15.116:0/643738623 shutdown_connections 2023-11-28T17:09:17.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:17.622+0000 7f52c94d3640 1 -- 172.21.15.116:0/643738623 wait complete. 2023-11-28T17:09:18.043 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T17:09:00.673159+0000', 'modified': '2023-11-28T17:09:03.771842+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4254}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4254': {'gid': 4254, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.156:6834/3268354971', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6832', 'nonce': 3268354971}, {'type': 'v1', 'addr': '172.21.15.156:6834', 'nonce': 3268354971}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T17:09:18.043 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T17:09:18.043 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-11-28T17:09:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.265+0000 7f84b8f89640 1 Processor -- start 2023-11-28T17:09:18.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.265+0000 7f84b8f89640 1 -- start start 2023-11-28T17:09:18.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.267+0000 7f84b8f89640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b406c960 0x7f84b415a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.267+0000 7f84b8f89640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b40aea20 con 0x7f84b4140200 2023-11-28T17:09:18.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.267+0000 7f84b8f89640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b413d2c0 con 0x7f84b413cee0 2023-11-28T17:09:18.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.267+0000 7f84b8f89640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b415b3a0 con 0x7f84b406c960 2023-11-28T17:09:18.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.267+0000 7f84b2d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f84b4140200 0x7f84b40cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41934/0 (socket says 172.21.15.116:41934) 2023-11-28T17:09:18.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.267+0000 7f84b2d9d640 1 -- 172.21.15.116:0/3936353877 learned_addr learned my addr 172.21.15.116:0/3936353877 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:18.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.267+0000 7f84b1d9b640 1 --2- 172.21.15.116:0/3936353877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b406c960 0x7f84b415a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.267+0000 7f84b159a640 1 -- 172.21.15.116:0/3936353877 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162312156 0 0) 0x7f84b40aea20 con 0x7f84b4140200 2023-11-28T17:09:18.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b159a640 1 -- 172.21.15.116:0/3936353877 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f849c003520 con 0x7f84b4140200 2023-11-28T17:09:18.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b159a640 1 -- 172.21.15.116:0/3936353877 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4176916870 0 0) 0x7f849c003520 con 0x7f84b4140200 2023-11-28T17:09:18.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b159a640 1 -- 172.21.15.116:0/3936353877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b406c960 msgr2=0x7f84b415a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:18.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b159a640 1 --2- 172.21.15.116:0/3936353877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b406c960 0x7f84b415a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b159a640 1 -- 172.21.15.116:0/3936353877 >> v1:172.21.15.156:6789/0 conn(0x7f84b413cee0 legacy=0x7f84b406c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b159a640 1 -- 172.21.15.116:0/3936353877 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b415b0c0 con 0x7f84b4140200 2023-11-28T17:09:18.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b159a640 1 -- 172.21.15.116:0/3936353877 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f84ac0019f0 con 0x7f84b4140200 2023-11-28T17:09:18.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b159a640 1 -- 172.21.15.116:0/3936353877 >> v1:172.21.15.116:6789/0 conn(0x7f84b4140200 legacy=0x7f84b40cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b159a640 1 --2- 172.21.15.116:0/3936353877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f849c0039a0 0x7f849c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b159a640 1 --2- 172.21.15.116:0/3936353877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f849c0047c0 0x7f849c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b159a640 1 --2- 172.21.15.116:0/3936353877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f849c0051d0 0x7f849c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b159a640 1 -- 172.21.15.116:0/3936353877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b415b3a0 con 0x7f849c0051d0 2023-11-28T17:09:18.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b159a640 1 -- 172.21.15.116:0/3936353877 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c00c0d0 con 0x7f849c0039a0 2023-11-28T17:09:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b159a640 1 -- 172.21.15.116:0/3936353877 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f849c00c290 con 0x7f849c0047c0 2023-11-28T17:09:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b1d9b640 1 --2- 172.21.15.116:0/3936353877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b406c960 0x7f84b415a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:09:18.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.268+0000 7f84b2d9d640 1 --2- 172.21.15.116:0/3936353877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f849c0051d0 0x7f849c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.269+0000 7f84b259c640 1 --2- 172.21.15.116:0/3936353877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f849c0039a0 0x7f849c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.269+0000 7f84b1d9b640 1 --2- 172.21.15.116:0/3936353877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f849c0047c0 0x7f849c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.269+0000 7f84b2d9d640 1 -- 172.21.15.116:0/3936353877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f849c0047c0 msgr2=0x7f849c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.269+0000 7f84b2d9d640 1 --2- 172.21.15.116:0/3936353877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f849c0047c0 0x7f849c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.269+0000 7f84b2d9d640 1 -- 172.21.15.116:0/3936353877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f849c0039a0 msgr2=0x7f849c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:18.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.269+0000 7f84b2d9d640 1 --2- 172.21.15.116:0/3936353877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f849c0039a0 0x7f849c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.269+0000 7f84b2d9d640 1 -- 172.21.15.116:0/3936353877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c003520 con 0x7f849c0051d0 2023-11-28T17:09:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.270+0000 7f84b2d9d640 1 --2- 172.21.15.116:0/3936353877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f849c0051d0 0x7f849c00b980 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00a410 tx=0x7f849c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1235c2ccc6398a97 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.270+0000 7f84b159a640 1 -- 172.21.15.116:0/3936353877 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac0072d0 con 0x7f849c0051d0 2023-11-28T17:09:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.270+0000 7f84b159a640 1 -- 172.21.15.116:0/3936353877 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ac007430 con 0x7f849c0051d0 2023-11-28T17:09:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.270+0000 7f84b159a640 1 -- 172.21.15.116:0/3936353877 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac007820 con 0x7f849c0051d0 2023-11-28T17:09:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.271+0000 7f84b8f89640 1 -- 172.21.15.116:0/3936353877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f849c0051d0 msgr2=0x7f849c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.271+0000 7f84b8f89640 1 --2- 172.21.15.116:0/3936353877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f849c0051d0 0x7f849c00b980 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00a410 tx=0x7f849c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.271+0000 7f84b2d9d640 1 -- 172.21.15.116:0/3936353877 reap_dead start 2023-11-28T17:09:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.271+0000 7f84b8f89640 1 -- 172.21.15.116:0/3936353877 shutdown_connections 2023-11-28T17:09:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.271+0000 7f84b8f89640 1 -- 172.21.15.116:0/3936353877 >> 172.21.15.116:0/3936353877 conn(0x7f84b406a4c0 msgr2=0x7f84b406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.271+0000 7f84b8f89640 1 -- 172.21.15.116:0/3936353877 shutdown_connections 2023-11-28T17:09:18.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.271+0000 7f84b8f89640 1 -- 172.21.15.116:0/3936353877 wait complete. 2023-11-28T17:09:18.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.273+0000 7f84b8f89640 1 Processor -- start 2023-11-28T17:09:18.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.273+0000 7f84b8f89640 1 -- start start 2023-11-28T17:09:18.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.274+0000 7f84b8f89640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f849c0051d0 0x7f84b40b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.274+0000 7f84b8f89640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b413a0e0 0x7f84b40b3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.274+0000 7f84b8f89640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84b40b3970 0x7f84b413b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.274+0000 7f84b8f89640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b4139cc0 con 0x7f849c0051d0 2023-11-28T17:09:18.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.274+0000 7f84b8f89640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b40b3e20 con 0x7f84b40b3970 2023-11-28T17:09:18.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.274+0000 7f84b8f89640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b413c4b0 con 0x7f84b413a0e0 2023-11-28T17:09:18.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.274+0000 7f84b259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f849c0051d0 0x7f84b40b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.274+0000 7f84b259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f849c0051d0 0x7f84b40b0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56288/0 (socket says 172.21.15.116:56288) 2023-11-28T17:09:18.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.274+0000 7f84b259c640 1 -- 172.21.15.116:0/3037761741 learned_addr learned my addr 172.21.15.116:0/3037761741 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:18.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.274+0000 7f84b259c640 1 -- 172.21.15.116:0/3037761741 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b413a0e0 msgr2=0x7f84b40b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.274+0000 7f84b259c640 1 --2- 172.21.15.116:0/3037761741 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b413a0e0 0x7f84b40b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.274+0000 7f84b259c640 1 -- 172.21.15.116:0/3037761741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84b40b3970 msgr2=0x7f84b413b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.274+0000 7f84b259c640 1 --2- 172.21.15.116:0/3037761741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84b40b3970 0x7f84b413b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.274+0000 7f84b259c640 1 -- 172.21.15.116:0/3037761741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b41f7500 con 0x7f849c0051d0 2023-11-28T17:09:18.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.275+0000 7f84b259c640 1 --2- 172.21.15.116:0/3037761741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f849c0051d0 0x7f84b40b0530 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f84a400e580 tx=0x7f84a400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:18.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.275+0000 7f84a2ffd640 1 -- 172.21.15.116:0/3037761741 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a40052e0 con 0x7f849c0051d0 2023-11-28T17:09:18.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.275+0000 7f84a2ffd640 1 -- 172.21.15.116:0/3037761741 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a4005440 con 0x7f849c0051d0 2023-11-28T17:09:18.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.275+0000 7f84a2ffd640 1 -- 172.21.15.116:0/3037761741 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a4003810 con 0x7f849c0051d0 2023-11-28T17:09:18.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.275+0000 7f84b8f89640 1 -- 172.21.15.116:0/3037761741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b4139320 con 0x7f849c0051d0 2023-11-28T17:09:18.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.275+0000 7f84b8f89640 1 -- 172.21.15.116:0/3037761741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b41398e0 con 0x7f849c0051d0 2023-11-28T17:09:18.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.278+0000 7f84a2ffd640 1 -- 172.21.15.116:0/3037761741 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f84a4003970 con 0x7f849c0051d0 2023-11-28T17:09:18.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.279+0000 7f84a2ffd640 1 --2- 172.21.15.116:0/3037761741 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f849406b2a0 0x7f849406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.279+0000 7f84a2ffd640 1 -- 172.21.15.116:0/3037761741 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84a4089710 con 0x7f849c0051d0 2023-11-28T17:09:18.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.279+0000 7f84b1d9b640 1 --2- 172.21.15.116:0/3037761741 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f849406b2a0 0x7f849406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.280+0000 7f84bbbf9740 1 Processor -- start 2023-11-28T17:09:18.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.280+0000 7f84bbbf9740 1 -- start start 2023-11-28T17:09:18.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.280+0000 7f84b1d9b640 1 --2- 172.21.15.116:0/3037761741 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f849406b2a0 0x7f849406d760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f84a8006e80 tx=0x7f84a8006df0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:18.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.280+0000 7f84bbbf9740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 0x55be60f211f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84bbbf9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60ef10c0 0x55be60f1e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84bbbf9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 0x55be60f1e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84bbbf9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be60e55880 con 0x55be60ef10c0 2023-11-28T17:09:18.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84bbbf9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be60e64ae0 con 0x55be60ef4990 2023-11-28T17:09:18.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84bbbf9740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55be60e621d0 con 0x55be60f20e10 2023-11-28T17:09:18.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84b1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60ef10c0 0x55be60f1e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84b1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60ef10c0 0x55be60f1e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56296/0 (socket says 172.21.15.116:56296) 2023-11-28T17:09:18.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84b1d9b640 1 -- 172.21.15.116:0/1152745493 learned_addr learned my addr 172.21.15.116:0/1152745493 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:18.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84b259c640 1 --2- 172.21.15.116:0/1152745493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 0x55be60f211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84b1d9b640 1 -- 172.21.15.116:0/1152745493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 msgr2=0x55be60f211f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:18.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84b1d9b640 1 --2- 172.21.15.116:0/1152745493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 0x55be60f211f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84b2d9d640 1 --2- 172.21.15.116:0/1152745493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 0x55be60f1e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84b1d9b640 1 -- 172.21.15.116:0/1152745493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 msgr2=0x55be60f1e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:18.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84b1d9b640 1 --2- 172.21.15.116:0/1152745493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 0x55be60f1e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.281+0000 7f84b1d9b640 1 -- 172.21.15.116:0/1152745493 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be60e25860 con 0x55be60ef10c0 2023-11-28T17:09:18.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.282+0000 7f84b1d9b640 1 --2- 172.21.15.116:0/1152745493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60ef10c0 0x55be60f1e080 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f84b406cea0 tx=0x7f84a8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dfa6e74d0ec3a69 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:18.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.282+0000 7f8493fff640 1 -- 172.21.15.116:0/1152745493 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8014960 con 0x55be60ef10c0 2023-11-28T17:09:18.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.282+0000 7f8493fff640 1 -- 172.21.15.116:0/1152745493 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a8019070 con 0x55be60ef10c0 2023-11-28T17:09:18.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.282+0000 7f8493fff640 1 -- 172.21.15.116:0/1152745493 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8014cc0 con 0x55be60ef10c0 2023-11-28T17:09:18.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.282+0000 7f84bbbf9740 1 -- 172.21.15.116:0/1152745493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60ef10c0 msgr2=0x55be60f1e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.282+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/1152745493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60ef10c0 0x55be60f1e080 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f84b406cea0 tx=0x7f84a8012000 comp rx=0 tx=0).stop 2023-11-28T17:09:18.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.283+0000 7f84bbbf9740 1 -- 172.21.15.116:0/1152745493 shutdown_connections 2023-11-28T17:09:18.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.283+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/1152745493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 0x55be60f1e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.283+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/1152745493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60ef10c0 0x55be60f1e080 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.283+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/1152745493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 0x55be60f211f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.283+0000 7f84bbbf9740 1 -- 172.21.15.116:0/1152745493 >> 172.21.15.116:0/1152745493 conn(0x55be60ef3330 msgr2=0x55be60f3bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:18.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.283+0000 7f84bbbf9740 1 -- 172.21.15.116:0/1152745493 shutdown_connections 2023-11-28T17:09:18.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.283+0000 7f84bbbf9740 1 -- 172.21.15.116:0/1152745493 wait complete. 2023-11-28T17:09:18.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.283+0000 7f84bbbf9740 1 -- start start 2023-11-28T17:09:18.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.284+0000 7f84bbbf9740 1 Processor -- start 2023-11-28T17:09:18.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.284+0000 7f84bbbf9740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60ef4990 0x55be60f4bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84bbbf9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f20e10 0x55be60f081b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84bbbf9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60f211f0 0x55be60f21670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84bbbf9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be60e5dac0 con 0x55be60f20e10 2023-11-28T17:09:18.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84bbbf9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be60dfa2f0 con 0x55be60f211f0 2023-11-28T17:09:18.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84bbbf9740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55be60e17c70 con 0x55be60ef4990 2023-11-28T17:09:18.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84bbbf9740 10 client.0 _ll_register_callbacks cb 0x55be60b6fac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:09:18.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84b1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f20e10 0x55be60f081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84b1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f20e10 0x55be60f081b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56302/0 (socket says 172.21.15.116:56302) 2023-11-28T17:09:18.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84b1d9b640 1 -- 172.21.15.116:0/466163666 learned_addr learned my addr 172.21.15.116:0/466163666 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:18.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84b259c640 1 --2- 172.21.15.116:0/466163666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60ef4990 0x55be60f4bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84b1d9b640 1 -- 172.21.15.116:0/466163666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60ef4990 msgr2=0x55be60f4bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:18.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84b1d9b640 1 --2- 172.21.15.116:0/466163666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60ef4990 0x55be60f4bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84b1d9b640 1 -- 172.21.15.116:0/466163666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60f211f0 msgr2=0x55be60f21670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:18.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84b1d9b640 1 --2- 172.21.15.116:0/466163666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60f211f0 0x55be60f21670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84b1d9b640 1 -- 172.21.15.116:0/466163666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a800e040 con 0x55be60f20e10 2023-11-28T17:09:18.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84b1d9b640 1 --2- 172.21.15.116:0/466163666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f20e10 0x55be60f081b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f84a8014830 tx=0x7f84a8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a685d1d69d255f3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:18.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.285+0000 7f84917fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-28T17:09:18.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.286+0000 7f84917fa640 1 -- 172.21.15.116:0/466163666 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8005da0 con 0x55be60f20e10 2023-11-28T17:09:18.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.286+0000 7f84917fa640 1 -- 172.21.15.116:0/466163666 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84a8006030 con 0x55be60f20e10 2023-11-28T17:09:18.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.286+0000 7f84bbbf9740 1 -- 172.21.15.116:0/466163666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55be60e56aa0 con 0x55be60f20e10 2023-11-28T17:09:18.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.286+0000 7f84917fa640 1 -- 172.21.15.116:0/466163666 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84a80061f0 con 0x55be60f20e10 2023-11-28T17:09:18.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.286+0000 7f84bbbf9740 10 client.4617 fetch_fsmap learned FSMap version 8 2023-11-28T17:09:18.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.286+0000 7f84bbbf9740 1 -- 172.21.15.116:0/466163666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55be60e5c140 con 0x55be60f20e10 2023-11-28T17:09:18.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.286+0000 7f84917fa640 1 -- 172.21.15.116:0/466163666 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55be60e56aa0 con 0x55be60f20e10 2023-11-28T17:09:18.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.286+0000 7f84917fa640 1 -- 172.21.15.116:0/466163666 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f84a8006350 con 0x55be60f20e10 2023-11-28T17:09:18.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.286+0000 7f84bbbf9740 10 client.4617 fetch_fsmap finished waiting for FSMap version 8 2023-11-28T17:09:18.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.286+0000 7f84bbbf9740 10 client.4617 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-28T17:09:18.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.286+0000 7f84bbbf9740 20 client.4617 populate_metadata read hostname 'smithi116' 2023-11-28T17:09:18.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.286+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/466163666 >> [v2:172.21.15.156:6832/3268354971,v1:172.21.15.156:6834/3268354971] conn(0x55be60f1a620 0x55be60f1aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.286+0000 7f84bbbf9740 4 client.4617 mds_command: new command op to 4254 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T17:09:18.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.286+0000 7f84bbbf9740 1 -- 172.21.15.116:0/466163666 --> [v2:172.21.15.156:6832/3268354971,v1:172.21.15.156:6834/3268354971] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55be60e567c0 con 0x55be60f1a620 2023-11-28T17:09:18.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.287+0000 7f84b259c640 1 --2- 172.21.15.116:0/466163666 >> [v2:172.21.15.156:6832/3268354971,v1:172.21.15.156:6834/3268354971] conn(0x55be60f1a620 0x55be60f1aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.288+0000 7f84917fa640 10 client.4617 ms_handle_connect on v2:172.21.15.156:6832/3268354971 2023-11-28T17:09:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.288+0000 7f84b259c640 1 --2- 172.21.15.116:0/466163666 >> [v2:172.21.15.156:6832/3268354971,v1:172.21.15.156:6834/3268354971] conn(0x55be60f1a620 0x55be60f1aa00 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d21d048f15bbe659 server_cookie=b806cc9b2cad8168 in_seq=0 out_seq=0 2023-11-28T17:09:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.290+0000 7f84917fa640 1 -- 172.21.15.116:0/466163666 <== mds.0 v2:172.21.15.156:6832/3268354971 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55be60e567c0 con 0x55be60f1a620 2023-11-28T17:09:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.290+0000 7f84917fa640 10 client.4617 handle_command_reply: tid=0 2023-11-28T17:09:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.290+0000 7f84bbbf9740 1 client.4617 shutdown 2023-11-28T17:09:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.290+0000 7f84bbbf9740 1 -- 172.21.15.116:0/466163666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f20e10 msgr2=0x55be60f081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.290+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/466163666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f20e10 0x55be60f081b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f84a8014830 tx=0x7f84a8008040 comp rx=0 tx=0).stop 2023-11-28T17:09:18.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.290+0000 7f84bbbf9740 1 -- 172.21.15.116:0/466163666 shutdown_connections 2023-11-28T17:09:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.290+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/466163666 >> [v2:172.21.15.156:6832/3268354971,v1:172.21.15.156:6834/3268354971] conn(0x55be60f1a620 0x55be60f1aa00 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.290+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/466163666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60f211f0 0x55be60f21670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.290+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/466163666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f20e10 0x55be60f081b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.290+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/466163666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60ef4990 0x55be60f4bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.290+0000 7f84917fa640 0 client.4617 ms_handle_reset on v2:172.21.15.156:6832/3268354971 2023-11-28T17:09:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.290+0000 7f84bbbf9740 1 -- 172.21.15.116:0/466163666 >> 172.21.15.116:0/466163666 conn(0x55be60ef10c0 msgr2=0x55be60f24b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.291+0000 7f84bbbf9740 1 -- 172.21.15.116:0/466163666 shutdown_connections 2023-11-28T17:09:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.291+0000 7f84bbbf9740 1 -- 172.21.15.116:0/466163666 wait complete. 2023-11-28T17:09:18.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.291+0000 7f84bbbf9740 20 client.4617 trim_cache size 0 max 16384 2023-11-28T17:09:18.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.307+0000 7f84bbbf9740 1 Processor -- start 2023-11-28T17:09:18.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.308+0000 7f84bbbf9740 1 -- start start 2023-11-28T17:09:18.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.308+0000 7f84bbbf9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 0x55be60f1ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.308+0000 7f84bbbf9740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 0x55be60f1e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.308+0000 7f84bbbf9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f211f0 0x55be60ef01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.308+0000 7f84bbbf9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be60bc0f20 con 0x55be60f211f0 2023-11-28T17:09:18.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.308+0000 7f84bbbf9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be60ef7a00 con 0x55be60ef4990 2023-11-28T17:09:18.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.308+0000 7f84bbbf9740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55be60f3ea90 con 0x55be60f20e10 2023-11-28T17:09:18.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.309+0000 7f84b2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f211f0 0x55be60ef01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.309+0000 7f84b2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f211f0 0x55be60ef01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56314/0 (socket says 172.21.15.116:56314) 2023-11-28T17:09:18.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.309+0000 7f84b2d9d640 1 -- 172.21.15.116:0/570202538 learned_addr learned my addr 172.21.15.116:0/570202538 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:18.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.309+0000 7f84b2d9d640 1 -- 172.21.15.116:0/570202538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 msgr2=0x55be60f1e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:18.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.309+0000 7f84b259c640 1 --2- 172.21.15.116:0/570202538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 0x55be60f1ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.309+0000 7f84b1d9b640 1 --2- 172.21.15.116:0/570202538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 0x55be60f1e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.309+0000 7f84b2d9d640 1 --2- 172.21.15.116:0/570202538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 0x55be60f1e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.309+0000 7f84b2d9d640 1 -- 172.21.15.116:0/570202538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 msgr2=0x55be60f1ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:18.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.309+0000 7f84b2d9d640 1 --2- 172.21.15.116:0/570202538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 0x55be60f1ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.309+0000 7f84b2d9d640 1 -- 172.21.15.116:0/570202538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be60df9b80 con 0x55be60f211f0 2023-11-28T17:09:18.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.309+0000 7f84b2d9d640 1 --2- 172.21.15.116:0/570202538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f211f0 0x55be60ef01c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00a8f0 tx=0x7f84ac009050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50b0ebcb5bcea4b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:18.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.310+0000 7f84a0ff9640 1 -- 172.21.15.116:0/570202538 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac0072d0 con 0x55be60f211f0 2023-11-28T17:09:18.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.310+0000 7f84a0ff9640 1 -- 172.21.15.116:0/570202538 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ac0017b0 con 0x55be60f211f0 2023-11-28T17:09:18.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.310+0000 7f84a0ff9640 1 -- 172.21.15.116:0/570202538 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac004b60 con 0x55be60f211f0 2023-11-28T17:09:18.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.310+0000 7f84bbbf9740 1 -- 172.21.15.116:0/570202538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f211f0 msgr2=0x55be60ef01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.310+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/570202538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f211f0 0x55be60ef01c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00a8f0 tx=0x7f84ac009050 comp rx=0 tx=0).stop 2023-11-28T17:09:18.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.310+0000 7f84bbbf9740 1 -- 172.21.15.116:0/570202538 shutdown_connections 2023-11-28T17:09:18.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.310+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/570202538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f211f0 0x55be60ef01c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.310+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/570202538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 0x55be60f1e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.310+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/570202538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 0x55be60f1ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.310+0000 7f84bbbf9740 1 -- 172.21.15.116:0/570202538 >> 172.21.15.116:0/570202538 conn(0x55be60ef10c0 msgr2=0x55be60f19640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:18.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.310+0000 7f84bbbf9740 1 -- 172.21.15.116:0/570202538 shutdown_connections 2023-11-28T17:09:18.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.310+0000 7f84bbbf9740 1 -- 172.21.15.116:0/570202538 wait complete. 2023-11-28T17:09:18.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.311+0000 7f84bbbf9740 1 -- start start 2023-11-28T17:09:18.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.311+0000 7f84bbbf9740 1 Processor -- start 2023-11-28T17:09:18.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.312+0000 7f84bbbf9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 0x55be60fa6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.312+0000 7f84bbbf9740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 0x55be60fa73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.312+0000 7f84bbbf9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f211f0 0x55be60f149f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.312+0000 7f84bbbf9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be60f25090 con 0x55be60f211f0 2023-11-28T17:09:18.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.312+0000 7f84bbbf9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be60fa7910 con 0x55be60ef4990 2023-11-28T17:09:18.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.312+0000 7f84bbbf9740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55be60faa670 con 0x55be60f20e10 2023-11-28T17:09:18.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.312+0000 7f84bbbf9740 10 client.0 _ll_register_callbacks cb 0x7f84a8009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:09:18.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.312+0000 7f84b2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f211f0 0x55be60f149f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.312+0000 7f84b2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f211f0 0x55be60f149f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56324/0 (socket says 172.21.15.116:56324) 2023-11-28T17:09:18.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.312+0000 7f84b2d9d640 1 -- 172.21.15.116:0/4098008136 learned_addr learned my addr 172.21.15.116:0/4098008136 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:18.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.313+0000 7f84b1d9b640 1 --2- 172.21.15.116:0/4098008136 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 0x55be60fa73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.313+0000 7f84b259c640 1 --2- 172.21.15.116:0/4098008136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 0x55be60fa6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.313+0000 7f84b2d9d640 1 -- 172.21.15.116:0/4098008136 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 msgr2=0x55be60fa73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:18.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.313+0000 7f84b2d9d640 1 --2- 172.21.15.116:0/4098008136 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 0x55be60fa73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.313+0000 7f84b2d9d640 1 -- 172.21.15.116:0/4098008136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 msgr2=0x55be60fa6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:18.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.313+0000 7f84b2d9d640 1 --2- 172.21.15.116:0/4098008136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 0x55be60fa6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.313+0000 7f84b2d9d640 1 -- 172.21.15.116:0/4098008136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f849c003520 con 0x55be60f211f0 2023-11-28T17:09:18.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.313+0000 7f84b259c640 1 --2- 172.21.15.116:0/4098008136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 0x55be60fa6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:18.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.313+0000 7f84b2d9d640 1 --2- 172.21.15.116:0/4098008136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f211f0 0x55be60f149f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f84ac007040 tx=0x7f84ac033000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45813d5539bf8776 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:18.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.313+0000 7f84917fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-28T17:09:18.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.313+0000 7f84917fa640 1 -- 172.21.15.116:0/4098008136 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac050070 con 0x55be60f211f0 2023-11-28T17:09:18.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.313+0000 7f84917fa640 1 -- 172.21.15.116:0/4098008136 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84ac0075a0 con 0x55be60f211f0 2023-11-28T17:09:18.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.313+0000 7f84bbbf9740 1 -- 172.21.15.116:0/4098008136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84a8019070 con 0x55be60f211f0 2023-11-28T17:09:18.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.314+0000 7f84917fa640 1 -- 172.21.15.116:0/4098008136 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84ac001fa0 con 0x55be60f211f0 2023-11-28T17:09:18.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.314+0000 7f84bbbf9740 10 client.4623 fetch_fsmap learned FSMap version 8 2023-11-28T17:09:18.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.314+0000 7f84bbbf9740 1 -- 172.21.15.116:0/4098008136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55be60e1f170 con 0x55be60f211f0 2023-11-28T17:09:18.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.314+0000 7f84917fa640 1 -- 172.21.15.116:0/4098008136 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a8019070 con 0x55be60f211f0 2023-11-28T17:09:18.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.314+0000 7f84917fa640 1 -- 172.21.15.116:0/4098008136 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f84ac053030 con 0x55be60f211f0 2023-11-28T17:09:18.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.314+0000 7f84bbbf9740 10 client.4623 fetch_fsmap finished waiting for FSMap version 8 2023-11-28T17:09:18.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.314+0000 7f84bbbf9740 10 client.4623 resolve_mds: resolved name 'b' to daemon mds.b 2023-11-28T17:09:18.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.314+0000 7f84bbbf9740 20 client.4623 populate_metadata read hostname 'smithi116' 2023-11-28T17:09:18.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.314+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/4098008136 >> [v2:172.21.15.156:6832/3268354971,v1:172.21.15.156:6834/3268354971] conn(0x55be60ef01c0 0x55be60ef05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.314+0000 7f84bbbf9740 4 client.4623 mds_command: new command op to 4254 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T17:09:18.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.314+0000 7f84bbbf9740 1 -- 172.21.15.116:0/4098008136 --> [v2:172.21.15.156:6832/3268354971,v1:172.21.15.156:6834/3268354971] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55be60b5e6c0 con 0x55be60ef01c0 2023-11-28T17:09:18.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.314+0000 7f84b259c640 1 --2- 172.21.15.116:0/4098008136 >> [v2:172.21.15.156:6832/3268354971,v1:172.21.15.156:6834/3268354971] conn(0x55be60ef01c0 0x55be60ef05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.315+0000 7f84917fa640 10 client.4623 ms_handle_connect on v2:172.21.15.156:6832/3268354971 2023-11-28T17:09:18.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.315+0000 7f84b259c640 1 --2- 172.21.15.116:0/4098008136 >> [v2:172.21.15.156:6832/3268354971,v1:172.21.15.156:6834/3268354971] conn(0x55be60ef01c0 0x55be60ef05a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e3b96187b09b64f server_cookie=b05048e08c82b470 in_seq=0 out_seq=0 2023-11-28T17:09:18.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.316+0000 7f84917fa640 1 -- 172.21.15.116:0/4098008136 <== mds.0 v2:172.21.15.156:6832/3268354971 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55be60b5e6c0 con 0x55be60ef01c0 2023-11-28T17:09:18.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.316+0000 7f84917fa640 10 client.4623 handle_command_reply: tid=0 2023-11-28T17:09:18.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.316+0000 7f84bbbf9740 1 client.4623 shutdown 2023-11-28T17:09:18.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.316+0000 7f84bbbf9740 1 -- 172.21.15.116:0/4098008136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f211f0 msgr2=0x55be60f149f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.316+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/4098008136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f211f0 0x55be60f149f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f84ac007040 tx=0x7f84ac033000 comp rx=0 tx=0).stop 2023-11-28T17:09:18.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.316+0000 7f84bbbf9740 1 -- 172.21.15.116:0/4098008136 shutdown_connections 2023-11-28T17:09:18.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.316+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/4098008136 >> [v2:172.21.15.156:6832/3268354971,v1:172.21.15.156:6834/3268354971] conn(0x55be60ef01c0 0x55be60ef05a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.316+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/4098008136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be60f211f0 0x55be60f149f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.316+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/4098008136 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55be60f20e10 0x55be60fa73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.316+0000 7f84bbbf9740 1 --2- 172.21.15.116:0/4098008136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be60ef4990 0x55be60fa6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.316+0000 7f84917fa640 0 client.4623 ms_handle_reset on v2:172.21.15.156:6832/3268354971 2023-11-28T17:09:18.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.316+0000 7f84bbbf9740 1 -- 172.21.15.116:0/4098008136 >> 172.21.15.116:0/4098008136 conn(0x55be60ef10c0 msgr2=0x55be60ef6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:18.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.316+0000 7f84bbbf9740 1 -- 172.21.15.116:0/4098008136 shutdown_connections 2023-11-28T17:09:18.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.316+0000 7f84bbbf9740 1 -- 172.21.15.116:0/4098008136 wait complete. 2023-11-28T17:09:18.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.317+0000 7f84bbbf9740 20 client.4623 trim_cache size 0 max 16384 2023-11-28T17:09:18.322 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-28T17:09:18.322 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a5f2fbcf-fde4-4c38-94da-08178784d9fc", 2023-11-28T17:09:18.322 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2023-11-28T17:09:18.322 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 4254, 2023-11-28T17:09:18.323 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2023-11-28T17:09:18.323 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2023-11-28T17:09:18.323 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2023-11-28T17:09:18.323 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 17.067105552000001, 2023-11-28T17:09:18.323 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 8, 2023-11-28T17:09:18.323 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 20, 2023-11-28T17:09:18.323 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 20, 2023-11-28T17:09:18.323 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 22.102839132 2023-11-28T17:09:18.323 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-28T17:09:18.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.317+0000 7f84b8f89640 1 -- 172.21.15.116:0/3037761741 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f849406b2a0 msgr2=0x7f849406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.317+0000 7f84b8f89640 1 --2- 172.21.15.116:0/3037761741 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f849406b2a0 0x7f849406d760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f84a8006e80 tx=0x7f84a8006df0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.317+0000 7f84b8f89640 1 -- 172.21.15.116:0/3037761741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f849c0051d0 msgr2=0x7f84b40b0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.317+0000 7f84b8f89640 1 --2- 172.21.15.116:0/3037761741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f849c0051d0 0x7f84b40b0530 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f84a400e580 tx=0x7f84a400ea40 comp rx=0 tx=0).stop 2023-11-28T17:09:18.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.317+0000 7f84b8f89640 1 -- 172.21.15.116:0/3037761741 shutdown_connections 2023-11-28T17:09:18.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.317+0000 7f84b8f89640 1 --2- 172.21.15.116:0/3037761741 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f849406b2a0 0x7f849406d760 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.317+0000 7f84b8f89640 1 --2- 172.21.15.116:0/3037761741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84b40b3970 0x7f84b413b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.317+0000 7f84b8f89640 1 --2- 172.21.15.116:0/3037761741 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b413a0e0 0x7f84b40b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.317+0000 7f84b8f89640 1 --2- 172.21.15.116:0/3037761741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f849c0051d0 0x7f84b40b0530 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.317+0000 7f84b8f89640 1 -- 172.21.15.116:0/3037761741 >> 172.21.15.116:0/3037761741 conn(0x7f84b406a4c0 msgr2=0x7f84b40ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:18.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.317+0000 7f84b8f89640 1 -- 172.21.15.116:0/3037761741 shutdown_connections 2023-11-28T17:09:18.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.317+0000 7f84b8f89640 1 -- 172.21.15.116:0/3037761741 wait complete. 2023-11-28T17:09:18.335 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-11-28T17:09:18.361 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-11-28T17:09:18.361 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-11-28T17:09:18.361 INFO:tasks.ceph_fuse:client.0 config is {} 2023-11-28T17:09:18.362 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T17:09:18.362 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T17:09:18.362 INFO:tasks.ceph_fuse:client.2 config is {} 2023-11-28T17:09:18.362 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T17:09:18.362 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-11-28T17:09:18.362 INFO:tasks.ceph_fuse:client.1 config is {} 2023-11-28T17:09:18.362 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T17:09:18.362 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-28T17:09:18.362 INFO:tasks.ceph_fuse:client.3 config is {} 2023-11-28T17:09:18.362 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-28T17:09:18.363 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-11-28T17:09:18.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:18.363 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2023-11-28T17:09:18.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:18.428 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link delete ceph-brx 2023-11-28T17:09:18.530 INFO:teuthology.orchestra.run.smithi156.stderr:Cannot find device "ceph-brx" 2023-11-28T17:09:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:09:18.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:18.534 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2023-11-28T17:09:18.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:18.594 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2023-11-28T17:09:18.693 INFO:teuthology.orchestra.run.smithi116.stderr:Cannot find device "ceph-brx" 2023-11-28T17:09:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:09:18.697 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-11-28T17:09:18.698 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T17:09:18.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.910+0000 7fdc6ffff640 1 Processor -- start 2023-11-28T17:09:18.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.910+0000 7fdc6ffff640 1 -- start start 2023-11-28T17:09:18.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.912+0000 7fdc6ffff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc700cf2a0 0x7fdc700701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.912+0000 7fdc6ffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdc7015ae90 0x7fdc700ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.912+0000 7fdc6ffff640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc700cd8e0 con 0x7fdc70159f30 2023-11-28T17:09:18.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.912+0000 7fdc6ffff640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc70070730 con 0x7fdc7015ae90 2023-11-28T17:09:18.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.912+0000 7fdc6ffff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc70070ee0 con 0x7fdc700cf2a0 2023-11-28T17:09:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.913+0000 7fdc6e7fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc700cf2a0 0x7fdc700701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.913+0000 7fdc6e7fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc700cf2a0 0x7fdc700701f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:50296/0 (socket says 172.21.15.116:50296) 2023-11-28T17:09:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.913+0000 7fdc6e7fc640 1 -- 172.21.15.116:0/887787567 learned_addr learned my addr 172.21.15.116:0/887787567 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.913+0000 7fdc6f7fe640 1 --2- 172.21.15.116:0/887787567 >> v2:172.21.15.156:3300/0 conn(0x7fdc7015ae90 0x7fdc700ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.913+0000 7fdc6e7fc640 1 -- 172.21.15.116:0/887787567 >> v2:172.21.15.156:3300/0 conn(0x7fdc7015ae90 msgr2=0x7fdc700ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.913+0000 7fdc6e7fc640 1 --2- 172.21.15.116:0/887787567 >> v2:172.21.15.156:3300/0 conn(0x7fdc7015ae90 0x7fdc700ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.913+0000 7fdc6e7fc640 1 -- 172.21.15.116:0/887787567 >> v1:172.21.15.156:6789/0 conn(0x7fdc70159f30 legacy=0x7fdc700ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.913+0000 7fdc6e7fc640 1 -- 172.21.15.116:0/887787567 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7015b310 con 0x7fdc700cf2a0 2023-11-28T17:09:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.915+0000 7fdc6e7fc640 1 --2- 172.21.15.116:0/887787567 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc700cf2a0 0x7fdc700701f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdc64009870 tx=0x7fdc6402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7507c4014d8d2d39 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:18.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.915+0000 7fdc6dffb640 1 -- 172.21.15.116:0/887787567 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc64030840 con 0x7fdc700cf2a0 2023-11-28T17:09:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.915+0000 7fdc6dffb640 1 -- 172.21.15.116:0/887787567 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc640309a0 con 0x7fdc700cf2a0 2023-11-28T17:09:18.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.915+0000 7fdc6dffb640 1 -- 172.21.15.116:0/887787567 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc64030d30 con 0x7fdc700cf2a0 2023-11-28T17:09:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.916+0000 7fdc6ffff640 1 -- 172.21.15.116:0/887787567 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc700cf2a0 msgr2=0x7fdc700701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.916+0000 7fdc6ffff640 1 --2- 172.21.15.116:0/887787567 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc700cf2a0 0x7fdc700701f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdc64009870 tx=0x7fdc6402b980 comp rx=0 tx=0).stop 2023-11-28T17:09:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.916+0000 7fdc6ffff640 1 -- 172.21.15.116:0/887787567 shutdown_connections 2023-11-28T17:09:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.916+0000 7fdc6ffff640 1 --2- 172.21.15.116:0/887787567 >> v2:172.21.15.156:3300/0 conn(0x7fdc7015ae90 0x7fdc700ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.916+0000 7fdc6ffff640 1 --2- 172.21.15.116:0/887787567 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc700cf2a0 0x7fdc700701f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.916+0000 7fdc6ffff640 1 -- 172.21.15.116:0/887787567 >> 172.21.15.116:0/887787567 conn(0x7fdc700b0950 msgr2=0x7fdc700b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.916+0000 7fdc6ffff640 1 -- 172.21.15.116:0/887787567 shutdown_connections 2023-11-28T17:09:18.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.916+0000 7fdc6ffff640 1 -- 172.21.15.116:0/887787567 wait complete. 2023-11-28T17:09:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.918+0000 7fdc6ffff640 1 Processor -- start 2023-11-28T17:09:18.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.918+0000 7fdc6ffff640 1 -- start start 2023-11-28T17:09:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.919+0000 7fdc6ffff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc700cf2a0 0x7fdc7015e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.919+0000 7fdc6ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc70159f30 0x7fdc7009bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.919+0000 7fdc6ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc7015ae90 0x7fdc701422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.919+0000 7fdc6ffff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7009aa50 con 0x7fdc7015ae90 2023-11-28T17:09:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.919+0000 7fdc6ffff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7009b8b0 con 0x7fdc70159f30 2023-11-28T17:09:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.919+0000 7fdc6ffff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7009aca0 con 0x7fdc700cf2a0 2023-11-28T17:09:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.920+0000 7fdc6f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc7015ae90 0x7fdc701422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.920+0000 7fdc6f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc7015ae90 0x7fdc701422d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56340/0 (socket says 172.21.15.116:56340) 2023-11-28T17:09:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.920+0000 7fdc6f7fe640 1 -- 172.21.15.116:0/1821440506 learned_addr learned my addr 172.21.15.116:0/1821440506 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:18.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.920+0000 7fdc6effd640 1 --2- 172.21.15.116:0/1821440506 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc700cf2a0 0x7fdc7015e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.920+0000 7fdc6e7fc640 1 --2- 172.21.15.116:0/1821440506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc70159f30 0x7fdc7009bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.920+0000 7fdc6f7fe640 1 -- 172.21.15.116:0/1821440506 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc700cf2a0 msgr2=0x7fdc7015e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.920+0000 7fdc6f7fe640 1 --2- 172.21.15.116:0/1821440506 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc700cf2a0 0x7fdc7015e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.920+0000 7fdc6f7fe640 1 -- 172.21.15.116:0/1821440506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc70159f30 msgr2=0x7fdc7009bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.920+0000 7fdc6f7fe640 1 --2- 172.21.15.116:0/1821440506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc70159f30 0x7fdc7009bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.920+0000 7fdc6f7fe640 1 -- 172.21.15.116:0/1821440506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc640094d0 con 0x7fdc7015ae90 2023-11-28T17:09:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.920+0000 7fdc6f7fe640 1 --2- 172.21.15.116:0/1821440506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc7015ae90 0x7fdc701422d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009dd0 tx=0x7fdc6800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.921+0000 7fdc577fe640 1 -- 172.21.15.116:0/1821440506 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68014480 con 0x7fdc7015ae90 2023-11-28T17:09:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.921+0000 7fdc577fe640 1 -- 172.21.15.116:0/1821440506 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc6800f040 con 0x7fdc7015ae90 2023-11-28T17:09:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.921+0000 7fdc6ffff640 1 -- 172.21.15.116:0/1821440506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7009af80 con 0x7fdc7015ae90 2023-11-28T17:09:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.921+0000 7fdc577fe640 1 -- 172.21.15.116:0/1821440506 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68014840 con 0x7fdc7015ae90 2023-11-28T17:09:18.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.921+0000 7fdc6ffff640 1 -- 172.21.15.116:0/1821440506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7009b540 con 0x7fdc7015ae90 2023-11-28T17:09:18.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.922+0000 7fdc6ffff640 1 -- 172.21.15.116:0/1821440506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7009ba40 con 0x7fdc7015ae90 2023-11-28T17:09:18.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.923+0000 7fdc577fe640 1 -- 172.21.15.116:0/1821440506 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fdc680149a0 con 0x7fdc7015ae90 2023-11-28T17:09:18.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.924+0000 7fdc577fe640 1 --2- 172.21.15.116:0/1821440506 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdc5c06b2a0 0x7fdc5c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:18.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.924+0000 7fdc577fe640 1 -- 172.21.15.116:0/1821440506 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc6808a7b0 con 0x7fdc7015ae90 2023-11-28T17:09:18.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.924+0000 7fdc6effd640 1 --2- 172.21.15.116:0/1821440506 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdc5c06b2a0 0x7fdc5c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:18.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.925+0000 7fdc6effd640 1 --2- 172.21.15.116:0/1821440506 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdc5c06b2a0 0x7fdc5c06d760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdc60007fd0 tx=0x7fdc60009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:18.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:18.928+0000 7fdc577fe640 1 -- 172.21.15.116:0/1821440506 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdc68055600 con 0x7fdc7015ae90 2023-11-28T17:09:19.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:19.156+0000 7fdc6ffff640 1 -- 172.21.15.116:0/1821440506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdc7009d0b0 con 0x7fdc7015ae90 2023-11-28T17:09:19.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:19.157+0000 7fdc577fe640 1 -- 172.21.15.116:0/1821440506 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fdc68018060 con 0x7fdc7015ae90 2023-11-28T17:09:19.158 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T17:09:19.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:19.159+0000 7fdc6ffff640 1 -- 172.21.15.116:0/1821440506 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdc5c06b2a0 msgr2=0x7fdc5c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:19.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:19.159+0000 7fdc6ffff640 1 --2- 172.21.15.116:0/1821440506 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdc5c06b2a0 0x7fdc5c06d760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdc60007fd0 tx=0x7fdc60009040 comp rx=0 tx=0).stop 2023-11-28T17:09:19.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:19.159+0000 7fdc6ffff640 1 -- 172.21.15.116:0/1821440506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc7015ae90 msgr2=0x7fdc701422d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:19.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:19.159+0000 7fdc6ffff640 1 --2- 172.21.15.116:0/1821440506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc7015ae90 0x7fdc701422d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009dd0 tx=0x7fdc6800c490 comp rx=0 tx=0).stop 2023-11-28T17:09:19.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:19.160+0000 7fdc6ffff640 1 -- 172.21.15.116:0/1821440506 shutdown_connections 2023-11-28T17:09:19.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:19.160+0000 7fdc6ffff640 1 --2- 172.21.15.116:0/1821440506 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdc5c06b2a0 0x7fdc5c06d760 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:19.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:19.160+0000 7fdc6ffff640 1 --2- 172.21.15.116:0/1821440506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc7015ae90 0x7fdc701422d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:19.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:19.160+0000 7fdc6ffff640 1 --2- 172.21.15.116:0/1821440506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc70159f30 0x7fdc7009bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:19.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:19.160+0000 7fdc6ffff640 1 --2- 172.21.15.116:0/1821440506 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc700cf2a0 0x7fdc7015e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:19.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:19.160+0000 7fdc6ffff640 1 -- 172.21.15.116:0/1821440506 >> 172.21.15.116:0/1821440506 conn(0x7fdc700b0950 msgr2=0x7fdc700b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:19.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:19.160+0000 7fdc6ffff640 1 -- 172.21.15.116:0/1821440506 shutdown_connections 2023-11-28T17:09:19.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:19.160+0000 7fdc6ffff640 1 -- 172.21.15.116:0/1821440506 wait complete. 2023-11-28T17:09:19.180 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T17:09:19.180 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T17:09:19.181 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2023-11-28T17:09:19.181 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T17:09:19.181 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T17:09:19.181 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T17:09:19.181 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T17:09:19.181 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T17:09:19.181 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T17:09:19.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:19.181 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2023-11-28T17:09:19.236 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T17:09:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T17:09:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T17:09:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2023-11-28T17:09:19.236 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:19.237 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:09:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2023-11-28T17:09:19.237 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:09:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2023-11-28T17:09:19.237 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:09:19.237 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:19.238 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T17:09:19.238 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:19.238 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T17:09:19.238 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42379sec preferred_lft 42379sec 2023-11-28T17:09:19.238 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2023-11-28T17:09:19.238 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:19.238 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-28T17:09:19.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:19.239 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-28T17:09:19.239 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-28T17:09:19.239 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add name ceph-brx type bridge 2023-11-28T17:09:19.239 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr flush dev ceph-brx 2023-11-28T17:09:19.239 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set ceph-brx up 2023-11-28T17:09:19.239 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-28T17:09:19.239 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-28T17:09:19.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:19.704 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T17:09:19.783 INFO:teuthology.orchestra.run.smithi116.stdout:1 2023-11-28T17:09:19.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:19.787 DEBUG:teuthology.orchestra.run.smithi116:> route 2023-11-28T17:09:19.838 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2023-11-28T17:09:19.838 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T17:09:19.839 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T17:09:19.839 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T17:09:19.839 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T17:09:19.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:19.840 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-28T17:09:19.840 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-28T17:09:19.840 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T17:09:19.840 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T17:09:19.840 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T17:09:19.840 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-28T17:09:20.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:20.629 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2023-11-28T17:09:20.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:20.686 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2023-11-28T17:09:20.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:20.740 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-28T17:09:20.740 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-28T17:09:20.740 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:09:20.740 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-11-28T17:09:20.741 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-28T17:09:20.877 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-11-28T17:09:20.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:20.877 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-28T17:09:20.878 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-28T17:09:20.878 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-11-28T17:09:20.878 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-28T17:09:20.878 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-11-28T17:09:20.878 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-11-28T17:09:20.878 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-11-28T17:09:20.878 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-28T17:09:21.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:21.158 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-28T17:09:21.158 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-28T17:09:21.158 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2023-11-28T17:09:21.158 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.0 master ceph-brx 2023-11-28T17:09:21.158 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-28T17:09:21.320 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-11-28T17:09:21.321 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:09:21.321 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T17:09:21.367 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T17:09:21.367 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:09:21.367 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T17:09:21.463 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-28T17:09:21.569 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:09:21.621 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2023-11-28T17:09:21.621 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2023-11-28T17:09:21.621 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2023-11-28T17:09:21.621 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2023-11-28T17:09:21.622 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2023-11-28T17:09:21.622 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2023-11-28T17:09:21.622 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2023-11-28T17:09:21.622 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2023-11-28T17:09:21.622 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2023-11-28T17:09:21.622 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2023-11-28T17:09:21.622 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2023-11-28T17:09:21.623 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:09:21.623 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2023-11-28T17:09:21.623 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2023-11-28T17:09:21.623 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2023-11-28T17:09:21.623 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2023-11-28T17:09:21.623 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2023-11-28T17:09:21.623 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2023-11-28T17:09:21.623 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:09:21.624 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:09:21.624 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:09:21.624 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:09:21.624 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2023-11-28T17:09:21.624 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2023-11-28T17:09:21.624 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T17:09:21.624 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T17:09:21.624 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T17:09:21.625 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T17:09:21.625 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2023-11-28T17:09:21.625 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:09:21.625 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:09:21.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:21.626 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-28T17:09:21.673 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-28T17:09:21.673 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-11-28T17:09:21.722 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-28T17:09:21.803 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:09:21.846 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2023-11-28T17:09:21.846 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2023-11-28T17:09:21.847 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2023-11-28T17:09:21.847 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2023-11-28T17:09:21.847 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2023-11-28T17:09:21.847 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2023-11-28T17:09:21.847 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2023-11-28T17:09:21.847 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2023-11-28T17:09:21.847 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2023-11-28T17:09:21.848 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2023-11-28T17:09:21.848 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2023-11-28T17:09:21.848 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:09:21.848 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2023-11-28T17:09:21.848 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2023-11-28T17:09:21.848 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2023-11-28T17:09:21.848 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2023-11-28T17:09:21.848 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2023-11-28T17:09:21.849 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2023-11-28T17:09:21.849 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:09:21.849 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:09:21.849 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:09:21.849 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:09:21.849 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2023-11-28T17:09:21.849 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2023-11-28T17:09:21.849 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T17:09:21.849 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T17:09:21.849 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T17:09:21.850 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T17:09:21.850 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2023-11-28T17:09:21.850 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:09:21.850 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:09:21.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:21.851 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-28T17:09:21.915 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:2023-11-28T17:09:21.913+0000 7f72ca967480 -1 init, newargv = 0x558d92761ca0 newargc=15 2023-11-28T17:09:21.915 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[96908]: starting ceph client 2023-11-28T17:09:21.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[96908]: starting fuse 2023-11-28T17:09:21.937 INFO:teuthology.orchestra.run.smithi116.stdout:39 2023-11-28T17:09:21.938 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2023-11-28T17:09:21.938 DEBUG:teuthology.orchestra.run.smithi116:> sudo stdin-killer -- python3 -c ' 2023-11-28T17:09:21.938 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2023-11-28T17:09:21.938 DEBUG:teuthology.orchestra.run.smithi116:> import re 2023-11-28T17:09:21.939 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-28T17:09:21.939 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2023-11-28T17:09:21.939 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:09:21.939 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2023-11-28T17:09:21.939 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-11-28T17:09:21.939 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2023-11-28T17:09:21.939 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-11-28T17:09:21.939 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:09:21.939 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2023-11-28T17:09:21.939 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2023-11-28T17:09:21.939 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2023-11-28T17:09:21.939 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2023-11-28T17:09:21.939 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:09:21.939 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2023-11-28T17:09:21.940 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-28T17:09:21.940 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-28T17:09:21.940 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-28T17:09:21.940 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2023-11-28T17:09:21.940 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2023-11-28T17:09:21.940 DEBUG:teuthology.orchestra.run.smithi116:> return f 2023-11-28T17:09:21.940 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-28T17:09:21.940 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:09:21.940 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.0")) 2023-11-28T17:09:21.941 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-28T17:09:22.142 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.0.96908.asok 2023-11-28T17:09:22.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-11-28T17:09:22.159 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.96908.asok 2023-11-28T17:09:22.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:22.159 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.96908.asok status 2023-11-28T17:09:22.382 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:09:22.391 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:09:22.392 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-28T17:09:22.392 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2023-11-28T17:09:22.392 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "47c1c547372ed5bff7984647dc200bac0e019caf", 2023-11-28T17:09:22.392 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev)", 2023-11-28T17:09:22.392 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "0", 2023-11-28T17:09:22.392 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2023-11-28T17:09:22.393 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-11-28T17:09:22.393 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "96908", 2023-11-28T17:09:22.393 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2023-11-28T17:09:22.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:09:22.393 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2023-11-28T17:09:22.393 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2023-11-28T17:09:22.393 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 4632, 2023-11-28T17:09:22.393 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2023-11-28T17:09:22.393 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2023-11-28T17:09:22.393 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2023-11-28T17:09:22.393 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4632 2023-11-28T17:09:22.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:09:22.394 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-28T17:09:22.394 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-28T17:09:22.394 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-28T17:09:22.394 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 246812502 2023-11-28T17:09:22.394 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:09:22.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:09:22.394 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-28T17:09:22.394 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-28T17:09:22.394 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-28T17:09:22.394 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 246812502 2023-11-28T17:09:22.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:09:22.394 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.4632 192.168.0.1:0/246812502", 2023-11-28T17:09:22.394 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.1:0/246812502", 2023-11-28T17:09:22.395 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2023-11-28T17:09:22.395 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 8, 2023-11-28T17:09:22.395 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 20, 2023-11-28T17:09:22.395 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2023-11-28T17:09:22.395 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2023-11-28T17:09:22.395 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2023-11-28T17:09:22.395 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-28T17:09:22.406 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T17:09:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.631+0000 7f4649051640 1 Processor -- start 2023-11-28T17:09:22.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.631+0000 7f4649051640 1 -- start start 2023-11-28T17:09:22.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.632+0000 7f4649051640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f464413cee0 0x7f464406c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:22.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.632+0000 7f4649051640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4644140200 0x7f464406c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:22.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.632+0000 7f4649051640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46440aea20 con 0x7f464415a490 2023-11-28T17:09:22.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.632+0000 7f4649051640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464413d2c0 con 0x7f464413cee0 2023-11-28T17:09:22.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.632+0000 7f4649051640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464415b3a0 con 0x7f4644140200 2023-11-28T17:09:22.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.632+0000 7f4642d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f464413cee0 0x7f464406c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:22.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.632+0000 7f464359e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f464415a490 0x7f4644071500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41948/0 (socket says 172.21.15.116:41948) 2023-11-28T17:09:22.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.632+0000 7f464359e640 1 -- 172.21.15.116:0/3509560912 learned_addr learned my addr 172.21.15.116:0/3509560912 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:22.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.632+0000 7f464259c640 1 --2- 172.21.15.116:0/3509560912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4644140200 0x7f464406c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:22.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.632+0000 7f4641d9b640 1 -- 172.21.15.116:0/3509560912 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092428646 0 0) 0x7f46440aea20 con 0x7f464415a490 2023-11-28T17:09:22.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.632+0000 7f4642d9d640 1 -- 172.21.15.116:0/3509560912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4644140200 msgr2=0x7f464406c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:22.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.632+0000 7f4642d9d640 1 --2- 172.21.15.116:0/3509560912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4644140200 0x7f464406c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:22.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.632+0000 7f4642d9d640 1 -- 172.21.15.116:0/3509560912 >> v1:172.21.15.116:6789/0 conn(0x7f464415a490 legacy=0x7f4644071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:22.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.632+0000 7f4642d9d640 1 -- 172.21.15.116:0/3509560912 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464415b0c0 con 0x7f464413cee0 2023-11-28T17:09:22.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.634+0000 7f4642d9d640 1 --2- 172.21.15.116:0/3509560912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f464413cee0 0x7f464406c250 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4634009d90 tx=0x7f463402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24270c5a6ed0157 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:22.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.634+0000 7f4641d9b640 1 -- 172.21.15.116:0/3509560912 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463402cbc0 con 0x7f464413cee0 2023-11-28T17:09:22.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.634+0000 7f4641d9b640 1 -- 172.21.15.116:0/3509560912 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463402cd20 con 0x7f464413cee0 2023-11-28T17:09:22.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.634+0000 7f4641d9b640 1 -- 172.21.15.116:0/3509560912 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4634034660 con 0x7f464413cee0 2023-11-28T17:09:22.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.634+0000 7f4649051640 1 -- 172.21.15.116:0/3509560912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f464413cee0 msgr2=0x7f464406c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.634+0000 7f4649051640 1 --2- 172.21.15.116:0/3509560912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f464413cee0 0x7f464406c250 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4634009d90 tx=0x7f463402be10 comp rx=0 tx=0).stop 2023-11-28T17:09:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.636+0000 7f4649051640 1 -- 172.21.15.116:0/3509560912 shutdown_connections 2023-11-28T17:09:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.636+0000 7f4649051640 1 --2- 172.21.15.116:0/3509560912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4644140200 0x7f464406c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.636+0000 7f4649051640 1 --2- 172.21.15.116:0/3509560912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f464413cee0 0x7f464406c250 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.636+0000 7f4649051640 1 -- 172.21.15.116:0/3509560912 >> 172.21.15.116:0/3509560912 conn(0x7f464406a4c0 msgr2=0x7f464406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.636+0000 7f4649051640 1 -- 172.21.15.116:0/3509560912 shutdown_connections 2023-11-28T17:09:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.636+0000 7f4649051640 1 -- 172.21.15.116:0/3509560912 wait complete. 2023-11-28T17:09:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.637+0000 7f4649051640 1 Processor -- start 2023-11-28T17:09:22.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.638+0000 7f4649051640 1 -- start start 2023-11-28T17:09:22.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.638+0000 7f4649051640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f464413cee0 0x7f4644138c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:22.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.638+0000 7f4649051640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4644140200 0x7f46441f4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.639+0000 7f4649051640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f464415a490 0x7f46440add10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.639+0000 7f4649051640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4644139f30 con 0x7f464415a490 2023-11-28T17:09:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.639+0000 7f4649051640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46441f5080 con 0x7f4644140200 2023-11-28T17:09:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.639+0000 7f4649051640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4644139320 con 0x7f464413cee0 2023-11-28T17:09:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.639+0000 7f464359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f464415a490 0x7f46440add10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.639+0000 7f464359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f464415a490 0x7f46440add10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56366/0 (socket says 172.21.15.116:56366) 2023-11-28T17:09:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.639+0000 7f464359e640 1 -- 172.21.15.116:0/828107072 learned_addr learned my addr 172.21.15.116:0/828107072 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.639+0000 7f4642d9d640 1 --2- 172.21.15.116:0/828107072 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f464413cee0 0x7f4644138c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.639+0000 7f464259c640 1 --2- 172.21.15.116:0/828107072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4644140200 0x7f46441f4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.639+0000 7f464359e640 1 -- 172.21.15.116:0/828107072 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f464413cee0 msgr2=0x7f4644138c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.639+0000 7f464359e640 1 --2- 172.21.15.116:0/828107072 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f464413cee0 0x7f4644138c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.639+0000 7f464359e640 1 -- 172.21.15.116:0/828107072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4644140200 msgr2=0x7f46441f4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.639+0000 7f464359e640 1 --2- 172.21.15.116:0/828107072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4644140200 0x7f46441f4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:22.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.639+0000 7f464359e640 1 -- 172.21.15.116:0/828107072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4634009a70 con 0x7f464415a490 2023-11-28T17:09:22.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.640+0000 7f464359e640 1 --2- 172.21.15.116:0/828107072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f464415a490 0x7f46440add10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f463c00c410 tx=0x7f463c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:22.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.640+0000 7f463b7fe640 1 -- 172.21.15.116:0/828107072 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0074e0 con 0x7f464415a490 2023-11-28T17:09:22.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.640+0000 7f463b7fe640 1 -- 172.21.15.116:0/828107072 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c007640 con 0x7f464415a490 2023-11-28T17:09:22.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.640+0000 7f463b7fe640 1 -- 172.21.15.116:0/828107072 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c004a20 con 0x7f464415a490 2023-11-28T17:09:22.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.640+0000 7f4649051640 1 -- 172.21.15.116:0/828107072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4644139600 con 0x7f464415a490 2023-11-28T17:09:22.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.640+0000 7f4649051640 1 -- 172.21.15.116:0/828107072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4644139bc0 con 0x7f464415a490 2023-11-28T17:09:22.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.641+0000 7f4649051640 1 -- 172.21.15.116:0/828107072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464413a0e0 con 0x7f464415a490 2023-11-28T17:09:22.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.642+0000 7f463b7fe640 1 -- 172.21.15.116:0/828107072 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f463c004b80 con 0x7f464415a490 2023-11-28T17:09:22.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.643+0000 7f463b7fe640 1 --2- 172.21.15.116:0/828107072 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f462406b2a0 0x7f462406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:22.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.643+0000 7f463b7fe640 1 -- 172.21.15.116:0/828107072 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f463c089910 con 0x7f464415a490 2023-11-28T17:09:22.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.643+0000 7f4642d9d640 1 --2- 172.21.15.116:0/828107072 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f462406b2a0 0x7f462406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:22.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.644+0000 7f4642d9d640 1 --2- 172.21.15.116:0/828107072 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f462406b2a0 0x7f462406d760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4634036000 tx=0x7f4634030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:22.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.648+0000 7f463b7fe640 1 -- 172.21.15.116:0/828107072 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f464413a0e0 con 0x7f464415a490 2023-11-28T17:09:22.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.867+0000 7f4649051640 1 -- 172.21.15.116:0/828107072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f46440693d0 con 0x7f464415a490 2023-11-28T17:09:22.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.868+0000 7f463b7fe640 1 -- 172.21.15.116:0/828107072 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f463c054780 con 0x7f464415a490 2023-11-28T17:09:22.869 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T17:09:22.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.870+0000 7f4649051640 1 -- 172.21.15.116:0/828107072 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f462406b2a0 msgr2=0x7f462406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.870+0000 7f4649051640 1 --2- 172.21.15.116:0/828107072 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f462406b2a0 0x7f462406d760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4634036000 tx=0x7f4634030000 comp rx=0 tx=0).stop 2023-11-28T17:09:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.870+0000 7f4649051640 1 -- 172.21.15.116:0/828107072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f464415a490 msgr2=0x7f46440add10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.870+0000 7f4649051640 1 --2- 172.21.15.116:0/828107072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f464415a490 0x7f46440add10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f463c00c410 tx=0x7f463c00c8d0 comp rx=0 tx=0).stop 2023-11-28T17:09:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.870+0000 7f4649051640 1 -- 172.21.15.116:0/828107072 shutdown_connections 2023-11-28T17:09:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.870+0000 7f4649051640 1 --2- 172.21.15.116:0/828107072 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f462406b2a0 0x7f462406d760 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.870+0000 7f4649051640 1 --2- 172.21.15.116:0/828107072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f464415a490 0x7f46440add10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.870+0000 7f4649051640 1 --2- 172.21.15.116:0/828107072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4644140200 0x7f46441f4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.870+0000 7f4649051640 1 --2- 172.21.15.116:0/828107072 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f464413cee0 0x7f4644138c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.870+0000 7f4649051640 1 -- 172.21.15.116:0/828107072 >> 172.21.15.116:0/828107072 conn(0x7f464406a4c0 msgr2=0x7f464415bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.870+0000 7f4649051640 1 -- 172.21.15.116:0/828107072 shutdown_connections 2023-11-28T17:09:22.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:22.871+0000 7f4649051640 1 -- 172.21.15.116:0/828107072 wait complete. 2023-11-28T17:09:22.889 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T17:09:22.889 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T17:09:22.889 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2023-11-28T17:09:22.889 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-11-28T17:09:22.889 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-11-28T17:09:22.889 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T17:09:22.889 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T17:09:22.889 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T17:09:22.889 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-11-28T17:09:22.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:22.890 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2023-11-28T17:09:22.941 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T17:09:22.942 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T17:09:22.942 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T17:09:22.942 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:22.942 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2023-11-28T17:09:22.942 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:22.942 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:09:22.942 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:22.942 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2023-11-28T17:09:22.942 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:09:22.942 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:22.942 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2023-11-28T17:09:22.943 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:09:22.943 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:22.943 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T17:09:22.943 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:22.943 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T17:09:22.943 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42375sec preferred_lft 42375sec 2023-11-28T17:09:22.943 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2023-11-28T17:09:22.943 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:22.943 INFO:teuthology.orchestra.run.smithi116.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T17:09:22.943 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 66:db:2d:c1:d7:e7 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:22.943 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T17:09:22.943 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:22.944 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::c06c:35ff:fe15:dc8e/64 scope link 2023-11-28T17:09:22.944 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:22.944 INFO:teuthology.orchestra.run.smithi116.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T17:09:22.944 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 66:db:2d:c1:d7:e7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:09:22.944 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::64db:2dff:fec1:d7e7/64 scope link 2023-11-28T17:09:22.944 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:22.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:22.945 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T17:09:23.037 INFO:teuthology.orchestra.run.smithi116.stdout:1 2023-11-28T17:09:23.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:23.042 DEBUG:teuthology.orchestra.run.smithi116:> route 2023-11-28T17:09:23.097 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2023-11-28T17:09:23.097 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T17:09:23.097 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T17:09:23.097 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T17:09:23.097 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T17:09:23.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:23.098 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-28T17:09:23.098 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-28T17:09:23.098 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T17:09:23.098 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T17:09:23.098 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T17:09:23.098 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-28T17:09:23.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:23.282 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2023-11-28T17:09:23.339 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T17:09:23.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:23.341 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2023-11-28T17:09:23.394 INFO:teuthology.orchestra.run.smithi116.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-11-28T17:09:23.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:23.396 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-28T17:09:23.396 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-28T17:09:23.396 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:09:23.396 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-11-28T17:09:23.396 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-28T17:09:23.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:23.540 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-11-28T17:09:23.626 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T17:09:23.627 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T17:09:23.627 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T17:09:23.627 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:23.627 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2023-11-28T17:09:23.627 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:23.627 INFO:teuthology.orchestra.run.smithi116.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T17:09:23.627 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether d2:f4:46:f0:62:ec brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-28T17:09:23.627 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-28T17:09:23.627 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:23.627 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::d0f4:46ff:fef0:62ec/64 scope link 2023-11-28T17:09:23.627 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:23.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:23.631 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-11-28T17:09:23.723 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T17:09:23.723 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T17:09:23.723 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T17:09:23.723 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:23.724 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2023-11-28T17:09:23.724 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:23.724 INFO:teuthology.orchestra.run.smithi116.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T17:09:23.724 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether d2:f4:46:f0:62:ec brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-28T17:09:23.724 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-28T17:09:23.724 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:23.724 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::d0f4:46ff:fef0:62ec/64 scope link 2023-11-28T17:09:23.724 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:23.727 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-11-28T17:09:23.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:23.728 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-28T17:09:23.728 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-28T17:09:23.728 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-11-28T17:09:23.728 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-11-28T17:09:23.728 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-11-28T17:09:23.728 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-11-28T17:09:23.728 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-11-28T17:09:23.728 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-28T17:09:23.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:23.998 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-28T17:09:23.998 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-28T17:09:23.998 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.1 up 2023-11-28T17:09:23.998 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.1 master ceph-brx 2023-11-28T17:09:23.998 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-28T17:09:24.164 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2023-11-28T17:09:24.165 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:09:24.165 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-11-28T17:09:24.218 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-11-28T17:09:24.218 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:09:24.218 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-11-28T17:09:24.270 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-28T17:09:24.358 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:09:24.412 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2023-11-28T17:09:24.413 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2023-11-28T17:09:24.413 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2023-11-28T17:09:24.413 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2023-11-28T17:09:24.413 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2023-11-28T17:09:24.413 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2023-11-28T17:09:24.413 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2023-11-28T17:09:24.413 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2023-11-28T17:09:24.413 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2023-11-28T17:09:24.413 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2023-11-28T17:09:24.413 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2023-11-28T17:09:24.413 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:09:24.413 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2023-11-28T17:09:24.414 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2023-11-28T17:09:24.414 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2023-11-28T17:09:24.414 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2023-11-28T17:09:24.414 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2023-11-28T17:09:24.414 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2023-11-28T17:09:24.414 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:09:24.414 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:09:24.414 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:09:24.414 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:09:24.414 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2023-11-28T17:09:24.414 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2023-11-28T17:09:24.414 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T17:09:24.415 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T17:09:24.415 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T17:09:24.415 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T17:09:24.415 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2023-11-28T17:09:24.415 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:09:24.415 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:09:24.415 INFO:teuthology.orchestra.run.smithi116.stdout:/home/ubuntu/cephtest/mnt.0 2023-11-28T17:09:24.415 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:09:24.415 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:09:24.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:24.416 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-28T17:09:24.469 INFO:teuthology.orchestra.run.smithi116.stdout:39 2023-11-28T17:09:24.469 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [39] 2023-11-28T17:09:24.469 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2023-11-28T17:09:24.475 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-28T17:09:24.550 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:09:24.594 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2023-11-28T17:09:24.595 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2023-11-28T17:09:24.595 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2023-11-28T17:09:24.595 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2023-11-28T17:09:24.595 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2023-11-28T17:09:24.595 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2023-11-28T17:09:24.596 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2023-11-28T17:09:24.596 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2023-11-28T17:09:24.596 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2023-11-28T17:09:24.596 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2023-11-28T17:09:24.596 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2023-11-28T17:09:24.596 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:09:24.597 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2023-11-28T17:09:24.597 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2023-11-28T17:09:24.597 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2023-11-28T17:09:24.597 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2023-11-28T17:09:24.597 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2023-11-28T17:09:24.598 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2023-11-28T17:09:24.598 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:09:24.598 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:09:24.598 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:09:24.598 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:09:24.599 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2023-11-28T17:09:24.599 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2023-11-28T17:09:24.599 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T17:09:24.599 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T17:09:24.599 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T17:09:24.599 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T17:09:24.599 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2023-11-28T17:09:24.600 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:09:24.600 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:09:24.600 INFO:teuthology.orchestra.run.smithi116.stdout:/home/ubuntu/cephtest/mnt.0 2023-11-28T17:09:24.600 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:09:24.600 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:09:24.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:24.602 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-28T17:09:24.662 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi116.stderr:2023-11-28T17:09:24.661+0000 7f7f1a80c480 -1 init, newargv = 0x5569b5d4f660 newargc=15 2023-11-28T17:09:24.662 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi116.stderr:ceph-fuse[97420]: starting ceph client 2023-11-28T17:09:24.673 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi116.stderr:ceph-fuse[97420]: starting fuse 2023-11-28T17:09:24.689 INFO:teuthology.orchestra.run.smithi116.stdout:39 2023-11-28T17:09:24.690 INFO:teuthology.orchestra.run.smithi116.stdout:42 2023-11-28T17:09:24.690 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39, 42] 2023-11-28T17:09:24.690 DEBUG:teuthology.orchestra.run.smithi116:> sudo stdin-killer -- python3 -c ' 2023-11-28T17:09:24.690 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2023-11-28T17:09:24.691 DEBUG:teuthology.orchestra.run.smithi116:> import re 2023-11-28T17:09:24.691 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-28T17:09:24.691 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2023-11-28T17:09:24.691 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:09:24.691 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2023-11-28T17:09:24.691 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2023-11-28T17:09:24.691 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2023-11-28T17:09:24.692 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2023-11-28T17:09:24.692 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:09:24.692 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2023-11-28T17:09:24.692 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2023-11-28T17:09:24.692 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2023-11-28T17:09:24.692 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2023-11-28T17:09:24.692 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:09:24.692 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2023-11-28T17:09:24.693 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-28T17:09:24.693 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-28T17:09:24.693 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-28T17:09:24.693 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2023-11-28T17:09:24.693 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2023-11-28T17:09:24.693 DEBUG:teuthology.orchestra.run.smithi116:> return f 2023-11-28T17:09:24.693 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-28T17:09:24.694 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:09:24.694 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.2")) 2023-11-28T17:09:24.694 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-28T17:09:24.893 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.2.97420.asok 2023-11-28T17:09:24.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-11-28T17:09:24.910 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.97420.asok 2023-11-28T17:09:24.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:24.910 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.97420.asok status 2023-11-28T17:09:25.157 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-28T17:09:25.157 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2023-11-28T17:09:25.157 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "47c1c547372ed5bff7984647dc200bac0e019caf", 2023-11-28T17:09:25.157 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev)", 2023-11-28T17:09:25.157 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "2", 2023-11-28T17:09:25.157 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2023-11-28T17:09:25.157 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2023-11-28T17:09:25.157 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "97420", 2023-11-28T17:09:25.158 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2023-11-28T17:09:25.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:09:25.158 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2023-11-28T17:09:25.158 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2023-11-28T17:09:25.158 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 4650, 2023-11-28T17:09:25.158 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2023-11-28T17:09:25.158 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2023-11-28T17:09:25.158 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2023-11-28T17:09:25.158 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4650 2023-11-28T17:09:25.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:09:25.158 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-28T17:09:25.158 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-28T17:09:25.158 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.2:0", 2023-11-28T17:09:25.158 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1406685451 2023-11-28T17:09:25.159 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:09:25.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:09:25.159 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-28T17:09:25.159 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-28T17:09:25.159 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.2:0", 2023-11-28T17:09:25.159 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1406685451 2023-11-28T17:09:25.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:09:25.159 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.4650 192.168.0.2:0/1406685451", 2023-11-28T17:09:25.159 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.2:0/1406685451", 2023-11-28T17:09:25.159 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2023-11-28T17:09:25.159 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 8, 2023-11-28T17:09:25.159 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 20, 2023-11-28T17:09:25.159 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2023-11-28T17:09:25.159 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2023-11-28T17:09:25.160 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2023-11-28T17:09:25.160 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-28T17:09:25.172 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T17:09:25.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.386+0000 7f8ef2264640 1 Processor -- start 2023-11-28T17:09:25.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.386+0000 7f8ef2264640 1 -- start start 2023-11-28T17:09:25.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.387+0000 7f8ef2264640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8eec15ae20 0x7f8eec06ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:25.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.387+0000 7f8ef2264640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8eec0cdf90 0x7f8eec0ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:25.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.387+0000 7f8ef2264640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eec070560 con 0x7f8eec0ceb90 2023-11-28T17:09:25.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.387+0000 7f8ef2264640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eec070840 con 0x7f8eec0cdf90 2023-11-28T17:09:25.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.387+0000 7f8ef2264640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eec070b20 con 0x7f8eec15ae20 2023-11-28T17:09:25.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.388+0000 7f8ef1a63640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8eec0ceb90 0x7f8eec15ee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41950/0 (socket says 172.21.15.116:41950) 2023-11-28T17:09:25.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.388+0000 7f8ef1a63640 1 -- 172.21.15.116:0/2776193166 learned_addr learned my addr 172.21.15.116:0/2776193166 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:25.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.388+0000 7f8ef0a61640 1 --2- 172.21.15.116:0/2776193166 >> v2:172.21.15.116:3300/0 conn(0x7f8eec0cdf90 0x7f8eec0ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:25.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.388+0000 7f8ef1262640 1 --2- 172.21.15.116:0/2776193166 >> v2:172.21.15.156:3300/0 conn(0x7f8eec15ae20 0x7f8eec06ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:25.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.388+0000 7f8ef0a61640 1 -- 172.21.15.116:0/2776193166 >> v2:172.21.15.156:3300/0 conn(0x7f8eec15ae20 msgr2=0x7f8eec06ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:25.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.388+0000 7f8ef0a61640 1 --2- 172.21.15.116:0/2776193166 >> v2:172.21.15.156:3300/0 conn(0x7f8eec15ae20 0x7f8eec06ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:25.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.388+0000 7f8ef0a61640 1 -- 172.21.15.116:0/2776193166 >> v1:172.21.15.116:6789/0 conn(0x7f8eec0ceb90 legacy=0x7f8eec15ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:25.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.388+0000 7f8ef0a61640 1 -- 172.21.15.116:0/2776193166 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec070e00 con 0x7f8eec0cdf90 2023-11-28T17:09:25.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.389+0000 7f8ef0a61640 1 --2- 172.21.15.116:0/2776193166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eec0cdf90 0x7f8eec0ce420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8edc0097a0 tx=0x7f8edc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dbe4ef21b695a2c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:25.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.390+0000 7f8ee3fff640 1 -- 172.21.15.116:0/2776193166 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc02e070 con 0x7f8eec0cdf90 2023-11-28T17:09:25.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.390+0000 7f8ee3fff640 1 -- 172.21.15.116:0/2776193166 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8edc0315a0 con 0x7f8eec0cdf90 2023-11-28T17:09:25.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.390+0000 7f8ee3fff640 1 -- 172.21.15.116:0/2776193166 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc031930 con 0x7f8eec0cdf90 2023-11-28T17:09:25.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.390+0000 7f8ef2264640 1 -- 172.21.15.116:0/2776193166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eec0cdf90 msgr2=0x7f8eec0ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:25.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.390+0000 7f8ef2264640 1 --2- 172.21.15.116:0/2776193166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eec0cdf90 0x7f8eec0ce420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8edc0097a0 tx=0x7f8edc02b840 comp rx=0 tx=0).stop 2023-11-28T17:09:25.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.390+0000 7f8ef2264640 1 -- 172.21.15.116:0/2776193166 shutdown_connections 2023-11-28T17:09:25.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.390+0000 7f8ef2264640 1 --2- 172.21.15.116:0/2776193166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eec0cdf90 0x7f8eec0ce420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:25.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.390+0000 7f8ef2264640 1 --2- 172.21.15.116:0/2776193166 >> v2:172.21.15.156:3300/0 conn(0x7f8eec15ae20 0x7f8eec06ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:25.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.390+0000 7f8ef2264640 1 -- 172.21.15.116:0/2776193166 >> 172.21.15.116:0/2776193166 conn(0x7f8eec0b0950 msgr2=0x7f8eec0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:25.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.390+0000 7f8ef2264640 1 -- 172.21.15.116:0/2776193166 shutdown_connections 2023-11-28T17:09:25.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.391+0000 7f8ef2264640 1 -- 172.21.15.116:0/2776193166 wait complete. 2023-11-28T17:09:25.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.392+0000 7f8ef2264640 1 Processor -- start 2023-11-28T17:09:25.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.392+0000 7f8ef2264640 1 -- start start 2023-11-28T17:09:25.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.393+0000 7f8ef2264640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eec0cdf90 0x7f8eec14a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:25.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.393+0000 7f8ef2264640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eec0ceb90 0x7f8eec1f4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:25.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.393+0000 7f8ef2264640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eec15ae20 0x7f8eec1f4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:25.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.393+0000 7f8ef2264640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec157d60 con 0x7f8eec0ceb90 2023-11-28T17:09:25.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.393+0000 7f8ef2264640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec15cf70 con 0x7f8eec0cdf90 2023-11-28T17:09:25.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.393+0000 7f8ef2264640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec14acf0 con 0x7f8eec15ae20 2023-11-28T17:09:25.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.394+0000 7f8ef0a61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eec0ceb90 0x7f8eec1f4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:25.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.394+0000 7f8ef0a61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eec0ceb90 0x7f8eec1f4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56378/0 (socket says 172.21.15.116:56378) 2023-11-28T17:09:25.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.394+0000 7f8ef0a61640 1 -- 172.21.15.116:0/3676329022 learned_addr learned my addr 172.21.15.116:0/3676329022 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:25.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.394+0000 7f8ef1262640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eec0cdf90 0x7f8eec14a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:25.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.394+0000 7f8ef1a63640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eec15ae20 0x7f8eec1f4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:25.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.394+0000 7f8ef0a61640 1 -- 172.21.15.116:0/3676329022 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eec15ae20 msgr2=0x7f8eec1f4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:25.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.394+0000 7f8ef0a61640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eec15ae20 0x7f8eec1f4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:25.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.394+0000 7f8ef0a61640 1 -- 172.21.15.116:0/3676329022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eec0cdf90 msgr2=0x7f8eec14a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:25.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.394+0000 7f8ef0a61640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eec0cdf90 0x7f8eec14a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:25.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.394+0000 7f8ef0a61640 1 -- 172.21.15.116:0/3676329022 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc009400 con 0x7f8eec0ceb90 2023-11-28T17:09:25.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.394+0000 7f8ef1262640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eec0cdf90 0x7f8eec14a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:09:25.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.394+0000 7f8ef0a61640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eec0ceb90 0x7f8eec1f4610 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8edc034040 tx=0x7f8edc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:25.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.395+0000 7f8ee1ffb640 1 -- 172.21.15.116:0/3676329022 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc032840 con 0x7f8eec0ceb90 2023-11-28T17:09:25.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.395+0000 7f8ee1ffb640 1 -- 172.21.15.116:0/3676329022 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8edc0329a0 con 0x7f8eec0ceb90 2023-11-28T17:09:25.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.395+0000 7f8ee1ffb640 1 -- 172.21.15.116:0/3676329022 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8edc032d50 con 0x7f8eec0ceb90 2023-11-28T17:09:25.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.395+0000 7f8ef2264640 1 -- 172.21.15.116:0/3676329022 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eec15b380 con 0x7f8eec0ceb90 2023-11-28T17:09:25.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.395+0000 7f8ef2264640 1 -- 172.21.15.116:0/3676329022 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eec15d1c0 con 0x7f8eec0ceb90 2023-11-28T17:09:25.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.396+0000 7f8ef2264640 1 -- 172.21.15.116:0/3676329022 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eec09d0b0 con 0x7f8eec0ceb90 2023-11-28T17:09:25.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.397+0000 7f8ee1ffb640 1 -- 172.21.15.116:0/3676329022 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f8edc04e430 con 0x7f8eec0ceb90 2023-11-28T17:09:25.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.398+0000 7f8ee1ffb640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8ed006b2a0 0x7f8ed006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:25.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.398+0000 7f8ee1ffb640 1 -- 172.21.15.116:0/3676329022 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8edc0abc00 con 0x7f8eec0ceb90 2023-11-28T17:09:25.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.398+0000 7f8ef1262640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8ed006b2a0 0x7f8ed006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:25.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.399+0000 7f8ef1262640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8ed006b2a0 0x7f8ed006d760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8006fd0 tx=0x7f8ee8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:25.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.403+0000 7f8ee1ffb640 1 -- 172.21.15.116:0/3676329022 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8edc076a60 con 0x7f8eec0ceb90 2023-11-28T17:09:25.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.629+0000 7f8ef2264640 1 -- 172.21.15.116:0/3676329022 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8eec04e960 con 0x7f8eec0ceb90 2023-11-28T17:09:25.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.629+0000 7f8ee1ffb640 1 -- 172.21.15.116:0/3676329022 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8edc07a080 con 0x7f8eec0ceb90 2023-11-28T17:09:25.631 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T17:09:25.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.632+0000 7f8ef2264640 1 -- 172.21.15.116:0/3676329022 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8ed006b2a0 msgr2=0x7f8ed006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:25.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.632+0000 7f8ef2264640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8ed006b2a0 0x7f8ed006d760 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8006fd0 tx=0x7f8ee8008040 comp rx=0 tx=0).stop 2023-11-28T17:09:25.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.632+0000 7f8ef2264640 1 -- 172.21.15.116:0/3676329022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eec0ceb90 msgr2=0x7f8eec1f4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:25.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.632+0000 7f8ef2264640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eec0ceb90 0x7f8eec1f4610 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8edc034040 tx=0x7f8edc0046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:25.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.632+0000 7f8ef2264640 1 -- 172.21.15.116:0/3676329022 shutdown_connections 2023-11-28T17:09:25.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.632+0000 7f8ef2264640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8ed006b2a0 0x7f8ed006d760 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:25.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.632+0000 7f8ef2264640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eec15ae20 0x7f8eec1f4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:25.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.632+0000 7f8ef2264640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eec0ceb90 0x7f8eec1f4610 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:25.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.632+0000 7f8ef2264640 1 --2- 172.21.15.116:0/3676329022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eec0cdf90 0x7f8eec14a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:25.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.632+0000 7f8ef2264640 1 -- 172.21.15.116:0/3676329022 >> 172.21.15.116:0/3676329022 conn(0x7f8eec0b0950 msgr2=0x7f8eec0cefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:25.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.632+0000 7f8ef2264640 1 -- 172.21.15.116:0/3676329022 shutdown_connections 2023-11-28T17:09:25.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:25.632+0000 7f8ef2264640 1 -- 172.21.15.116:0/3676329022 wait complete. 2023-11-28T17:09:25.652 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T17:09:25.653 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T17:09:25.653 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi156.front.sepia.ceph.com 2023-11-28T17:09:25.653 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-28T17:09:25.653 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-28T17:09:25.653 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T17:09:25.653 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T17:09:25.653 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T17:09:25.653 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-28T17:09:25.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:25.653 DEBUG:teuthology.orchestra.run.smithi156:> ip addr 2023-11-28T17:09:25.715 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T17:09:25.715 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T17:09:25.715 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T17:09:25.715 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:25.715 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2023-11-28T17:09:25.715 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:25.715 INFO:teuthology.orchestra.run.smithi156.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:09:25.716 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c0 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:25.716 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f0 2023-11-28T17:09:25.716 INFO:teuthology.orchestra.run.smithi156.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:09:25.716 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c1 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:25.716 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f1 2023-11-28T17:09:25.716 INFO:teuthology.orchestra.run.smithi156.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:09:25.716 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fc brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:25.716 INFO:teuthology.orchestra.run.smithi156.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T17:09:25.716 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fd brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:25.716 INFO:teuthology.orchestra.run.smithi156.stdout: inet 172.21.15.156/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T17:09:25.716 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft 42370sec preferred_lft 42370sec 2023-11-28T17:09:25.716 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::ec4:7aff:fe88:6dfd/64 scope link 2023-11-28T17:09:25.716 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:25.717 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-28T17:09:25.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:25.718 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2023-11-28T17:09:25.718 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2023-11-28T17:09:25.718 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link add name ceph-brx type bridge 2023-11-28T17:09:25.718 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip addr flush dev ceph-brx 2023-11-28T17:09:25.718 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set ceph-brx up 2023-11-28T17:09:25.718 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-28T17:09:25.718 DEBUG:teuthology.orchestra.run.smithi156:> ') 2023-11-28T17:09:26.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:26.181 DEBUG:teuthology.orchestra.run.smithi156:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T17:09:26.261 INFO:teuthology.orchestra.run.smithi156.stdout:1 2023-11-28T17:09:26.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:26.265 DEBUG:teuthology.orchestra.run.smithi156:> route 2023-11-28T17:09:26.315 INFO:teuthology.orchestra.run.smithi156.stdout:Kernel IP routing table 2023-11-28T17:09:26.316 INFO:teuthology.orchestra.run.smithi156.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T17:09:26.316 INFO:teuthology.orchestra.run.smithi156.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T17:09:26.316 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T17:09:26.316 INFO:teuthology.orchestra.run.smithi156.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T17:09:26.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:26.316 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2023-11-28T17:09:26.316 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2023-11-28T17:09:26.316 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T17:09:26.316 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T17:09:26.317 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T17:09:26.317 DEBUG:teuthology.orchestra.run.smithi156:> ') 2023-11-28T17:09:27.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:27.083 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2023-11-28T17:09:27.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:27.143 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list-id 2023-11-28T17:09:27.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:27.197 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2023-11-28T17:09:27.197 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2023-11-28T17:09:27.197 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:09:27.197 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-11-28T17:09:27.197 DEBUG:teuthology.orchestra.run.smithi156:> ') 2023-11-28T17:09:27.340 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-11-28T17:09:27.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:27.340 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2023-11-28T17:09:27.340 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2023-11-28T17:09:27.340 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-11-28T17:09:27.340 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-28T17:09:27.341 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-11-28T17:09:27.341 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-11-28T17:09:27.341 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-11-28T17:09:27.341 DEBUG:teuthology.orchestra.run.smithi156:> ') 2023-11-28T17:09:27.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:27.666 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2023-11-28T17:09:27.666 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2023-11-28T17:09:27.666 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set brx.0 up 2023-11-28T17:09:27.666 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set dev brx.0 master ceph-brx 2023-11-28T17:09:27.667 DEBUG:teuthology.orchestra.run.smithi156:> ') 2023-11-28T17:09:27.827 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-11-28T17:09:27.827 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:09:27.827 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-28T17:09:27.875 INFO:teuthology.orchestra.run.smithi156.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-28T17:09:27.876 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:09:27.876 DEBUG:teuthology.orchestra.run.smithi156:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-28T17:09:27.927 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2023-11-28T17:09:28.022 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:09:28.079 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2023-11-28T17:09:28.079 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2023-11-28T17:09:28.079 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2023-11-28T17:09:28.079 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2023-11-28T17:09:28.080 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2023-11-28T17:09:28.080 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2023-11-28T17:09:28.080 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2023-11-28T17:09:28.080 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2023-11-28T17:09:28.080 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2023-11-28T17:09:28.080 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2023-11-28T17:09:28.080 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2023-11-28T17:09:28.080 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:09:28.080 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2023-11-28T17:09:28.080 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2023-11-28T17:09:28.080 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2023-11-28T17:09:28.080 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2023-11-28T17:09:28.081 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2023-11-28T17:09:28.081 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2023-11-28T17:09:28.081 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:09:28.081 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:09:28.081 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:09:28.081 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:09:28.081 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2023-11-28T17:09:28.081 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2023-11-28T17:09:28.081 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2023-11-28T17:09:28.081 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2023-11-28T17:09:28.081 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2023-11-28T17:09:28.081 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2023-11-28T17:09:28.081 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2023-11-28T17:09:28.081 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:09:28.082 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:09:28.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:28.083 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2023-11-28T17:09:28.129 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-28T17:09:28.130 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-11-28T17:09:28.178 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2023-11-28T17:09:28.261 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:09:28.306 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2023-11-28T17:09:28.306 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2023-11-28T17:09:28.306 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2023-11-28T17:09:28.306 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2023-11-28T17:09:28.306 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2023-11-28T17:09:28.306 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2023-11-28T17:09:28.307 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2023-11-28T17:09:28.307 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2023-11-28T17:09:28.307 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2023-11-28T17:09:28.307 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2023-11-28T17:09:28.307 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2023-11-28T17:09:28.307 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:09:28.307 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2023-11-28T17:09:28.307 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2023-11-28T17:09:28.307 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2023-11-28T17:09:28.307 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2023-11-28T17:09:28.307 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2023-11-28T17:09:28.307 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2023-11-28T17:09:28.308 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:09:28.308 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:09:28.308 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:09:28.308 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:09:28.308 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2023-11-28T17:09:28.308 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2023-11-28T17:09:28.308 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2023-11-28T17:09:28.308 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2023-11-28T17:09:28.308 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2023-11-28T17:09:28.308 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2023-11-28T17:09:28.308 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2023-11-28T17:09:28.308 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:09:28.309 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:09:28.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:28.310 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2023-11-28T17:09:28.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2023-11-28T17:09:28.377+0000 7ff6a8c46480 -1 init, newargv = 0x560a3609b860 newargc=15 2023-11-28T17:09:28.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[93186]: starting ceph client 2023-11-28T17:09:28.389 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[93186]: starting fuse 2023-11-28T17:09:28.400 INFO:teuthology.orchestra.run.smithi156.stdout:39 2023-11-28T17:09:28.400 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2023-11-28T17:09:28.400 DEBUG:teuthology.orchestra.run.smithi156:> sudo stdin-killer -- python3 -c ' 2023-11-28T17:09:28.401 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2023-11-28T17:09:28.401 DEBUG:teuthology.orchestra.run.smithi156:> import re 2023-11-28T17:09:28.401 DEBUG:teuthology.orchestra.run.smithi156:> import os 2023-11-28T17:09:28.401 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2023-11-28T17:09:28.401 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:09:28.401 DEBUG:teuthology.orchestra.run.smithi156:> def _find_admin_socket(client_name): 2023-11-28T17:09:28.401 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-11-28T17:09:28.401 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2023-11-28T17:09:28.401 DEBUG:teuthology.orchestra.run.smithi156:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-11-28T17:09:28.401 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:09:28.401 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2023-11-28T17:09:28.401 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2023-11-28T17:09:28.401 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2023-11-28T17:09:28.401 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2023-11-28T17:09:28.402 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:09:28.402 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2023-11-28T17:09:28.402 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-28T17:09:28.402 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-28T17:09:28.402 DEBUG:teuthology.orchestra.run.smithi156:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-28T17:09:28.402 DEBUG:teuthology.orchestra.run.smithi156:> contents = proc_f.read() 2023-11-28T17:09:28.402 DEBUG:teuthology.orchestra.run.smithi156:> if mountpoint in contents: 2023-11-28T17:09:28.402 DEBUG:teuthology.orchestra.run.smithi156:> return f 2023-11-28T17:09:28.402 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-28T17:09:28.402 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:09:28.402 DEBUG:teuthology.orchestra.run.smithi156:> print(_find_admin_socket("client.1")) 2023-11-28T17:09:28.402 DEBUG:teuthology.orchestra.run.smithi156:> ' 2023-11-28T17:09:28.597 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.93186.asok 2023-11-28T17:09:28.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:09:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-11-28T17:09:28.614 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.93186.asok 2023-11-28T17:09:28.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:28.615 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.93186.asok status 2023-11-28T17:09:28.856 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-11-28T17:09:28.856 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2023-11-28T17:09:28.856 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "47c1c547372ed5bff7984647dc200bac0e019caf", 2023-11-28T17:09:28.856 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev)", 2023-11-28T17:09:28.856 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2023-11-28T17:09:28.856 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2023-11-28T17:09:28.857 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-11-28T17:09:28.857 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "93186", 2023-11-28T17:09:28.857 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2023-11-28T17:09:28.857 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:09:28.857 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2023-11-28T17:09:28.857 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2023-11-28T17:09:28.858 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 4442, 2023-11-28T17:09:28.858 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2023-11-28T17:09:28.858 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2023-11-28T17:09:28.858 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2023-11-28T17:09:28.858 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 4442 2023-11-28T17:09:28.858 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:09:28.858 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2023-11-28T17:09:28.859 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2023-11-28T17:09:28.859 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2023-11-28T17:09:28.859 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3429416532 2023-11-28T17:09:28.859 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-28T17:09:28.859 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:09:28.859 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2023-11-28T17:09:28.859 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2023-11-28T17:09:28.859 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2023-11-28T17:09:28.860 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3429416532 2023-11-28T17:09:28.860 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:09:28.860 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.4442 192.168.0.1:0/3429416532", 2023-11-28T17:09:28.860 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "192.168.0.1:0/3429416532", 2023-11-28T17:09:28.860 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2023-11-28T17:09:28.860 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 8, 2023-11-28T17:09:28.860 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 20, 2023-11-28T17:09:28.860 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2023-11-28T17:09:28.861 INFO:teuthology.orchestra.run.smithi156.stdout: "blocklisted": false, 2023-11-28T17:09:28.861 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2023-11-28T17:09:28.861 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-11-28T17:09:28.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T17:09:29.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.093+0000 7fa894a90640 1 Processor -- start 2023-11-28T17:09:29.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.093+0000 7fa894a90640 1 -- start start 2023-11-28T17:09:29.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.095+0000 7fa894a90640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa890144430 0x7fa89015f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:29.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.095+0000 7fa894a90640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa89006c250 0x7fa89006c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:29.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.095+0000 7fa894a90640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa890070bd0 con 0x7fa8900703f0 2023-11-28T17:09:29.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.095+0000 7fa894a90640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa890144810 con 0x7fa89006c250 2023-11-28T17:09:29.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.095+0000 7fa894a90640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89006cc20 con 0x7fa890144430 2023-11-28T17:09:29.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.095+0000 7fa88e59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa890144430 0x7fa89015f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:29.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.096+0000 7fa88e59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa890144430 0x7fa89015f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:44726/0 (socket says 172.21.15.116:44726) 2023-11-28T17:09:29.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.096+0000 7fa88e59c640 1 -- 172.21.15.116:0/2369643223 learned_addr learned my addr 172.21.15.116:0/2369643223 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:29.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.096+0000 7fa88ed9d640 1 --2- 172.21.15.116:0/2369643223 >> v2:172.21.15.156:3300/0 conn(0x7fa89006c250 0x7fa89006c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:29.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.096+0000 7fa88ed9d640 1 -- 172.21.15.116:0/2369643223 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa890144430 msgr2=0x7fa89015f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:29.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.096+0000 7fa88ed9d640 1 --2- 172.21.15.116:0/2369643223 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa890144430 0x7fa89015f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:29.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.096+0000 7fa88ed9d640 1 -- 172.21.15.116:0/2369643223 >> v1:172.21.15.156:6789/0 conn(0x7fa8900703f0 legacy=0x7fa890071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:29.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.096+0000 7fa88ed9d640 1 -- 172.21.15.116:0/2369643223 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa890070eb0 con 0x7fa89006c250 2023-11-28T17:09:29.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.098+0000 7fa88ed9d640 1 --2- 172.21.15.116:0/2369643223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa89006c250 0x7fa89006c6e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa888009d90 tx=0x7fa88802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=169002cf37115747 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:29.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.098+0000 7fa88dd9b640 1 -- 172.21.15.116:0/2369643223 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88802cc50 con 0x7fa89006c250 2023-11-28T17:09:29.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.098+0000 7fa88dd9b640 1 -- 172.21.15.116:0/2369643223 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88802cdb0 con 0x7fa89006c250 2023-11-28T17:09:29.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.098+0000 7fa88dd9b640 1 -- 172.21.15.116:0/2369643223 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa888034660 con 0x7fa89006c250 2023-11-28T17:09:29.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.099+0000 7fa894a90640 1 -- 172.21.15.116:0/2369643223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa89006c250 msgr2=0x7fa89006c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:29.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.099+0000 7fa894a90640 1 --2- 172.21.15.116:0/2369643223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa89006c250 0x7fa89006c6e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa888009d90 tx=0x7fa88802bea0 comp rx=0 tx=0).stop 2023-11-28T17:09:29.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.099+0000 7fa894a90640 1 -- 172.21.15.116:0/2369643223 shutdown_connections 2023-11-28T17:09:29.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.099+0000 7fa894a90640 1 --2- 172.21.15.116:0/2369643223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa89006c250 0x7fa89006c6e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:29.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.099+0000 7fa894a90640 1 --2- 172.21.15.116:0/2369643223 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa890144430 0x7fa89015f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:29.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.099+0000 7fa894a90640 1 -- 172.21.15.116:0/2369643223 >> 172.21.15.116:0/2369643223 conn(0x7fa89006a4c0 msgr2=0x7fa89006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:29.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.099+0000 7fa894a90640 1 -- 172.21.15.116:0/2369643223 shutdown_connections 2023-11-28T17:09:29.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.099+0000 7fa894a90640 1 -- 172.21.15.116:0/2369643223 wait complete. 2023-11-28T17:09:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.101+0000 7fa894a90640 1 Processor -- start 2023-11-28T17:09:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.101+0000 7fa894a90640 1 -- start start 2023-11-28T17:09:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.103+0000 7fa894a90640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa89006c250 0x7fa89013a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.103+0000 7fa894a90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8900703f0 0x7fa89013a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:29.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.103+0000 7fa894a90640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa890144430 0x7fa89013cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:29.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.103+0000 7fa894a90640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89013c980 con 0x7fa8900703f0 2023-11-28T17:09:29.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.103+0000 7fa894a90640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89015ce60 con 0x7fa89006c250 2023-11-28T17:09:29.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.103+0000 7fa894a90640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89015d140 con 0x7fa890144430 2023-11-28T17:09:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.103+0000 7fa887fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8900703f0 0x7fa89013a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.103+0000 7fa887fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8900703f0 0x7fa89013a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60932/0 (socket says 172.21.15.116:60932) 2023-11-28T17:09:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.103+0000 7fa887fff640 1 -- 172.21.15.116:0/1434759906 learned_addr learned my addr 172.21.15.116:0/1434759906 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:29.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.103+0000 7fa88ed9d640 1 --2- 172.21.15.116:0/1434759906 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa890144430 0x7fa89013cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:29.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.103+0000 7fa88e59c640 1 --2- 172.21.15.116:0/1434759906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa89006c250 0x7fa89013a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:29.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.104+0000 7fa887fff640 1 -- 172.21.15.116:0/1434759906 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa890144430 msgr2=0x7fa89013cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:29.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.104+0000 7fa887fff640 1 --2- 172.21.15.116:0/1434759906 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa890144430 0x7fa89013cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:29.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.104+0000 7fa887fff640 1 -- 172.21.15.116:0/1434759906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa89006c250 msgr2=0x7fa89013a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:29.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.104+0000 7fa887fff640 1 --2- 172.21.15.116:0/1434759906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa89006c250 0x7fa89013a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:29.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.104+0000 7fa887fff640 1 -- 172.21.15.116:0/1434759906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa888009a70 con 0x7fa8900703f0 2023-11-28T17:09:29.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.104+0000 7fa887fff640 1 --2- 172.21.15.116:0/1434759906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8900703f0 0x7fa89013a760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa87800c3c0 tx=0x7fa87800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:29.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.104+0000 7fa886ffd640 1 -- 172.21.15.116:0/1434759906 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8780052d0 con 0x7fa8900703f0 2023-11-28T17:09:29.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.104+0000 7fa886ffd640 1 -- 172.21.15.116:0/1434759906 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa878005450 con 0x7fa8900703f0 2023-11-28T17:09:29.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.104+0000 7fa894a90640 1 -- 172.21.15.116:0/1434759906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa89013c5a0 con 0x7fa8900703f0 2023-11-28T17:09:29.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.104+0000 7fa886ffd640 1 -- 172.21.15.116:0/1434759906 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa878007260 con 0x7fa8900703f0 2023-11-28T17:09:29.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.104+0000 7fa894a90640 1 -- 172.21.15.116:0/1434759906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa89013c250 con 0x7fa8900703f0 2023-11-28T17:09:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.105+0000 7fa894a90640 1 -- 172.21.15.116:0/1434759906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa89015d300 con 0x7fa8900703f0 2023-11-28T17:09:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.106+0000 7fa886ffd640 1 -- 172.21.15.116:0/1434759906 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa8780073c0 con 0x7fa8900703f0 2023-11-28T17:09:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.107+0000 7fa886ffd640 1 --2- 172.21.15.116:0/1434759906 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa87006b2a0 0x7fa87006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:29.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.107+0000 7fa886ffd640 1 -- 172.21.15.116:0/1434759906 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa878088f80 con 0x7fa8900703f0 2023-11-28T17:09:29.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.108+0000 7fa88e59c640 1 --2- 172.21.15.116:0/1434759906 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa87006b2a0 0x7fa87006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:29.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.108+0000 7fa88e59c640 1 --2- 172.21.15.116:0/1434759906 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa87006b2a0 0x7fa87006d760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa890143060 tx=0x7fa880008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:29.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.112+0000 7fa886ffd640 1 -- 172.21.15.116:0/1434759906 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa878053de0 con 0x7fa8900703f0 2023-11-28T17:09:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.345+0000 7fa894a90640 1 -- 172.21.15.116:0/1434759906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa89013cac0 con 0x7fa8900703f0 2023-11-28T17:09:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.346+0000 7fa886ffd640 1 -- 172.21.15.116:0/1434759906 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa89013cac0 con 0x7fa8900703f0 2023-11-28T17:09:29.348 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T17:09:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.349+0000 7fa894a90640 1 -- 172.21.15.116:0/1434759906 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa87006b2a0 msgr2=0x7fa87006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.349+0000 7fa894a90640 1 --2- 172.21.15.116:0/1434759906 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa87006b2a0 0x7fa87006d760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa890143060 tx=0x7fa880008040 comp rx=0 tx=0).stop 2023-11-28T17:09:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.349+0000 7fa894a90640 1 -- 172.21.15.116:0/1434759906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8900703f0 msgr2=0x7fa89013a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.349+0000 7fa894a90640 1 --2- 172.21.15.116:0/1434759906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8900703f0 0x7fa89013a760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa87800c3c0 tx=0x7fa87800c880 comp rx=0 tx=0).stop 2023-11-28T17:09:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.349+0000 7fa894a90640 1 -- 172.21.15.116:0/1434759906 shutdown_connections 2023-11-28T17:09:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.349+0000 7fa894a90640 1 --2- 172.21.15.116:0/1434759906 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa87006b2a0 0x7fa87006d760 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.349+0000 7fa894a90640 1 --2- 172.21.15.116:0/1434759906 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa890144430 0x7fa89013cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.349+0000 7fa894a90640 1 --2- 172.21.15.116:0/1434759906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8900703f0 0x7fa89013a760 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.349+0000 7fa894a90640 1 --2- 172.21.15.116:0/1434759906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa89006c250 0x7fa89013a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.349+0000 7fa894a90640 1 -- 172.21.15.116:0/1434759906 >> 172.21.15.116:0/1434759906 conn(0x7fa89006a4c0 msgr2=0x7fa89015e270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.349+0000 7fa894a90640 1 -- 172.21.15.116:0/1434759906 shutdown_connections 2023-11-28T17:09:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:29.349+0000 7fa894a90640 1 -- 172.21.15.116:0/1434759906 wait complete. 2023-11-28T17:09:29.368 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T17:09:29.368 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T17:09:29.369 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi156.front.sepia.ceph.com 2023-11-28T17:09:29.369 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-11-28T17:09:29.369 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-11-28T17:09:29.369 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T17:09:29.369 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T17:09:29.369 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T17:09:29.369 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-11-28T17:09:29.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:29.369 DEBUG:teuthology.orchestra.run.smithi156:> ip addr 2023-11-28T17:09:29.432 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T17:09:29.432 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T17:09:29.432 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T17:09:29.432 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:29.432 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2023-11-28T17:09:29.432 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:29.432 INFO:teuthology.orchestra.run.smithi156.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:09:29.432 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c0 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:29.432 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f0 2023-11-28T17:09:29.433 INFO:teuthology.orchestra.run.smithi156.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:09:29.433 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c1 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:29.433 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f1 2023-11-28T17:09:29.433 INFO:teuthology.orchestra.run.smithi156.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:09:29.433 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fc brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:29.433 INFO:teuthology.orchestra.run.smithi156.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T17:09:29.433 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fd brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:29.433 INFO:teuthology.orchestra.run.smithi156.stdout: inet 172.21.15.156/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T17:09:29.433 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft 42367sec preferred_lft 42367sec 2023-11-28T17:09:29.433 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::ec4:7aff:fe88:6dfd/64 scope link 2023-11-28T17:09:29.433 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:29.434 INFO:teuthology.orchestra.run.smithi156.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T17:09:29.434 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether e6:68:69:2b:d0:94 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:09:29.434 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T17:09:29.434 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:29.434 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::54f0:12ff:fe79:1e2c/64 scope link 2023-11-28T17:09:29.434 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:29.434 INFO:teuthology.orchestra.run.smithi156.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T17:09:29.434 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether e6:68:69:2b:d0:94 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:09:29.434 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::e468:69ff:fe2b:d094/64 scope link 2023-11-28T17:09:29.434 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:29.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:29.437 DEBUG:teuthology.orchestra.run.smithi156:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T17:09:29.529 INFO:teuthology.orchestra.run.smithi156.stdout:1 2023-11-28T17:09:29.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:29.533 DEBUG:teuthology.orchestra.run.smithi156:> route 2023-11-28T17:09:29.588 INFO:teuthology.orchestra.run.smithi156.stdout:Kernel IP routing table 2023-11-28T17:09:29.588 INFO:teuthology.orchestra.run.smithi156.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T17:09:29.588 INFO:teuthology.orchestra.run.smithi156.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T17:09:29.588 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T17:09:29.588 INFO:teuthology.orchestra.run.smithi156.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T17:09:29.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:29.589 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2023-11-28T17:09:29.589 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2023-11-28T17:09:29.589 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T17:09:29.589 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T17:09:29.589 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T17:09:29.589 DEBUG:teuthology.orchestra.run.smithi156:> ') 2023-11-28T17:09:29.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:29.777 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2023-11-28T17:09:29.829 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-28T17:09:29.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:29.830 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list-id 2023-11-28T17:09:29.881 INFO:teuthology.orchestra.run.smithi156.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-11-28T17:09:29.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:29.883 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2023-11-28T17:09:29.883 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2023-11-28T17:09:29.883 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:09:29.883 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-11-28T17:09:29.883 DEBUG:teuthology.orchestra.run.smithi156:> ') 2023-11-28T17:09:30.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:30.019 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-11-28T17:09:30.101 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T17:09:30.101 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T17:09:30.102 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T17:09:30.102 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:30.102 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2023-11-28T17:09:30.102 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:30.102 INFO:teuthology.orchestra.run.smithi156.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T17:09:30.102 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether e6:b8:33:3c:79:2a brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-28T17:09:30.102 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-28T17:09:30.102 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:30.102 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::e4b8:33ff:fe3c:792a/64 scope link 2023-11-28T17:09:30.102 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:30.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:30.105 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-11-28T17:09:30.193 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T17:09:30.193 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T17:09:30.193 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T17:09:30.193 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:30.193 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2023-11-28T17:09:30.193 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:30.193 INFO:teuthology.orchestra.run.smithi156.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T17:09:30.193 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether e6:b8:33:3c:79:2a brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-28T17:09:30.194 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-28T17:09:30.194 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:30.194 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::e4b8:33ff:fe3c:792a/64 scope link 2023-11-28T17:09:30.194 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:09:30.196 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-11-28T17:09:30.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:30.197 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2023-11-28T17:09:30.197 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2023-11-28T17:09:30.197 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-11-28T17:09:30.197 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-11-28T17:09:30.197 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-11-28T17:09:30.197 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-11-28T17:09:30.197 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-11-28T17:09:30.197 DEBUG:teuthology.orchestra.run.smithi156:> ') 2023-11-28T17:09:30.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:30.459 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2023-11-28T17:09:30.459 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2023-11-28T17:09:30.459 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set brx.1 up 2023-11-28T17:09:30.459 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set dev brx.1 master ceph-brx 2023-11-28T17:09:30.459 DEBUG:teuthology.orchestra.run.smithi156:> ') 2023-11-28T17:09:30.625 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2023-11-28T17:09:30.626 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:09:30.626 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-11-28T17:09:30.675 INFO:teuthology.orchestra.run.smithi156.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-11-28T17:09:30.675 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:09:30.675 DEBUG:teuthology.orchestra.run.smithi156:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-11-28T17:09:30.723 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2023-11-28T17:09:30.821 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:09:30.879 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2023-11-28T17:09:30.879 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2023-11-28T17:09:30.879 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2023-11-28T17:09:30.879 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2023-11-28T17:09:30.879 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2023-11-28T17:09:30.879 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2023-11-28T17:09:30.879 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2023-11-28T17:09:30.880 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2023-11-28T17:09:30.880 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2023-11-28T17:09:30.880 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2023-11-28T17:09:30.880 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2023-11-28T17:09:30.880 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:09:30.880 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2023-11-28T17:09:30.880 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2023-11-28T17:09:30.881 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2023-11-28T17:09:30.881 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2023-11-28T17:09:30.881 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2023-11-28T17:09:30.881 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2023-11-28T17:09:30.881 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:09:30.881 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:09:30.881 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:09:30.881 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:09:30.881 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2023-11-28T17:09:30.882 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2023-11-28T17:09:30.882 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2023-11-28T17:09:30.882 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2023-11-28T17:09:30.882 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2023-11-28T17:09:30.882 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2023-11-28T17:09:30.882 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2023-11-28T17:09:30.883 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:09:30.883 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:09:30.883 INFO:teuthology.orchestra.run.smithi156.stdout:/home/ubuntu/cephtest/mnt.1 2023-11-28T17:09:30.883 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:09:30.883 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:09:30.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:30.884 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2023-11-28T17:09:30.936 INFO:teuthology.orchestra.run.smithi156.stdout:39 2023-11-28T17:09:30.936 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [39] 2023-11-28T17:09:30.937 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2023-11-28T17:09:30.943 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2023-11-28T17:09:31.026 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:09:31.070 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2023-11-28T17:09:31.070 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2023-11-28T17:09:31.070 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2023-11-28T17:09:31.070 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2023-11-28T17:09:31.070 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2023-11-28T17:09:31.070 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2023-11-28T17:09:31.070 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2023-11-28T17:09:31.070 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2023-11-28T17:09:31.070 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2023-11-28T17:09:31.071 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2023-11-28T17:09:31.071 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2023-11-28T17:09:31.071 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:09:31.071 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2023-11-28T17:09:31.071 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2023-11-28T17:09:31.071 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2023-11-28T17:09:31.071 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2023-11-28T17:09:31.071 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2023-11-28T17:09:31.071 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2023-11-28T17:09:31.071 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:09:31.071 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:09:31.072 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:09:31.072 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:09:31.072 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2023-11-28T17:09:31.072 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2023-11-28T17:09:31.072 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2023-11-28T17:09:31.072 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2023-11-28T17:09:31.072 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2023-11-28T17:09:31.072 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2023-11-28T17:09:31.072 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2023-11-28T17:09:31.072 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:09:31.072 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:09:31.072 INFO:teuthology.orchestra.run.smithi156.stdout:/home/ubuntu/cephtest/mnt.1 2023-11-28T17:09:31.073 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:09:31.073 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:09:31.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:31.074 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2023-11-28T17:09:31.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi156.stderr:2023-11-28T17:09:31.129+0000 7fa032331480 -1 init, newargv = 0x56409c19e690 newargc=15 2023-11-28T17:09:31.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi156.stderr:ceph-fuse[93638]: starting ceph client 2023-11-28T17:09:31.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi156.stderr:ceph-fuse[93638]: starting fuse 2023-11-28T17:09:31.159 INFO:teuthology.orchestra.run.smithi156.stdout:39 2023-11-28T17:09:31.159 INFO:teuthology.orchestra.run.smithi156.stdout:42 2023-11-28T17:09:31.160 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39, 42] 2023-11-28T17:09:31.160 DEBUG:teuthology.orchestra.run.smithi156:> sudo stdin-killer -- python3 -c ' 2023-11-28T17:09:31.160 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2023-11-28T17:09:31.160 DEBUG:teuthology.orchestra.run.smithi156:> import re 2023-11-28T17:09:31.160 DEBUG:teuthology.orchestra.run.smithi156:> import os 2023-11-28T17:09:31.161 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2023-11-28T17:09:31.161 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:09:31.161 DEBUG:teuthology.orchestra.run.smithi156:> def _find_admin_socket(client_name): 2023-11-28T17:09:31.161 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2023-11-28T17:09:31.161 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2023-11-28T17:09:31.161 DEBUG:teuthology.orchestra.run.smithi156:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2023-11-28T17:09:31.161 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:09:31.162 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2023-11-28T17:09:31.162 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2023-11-28T17:09:31.162 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2023-11-28T17:09:31.162 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2023-11-28T17:09:31.162 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:09:31.162 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2023-11-28T17:09:31.162 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-28T17:09:31.162 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-28T17:09:31.162 DEBUG:teuthology.orchestra.run.smithi156:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-28T17:09:31.162 DEBUG:teuthology.orchestra.run.smithi156:> contents = proc_f.read() 2023-11-28T17:09:31.163 DEBUG:teuthology.orchestra.run.smithi156:> if mountpoint in contents: 2023-11-28T17:09:31.163 DEBUG:teuthology.orchestra.run.smithi156:> return f 2023-11-28T17:09:31.163 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-28T17:09:31.163 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:09:31.163 DEBUG:teuthology.orchestra.run.smithi156:> print(_find_admin_socket("client.3")) 2023-11-28T17:09:31.163 DEBUG:teuthology.orchestra.run.smithi156:> ' 2023-11-28T17:09:31.360 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.3.93638.asok 2023-11-28T17:09:31.364 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:09:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-11-28T17:09:31.377 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.93638.asok 2023-11-28T17:09:31.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:31.378 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.93638.asok status 2023-11-28T17:09:31.619 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-11-28T17:09:31.620 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2023-11-28T17:09:31.620 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "47c1c547372ed5bff7984647dc200bac0e019caf", 2023-11-28T17:09:31.620 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev)", 2023-11-28T17:09:31.620 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "3", 2023-11-28T17:09:31.620 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2023-11-28T17:09:31.620 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2023-11-28T17:09:31.620 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "93638", 2023-11-28T17:09:31.620 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2023-11-28T17:09:31.620 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:09:31.620 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2023-11-28T17:09:31.620 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2023-11-28T17:09:31.620 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 4531, 2023-11-28T17:09:31.621 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2023-11-28T17:09:31.621 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2023-11-28T17:09:31.621 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2023-11-28T17:09:31.621 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 4531 2023-11-28T17:09:31.621 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:09:31.621 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2023-11-28T17:09:31.621 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2023-11-28T17:09:31.621 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.2:0", 2023-11-28T17:09:31.621 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2273151964 2023-11-28T17:09:31.621 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-28T17:09:31.621 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:09:31.621 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2023-11-28T17:09:31.621 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2023-11-28T17:09:31.622 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.2:0", 2023-11-28T17:09:31.622 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2273151964 2023-11-28T17:09:31.622 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:09:31.622 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.4531 192.168.0.2:0/2273151964", 2023-11-28T17:09:31.622 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "192.168.0.2:0/2273151964", 2023-11-28T17:09:31.622 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2023-11-28T17:09:31.622 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 8, 2023-11-28T17:09:31.622 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 20, 2023-11-28T17:09:31.622 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2023-11-28T17:09:31.622 INFO:teuthology.orchestra.run.smithi156.stdout: "blocklisted": false, 2023-11-28T17:09:31.622 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2023-11-28T17:09:31.622 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-11-28T17:09:31.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:31.635 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2023-11-28T17:09:31.635 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.0 2023-11-28T17:09:31.690 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2023-11-28T17:09:31.691 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-11-28T17:09:31.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:31.691 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T17:09:31.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:31.775 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2023-11-28T17:09:31.775 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.2 2023-11-28T17:09:31.822 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2023-11-28T17:09:31.823 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2023-11-28T17:09:31.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:31.823 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-11-28T17:09:31.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:31.950 DEBUG:teuthology.orchestra.run.smithi156:> stat --file-system '--printf=%T 2023-11-28T17:09:31.950 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1 2023-11-28T17:09:32.013 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2023-11-28T17:09:32.013 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-11-28T17:09:32.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:32.014 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-28T17:09:32.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:32.111 DEBUG:teuthology.orchestra.run.smithi156:> stat --file-system '--printf=%T 2023-11-28T17:09:32.111 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.3 2023-11-28T17:09:32.170 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2023-11-28T17:09:32.170 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2023-11-28T17:09:32.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:32.171 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-11-28T17:09:32.264 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-11-28T17:09:32.290 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.496+0000 7f2cd2cff640 1 Processor -- start 2023-11-28T17:09:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.496+0000 7f2cd2cff640 1 -- start start 2023-11-28T17:09:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.497+0000 7f2cd2cff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ccc0cdf90 0x7f2ccc0ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.497+0000 7f2cd2cff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2ccc0ceb90 0x7f2ccc0cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.497+0000 7f2cd2cff640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ccc070560 con 0x7f2ccc15ae20 2023-11-28T17:09:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.498+0000 7f2cd2cff640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ccc0cf560 con 0x7f2ccc0ceb90 2023-11-28T17:09:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.498+0000 7f2cd2cff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc06d3e0 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.498+0000 7f2cd24fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2ccc0ceb90 0x7f2ccc0cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.498+0000 7f2cd14fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ccc0cdf90 0x7f2ccc0ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.498+0000 7f2cd24fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2ccc0ceb90 0x7f2ccc0cf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:51136/0 (socket says 172.21.15.116:51136) 2023-11-28T17:09:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.498+0000 7f2cd24fe640 1 -- 172.21.15.116:0/352595986 learned_addr learned my addr 172.21.15.116:0/352595986 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.499+0000 7f2cd14fc640 1 -- 172.21.15.116:0/352595986 >> v2:172.21.15.156:3300/0 conn(0x7f2ccc0ceb90 msgr2=0x7f2ccc0cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:32.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.499+0000 7f2cd14fc640 1 --2- 172.21.15.116:0/352595986 >> v2:172.21.15.156:3300/0 conn(0x7f2ccc0ceb90 0x7f2ccc0cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:32.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.499+0000 7f2cd14fc640 1 -- 172.21.15.116:0/352595986 >> v1:172.21.15.156:6789/0 conn(0x7f2ccc15ae20 legacy=0x7f2ccc06ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:32.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.499+0000 7f2cd14fc640 1 -- 172.21.15.116:0/352595986 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ccc070840 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.499+0000 7f2cd24fe640 1 --2- 172.21.15.116:0/352595986 >> v2:172.21.15.156:3300/0 conn(0x7f2ccc0ceb90 0x7f2ccc0cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:09:32.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.500+0000 7f2cd14fc640 1 --2- 172.21.15.116:0/352595986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ccc0cdf90 0x7f2ccc0ce3f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0097a0 tx=0x7f2cbc02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec6b8fb4e1a4edf7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:32.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.501+0000 7f2cd0cfb640 1 -- 172.21.15.116:0/352595986 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc031780 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.501+0000 7f2cd0cfb640 1 -- 172.21.15.116:0/352595986 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cbc0318e0 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.501+0000 7f2cd0cfb640 1 -- 172.21.15.116:0/352595986 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc031c70 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.501+0000 7f2cd2cff640 1 -- 172.21.15.116:0/352595986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ccc0cdf90 msgr2=0x7f2ccc0ce3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.501+0000 7f2cd2cff640 1 --2- 172.21.15.116:0/352595986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ccc0cdf90 0x7f2ccc0ce3f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0097a0 tx=0x7f2cbc02b820 comp rx=0 tx=0).stop 2023-11-28T17:09:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.502+0000 7f2cd2cff640 1 -- 172.21.15.116:0/352595986 shutdown_connections 2023-11-28T17:09:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.502+0000 7f2cd2cff640 1 --2- 172.21.15.116:0/352595986 >> v2:172.21.15.156:3300/0 conn(0x7f2ccc0ceb90 0x7f2ccc0cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.502+0000 7f2cd2cff640 1 --2- 172.21.15.116:0/352595986 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ccc0cdf90 0x7f2ccc0ce3f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:32.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.502+0000 7f2cd2cff640 1 -- 172.21.15.116:0/352595986 >> 172.21.15.116:0/352595986 conn(0x7f2ccc0b0950 msgr2=0x7f2ccc0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.502+0000 7f2cd2cff640 1 -- 172.21.15.116:0/352595986 shutdown_connections 2023-11-28T17:09:32.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.502+0000 7f2cd2cff640 1 -- 172.21.15.116:0/352595986 wait complete. 2023-11-28T17:09:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.504+0000 7f2cd2cff640 1 Processor -- start 2023-11-28T17:09:32.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.504+0000 7f2cd2cff640 1 -- start start 2023-11-28T17:09:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.505+0000 7f2cd2cff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ccc0cdf90 0x7f2ccc0c3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.505+0000 7f2cd2cff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ccc0ceb90 0x7f2ccc1453e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.505+0000 7f2cd2cff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ccc15ae20 0x7f2ccc1574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.505+0000 7f2cd2cff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc145920 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.505+0000 7f2cd2cff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc145dd0 con 0x7f2ccc0ceb90 2023-11-28T17:09:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.505+0000 7f2cd2cff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc0c42c0 con 0x7f2ccc15ae20 2023-11-28T17:09:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.505+0000 7f2cd1cfd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ccc0cdf90 0x7f2ccc0c3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.505+0000 7f2cd1cfd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ccc0cdf90 0x7f2ccc0c3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60946/0 (socket says 172.21.15.116:60946) 2023-11-28T17:09:32.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.505+0000 7f2cd1cfd640 1 -- 172.21.15.116:0/2255184694 learned_addr learned my addr 172.21.15.116:0/2255184694 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:32.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.506+0000 7f2cd1cfd640 1 -- 172.21.15.116:0/2255184694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ccc15ae20 msgr2=0x7f2ccc1574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:32.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.506+0000 7f2cd1cfd640 1 --2- 172.21.15.116:0/2255184694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ccc15ae20 0x7f2ccc1574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:32.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.506+0000 7f2cd14fc640 1 --2- 172.21.15.116:0/2255184694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ccc0ceb90 0x7f2ccc1453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:32.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.506+0000 7f2cd1cfd640 1 -- 172.21.15.116:0/2255184694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ccc0ceb90 msgr2=0x7f2ccc1453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:32.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.506+0000 7f2cd1cfd640 1 --2- 172.21.15.116:0/2255184694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ccc0ceb90 0x7f2ccc1453e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:32.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.506+0000 7f2cd1cfd640 1 -- 172.21.15.116:0/2255184694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc4009b30 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.506+0000 7f2cd1cfd640 1 --2- 172.21.15.116:0/2255184694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ccc0cdf90 0x7f2ccc0c3bc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800c380 tx=0x7f2cc800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:32.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.506+0000 7f2cc27fc640 1 -- 172.21.15.116:0/2255184694 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc80052d0 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.506+0000 7f2cc27fc640 1 -- 172.21.15.116:0/2255184694 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc8005450 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.506+0000 7f2cc27fc640 1 -- 172.21.15.116:0/2255184694 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8007260 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.506+0000 7f2cd2cff640 1 -- 172.21.15.116:0/2255184694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cbc009400 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.507+0000 7f2cd2cff640 1 -- 172.21.15.116:0/2255184694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ccc09a200 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.508+0000 7f2cd2cff640 1 -- 172.21.15.116:0/2255184694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ccc145b80 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.509+0000 7f2cc27fc640 1 -- 172.21.15.116:0/2255184694 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2cc80073c0 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.510+0000 7f2cc27fc640 1 --2- 172.21.15.116:0/2255184694 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2cb006b2a0 0x7f2cb006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.510+0000 7f2cc27fc640 1 -- 172.21.15.116:0/2255184694 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2cc8088f70 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.511+0000 7f2cd14fc640 1 --2- 172.21.15.116:0/2255184694 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2cb006b2a0 0x7f2cb006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.511+0000 7f2cd14fc640 1 --2- 172.21.15.116:0/2255184694 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2cb006b2a0 0x7f2cb006d760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc036000 tx=0x7f2cbc03d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:32.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.515+0000 7f2cc27fc640 1 -- 172.21.15.116:0/2255184694 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2cc80bbe90 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.705+0000 7f2cd2cff640 1 -- 172.21.15.116:0/2255184694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ccc09d0b0 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.707+0000 7f2cc27fc640 1 -- 172.21.15.116:0/2255184694 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16521 (secure 0 0 0) 0x7f2cc8053de0 con 0x7f2ccc0cdf90 2023-11-28T17:09:32.708 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:32.708 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:01.970957+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.709+0000 7f2cd2cff640 1 -- 172.21.15.116:0/2255184694 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2cb006b2a0 msgr2=0x7f2cb006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.709+0000 7f2cd2cff640 1 --2- 172.21.15.116:0/2255184694 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2cb006b2a0 0x7f2cb006d760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc036000 tx=0x7f2cbc03d040 comp rx=0 tx=0).stop 2023-11-28T17:09:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.709+0000 7f2cd2cff640 1 -- 172.21.15.116:0/2255184694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ccc0cdf90 msgr2=0x7f2ccc0c3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.709+0000 7f2cd2cff640 1 --2- 172.21.15.116:0/2255184694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ccc0cdf90 0x7f2ccc0c3bc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800c380 tx=0x7f2cc800c840 comp rx=0 tx=0).stop 2023-11-28T17:09:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.709+0000 7f2cd2cff640 1 -- 172.21.15.116:0/2255184694 shutdown_connections 2023-11-28T17:09:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.709+0000 7f2cd2cff640 1 --2- 172.21.15.116:0/2255184694 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2cb006b2a0 0x7f2cb006d760 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.709+0000 7f2cd2cff640 1 --2- 172.21.15.116:0/2255184694 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ccc15ae20 0x7f2ccc1574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.709+0000 7f2cd2cff640 1 --2- 172.21.15.116:0/2255184694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ccc0ceb90 0x7f2ccc1453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.709+0000 7f2cd2cff640 1 --2- 172.21.15.116:0/2255184694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ccc0cdf90 0x7f2ccc0c3bc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.709+0000 7f2cd2cff640 1 -- 172.21.15.116:0/2255184694 >> 172.21.15.116:0/2255184694 conn(0x7f2ccc0b0950 msgr2=0x7f2ccc0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.709+0000 7f2cd2cff640 1 -- 172.21.15.116:0/2255184694 shutdown_connections 2023-11-28T17:09:32.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.709+0000 7f2cd2cff640 1 -- 172.21.15.116:0/2255184694 wait complete. 2023-11-28T17:09:32.729 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:09:32.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:09:32.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.922+0000 7f31cf400640 1 Processor -- start 2023-11-28T17:09:32.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.922+0000 7f31cf400640 1 -- start start 2023-11-28T17:09:32.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.924+0000 7f31cf400640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f31c815b0c0 0x7f31c815ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:32.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.924+0000 7f31cf400640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f31c80ce940 0x7f31c80cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:32.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.924+0000 7f31cf400640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31c80cd320 con 0x7f31c8071b40 2023-11-28T17:09:32.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.924+0000 7f31cf400640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c80ced80 con 0x7f31c815b0c0 2023-11-28T17:09:32.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.924+0000 7f31cf400640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31c80cbee0 con 0x7f31c80ce940 2023-11-28T17:09:32.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.924+0000 7f31cd19c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f31c815b0c0 0x7f31c815ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:32.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.924+0000 7f31cd99d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f31c8071b40 0x7f31c815b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42450/0 (socket says 172.21.15.116:42450) 2023-11-28T17:09:32.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.924+0000 7f31cd99d640 1 -- 172.21.15.116:0/1259164900 learned_addr learned my addr 172.21.15.116:0/1259164900 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:32.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.924+0000 7f31cd19c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f31c815b0c0 0x7f31c815ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60960/0 (socket says 172.21.15.116:60960) 2023-11-28T17:09:32.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.924+0000 7f31cc99b640 1 --2- 172.21.15.116:0/1259164900 >> v2:172.21.15.156:3300/0 conn(0x7f31c80ce940 0x7f31c80cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:32.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.924+0000 7f31cd19c640 1 -- 172.21.15.116:0/1259164900 >> v2:172.21.15.156:3300/0 conn(0x7f31c80ce940 msgr2=0x7f31c80cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:32.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.924+0000 7f31cd19c640 1 --2- 172.21.15.116:0/1259164900 >> v2:172.21.15.156:3300/0 conn(0x7f31c80ce940 0x7f31c80cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:32.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.924+0000 7f31cd19c640 1 -- 172.21.15.116:0/1259164900 >> v1:172.21.15.116:6789/0 conn(0x7f31c8071b40 legacy=0x7f31c815b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:32.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.924+0000 7f31cd19c640 1 -- 172.21.15.116:0/1259164900 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c80cc710 con 0x7f31c815b0c0 2023-11-28T17:09:32.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.926+0000 7f31cd19c640 1 --2- 172.21.15.116:0/1259164900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c815b0c0 0x7f31c815ed10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f31c806bf90 tx=0x7f31c402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c499db6f918e05 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:32.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.926+0000 7f31bffff640 1 -- 172.21.15.116:0/1259164900 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c402e070 con 0x7f31c815b0c0 2023-11-28T17:09:32.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.926+0000 7f31bffff640 1 -- 172.21.15.116:0/1259164900 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c40315a0 con 0x7f31c815b0c0 2023-11-28T17:09:32.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.926+0000 7f31bffff640 1 -- 172.21.15.116:0/1259164900 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c4031930 con 0x7f31c815b0c0 2023-11-28T17:09:32.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.927+0000 7f31cf400640 1 -- 172.21.15.116:0/1259164900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c815b0c0 msgr2=0x7f31c815ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:32.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.927+0000 7f31cf400640 1 --2- 172.21.15.116:0/1259164900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c815b0c0 0x7f31c815ed10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f31c806bf90 tx=0x7f31c402b840 comp rx=0 tx=0).stop 2023-11-28T17:09:32.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.927+0000 7f31cf400640 1 -- 172.21.15.116:0/1259164900 shutdown_connections 2023-11-28T17:09:32.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.927+0000 7f31cf400640 1 --2- 172.21.15.116:0/1259164900 >> v2:172.21.15.156:3300/0 conn(0x7f31c80ce940 0x7f31c80cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:32.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.927+0000 7f31cf400640 1 --2- 172.21.15.116:0/1259164900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c815b0c0 0x7f31c815ed10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:32.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.927+0000 7f31cf400640 1 -- 172.21.15.116:0/1259164900 >> 172.21.15.116:0/1259164900 conn(0x7f31c80b0cf0 msgr2=0x7f31c80b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:32.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.927+0000 7f31cf400640 1 -- 172.21.15.116:0/1259164900 shutdown_connections 2023-11-28T17:09:32.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.927+0000 7f31cf400640 1 -- 172.21.15.116:0/1259164900 wait complete. 2023-11-28T17:09:32.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.929+0000 7f31cf400640 1 Processor -- start 2023-11-28T17:09:32.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.929+0000 7f31cf400640 1 -- start start 2023-11-28T17:09:32.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.930+0000 7f31cf400640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f31c8071b40 0x7f31c814a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:32.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.930+0000 7f31cf400640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31c80ce940 0x7f31c814f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:32.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.930+0000 7f31cf400640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c815b0c0 0x7f31c814f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:32.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.930+0000 7f31cf400640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c80add50 con 0x7f31c815b0c0 2023-11-28T17:09:32.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.930+0000 7f31cf400640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c8151aa0 con 0x7f31c80ce940 2023-11-28T17:09:32.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.930+0000 7f31cf400640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31c80ae170 con 0x7f31c8071b40 2023-11-28T17:09:32.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.930+0000 7f31cd99d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c815b0c0 0x7f31c814f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:32.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.931+0000 7f31cd99d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c815b0c0 0x7f31c814f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60976/0 (socket says 172.21.15.116:60976) 2023-11-28T17:09:32.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.931+0000 7f31cd99d640 1 -- 172.21.15.116:0/3113778601 learned_addr learned my addr 172.21.15.116:0/3113778601 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:32.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.931+0000 7f31cd19c640 1 --2- 172.21.15.116:0/3113778601 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f31c8071b40 0x7f31c814a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:32.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.931+0000 7f31cd99d640 1 -- 172.21.15.116:0/3113778601 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f31c8071b40 msgr2=0x7f31c814a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:32.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.931+0000 7f31cd99d640 1 --2- 172.21.15.116:0/3113778601 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f31c8071b40 0x7f31c814a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:32.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.931+0000 7f31cd99d640 1 -- 172.21.15.116:0/3113778601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31c80ce940 msgr2=0x7f31c814f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:32.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.931+0000 7f31cd99d640 1 --2- 172.21.15.116:0/3113778601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31c80ce940 0x7f31c814f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:32.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.931+0000 7f31cd99d640 1 -- 172.21.15.116:0/3113778601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c4009400 con 0x7f31c815b0c0 2023-11-28T17:09:32.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.931+0000 7f31cd99d640 1 --2- 172.21.15.116:0/3113778601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c815b0c0 0x7f31c814f660 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f31c000ca80 tx=0x7f31c000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:32.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.931+0000 7f31bdffb640 1 -- 172.21.15.116:0/3113778601 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c000ac30 con 0x7f31c815b0c0 2023-11-28T17:09:32.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.931+0000 7f31bdffb640 1 -- 172.21.15.116:0/3113778601 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c000ad90 con 0x7f31c815b0c0 2023-11-28T17:09:32.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.931+0000 7f31cf400640 1 -- 172.21.15.116:0/3113778601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c80b4100 con 0x7f31c815b0c0 2023-11-28T17:09:32.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.931+0000 7f31bdffb640 1 -- 172.21.15.116:0/3113778601 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c0012600 con 0x7f31c815b0c0 2023-11-28T17:09:32.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.931+0000 7f31cf400640 1 -- 172.21.15.116:0/3113778601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c80ad690 con 0x7f31c815b0c0 2023-11-28T17:09:32.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.932+0000 7f31cf400640 1 -- 172.21.15.116:0/3113778601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c80ade90 con 0x7f31c815b0c0 2023-11-28T17:09:32.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.933+0000 7f31bdffb640 1 -- 172.21.15.116:0/3113778601 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f31c0007070 con 0x7f31c815b0c0 2023-11-28T17:09:32.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.934+0000 7f31bdffb640 1 --2- 172.21.15.116:0/3113778601 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f31ac06b2a0 0x7f31ac06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:32.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.934+0000 7f31bdffb640 1 -- 172.21.15.116:0/3113778601 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31c00899b0 con 0x7f31c815b0c0 2023-11-28T17:09:32.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.935+0000 7f31cd19c640 1 --2- 172.21.15.116:0/3113778601 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f31ac06b2a0 0x7f31ac06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:32.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.935+0000 7f31cd19c640 1 --2- 172.21.15.116:0/3113778601 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f31ac06b2a0 0x7f31ac06d760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f31c4036000 tx=0x7f31c403d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:32.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:32.939+0000 7f31bdffb640 1 -- 172.21.15.116:0/3113778601 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31c80ade90 con 0x7f31c815b0c0 2023-11-28T17:09:33.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.122+0000 7f31cf400640 1 -- 172.21.15.116:0/3113778601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31c80ade90 con 0x7f31c815b0c0 2023-11-28T17:09:33.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.123+0000 7f31bdffb640 1 -- 172.21.15.116:0/3113778601 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31c80ade90 con 0x7f31c815b0c0 2023-11-28T17:09:33.124 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:09:33.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.125+0000 7f31cf400640 1 -- 172.21.15.116:0/3113778601 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f31ac06b2a0 msgr2=0x7f31ac06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:33.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.125+0000 7f31cf400640 1 --2- 172.21.15.116:0/3113778601 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f31ac06b2a0 0x7f31ac06d760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f31c4036000 tx=0x7f31c403d040 comp rx=0 tx=0).stop 2023-11-28T17:09:33.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.125+0000 7f31cf400640 1 -- 172.21.15.116:0/3113778601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c815b0c0 msgr2=0x7f31c814f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:33.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.125+0000 7f31cf400640 1 --2- 172.21.15.116:0/3113778601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c815b0c0 0x7f31c814f660 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f31c000ca80 tx=0x7f31c000cf40 comp rx=0 tx=0).stop 2023-11-28T17:09:33.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.125+0000 7f31cf400640 1 -- 172.21.15.116:0/3113778601 shutdown_connections 2023-11-28T17:09:33.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.125+0000 7f31cf400640 1 --2- 172.21.15.116:0/3113778601 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f31ac06b2a0 0x7f31ac06d760 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.125+0000 7f31cf400640 1 --2- 172.21.15.116:0/3113778601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31c815b0c0 0x7f31c814f660 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.125+0000 7f31cf400640 1 --2- 172.21.15.116:0/3113778601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31c80ce940 0x7f31c814f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.125+0000 7f31cf400640 1 --2- 172.21.15.116:0/3113778601 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f31c8071b40 0x7f31c814a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.125+0000 7f31cf400640 1 -- 172.21.15.116:0/3113778601 >> 172.21.15.116:0/3113778601 conn(0x7f31c80b0cf0 msgr2=0x7f31c80aec40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:33.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.125+0000 7f31cf400640 1 -- 172.21.15.116:0/3113778601 shutdown_connections 2023-11-28T17:09:33.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.125+0000 7f31cf400640 1 -- 172.21.15.116:0/3113778601 wait complete. 2023-11-28T17:09:33.143 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:09:33.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.350+0000 7fe9fc1a9640 1 Processor -- start 2023-11-28T17:09:33.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.350+0000 7fe9fc1a9640 1 -- start start 2023-11-28T17:09:33.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.352+0000 7fe9fc1a9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe9f413f640 0x7fe9f413fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:33.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.352+0000 7fe9fc1a9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9f4071d20 0x7fe9f4072130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:33.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.352+0000 7fe9fc1a9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f413dd10 con 0x7fe9f413cee0 2023-11-28T17:09:33.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.352+0000 7fe9fc1a9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f4072670 con 0x7fe9f413f640 2023-11-28T17:09:33.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.352+0000 7fe9fc1a9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f413d2c0 con 0x7fe9f4071d20 2023-11-28T17:09:33.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.352+0000 7fe9f9f45640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe9f413cee0 0x7fe9f413d600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42458/0 (socket says 172.21.15.116:42458) 2023-11-28T17:09:33.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.352+0000 7fe9f9f45640 1 -- 172.21.15.116:0/377186462 learned_addr learned my addr 172.21.15.116:0/377186462 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:33.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.352+0000 7fe9f9744640 1 --2- 172.21.15.116:0/377186462 >> v2:172.21.15.116:3300/0 conn(0x7fe9f413f640 0x7fe9f413fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:33.354 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.352+0000 7fe9f8f43640 1 -- 172.21.15.116:0/377186462 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221813276 0 0) 0x7fe9f413dd10 con 0x7fe9f413cee0 2023-11-28T17:09:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.352+0000 7fe9f8f43640 1 -- 172.21.15.116:0/377186462 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e0003520 con 0x7fe9f413cee0 2023-11-28T17:09:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.353+0000 7fe9fa746640 1 --2- 172.21.15.116:0/377186462 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9f4071d20 0x7fe9f4072130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.353+0000 7fe9f9744640 1 -- 172.21.15.116:0/377186462 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9f4071d20 msgr2=0x7fe9f4072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.353+0000 7fe9f9744640 1 --2- 172.21.15.116:0/377186462 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9f4071d20 0x7fe9f4072130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.353+0000 7fe9f9744640 1 -- 172.21.15.116:0/377186462 >> v1:172.21.15.116:6789/0 conn(0x7fe9f413cee0 legacy=0x7fe9f413d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:33.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.353+0000 7fe9f9744640 1 -- 172.21.15.116:0/377186462 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f41400e0 con 0x7fe9f413f640 2023-11-28T17:09:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.354+0000 7fe9f9744640 1 --2- 172.21.15.116:0/377186462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f413f640 0x7fe9f413fab0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40097a0 tx=0x7fe9e402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860c489a08b61f8f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.354+0000 7fe9f8f43640 1 -- 172.21.15.116:0/377186462 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e402f050 con 0x7fe9f413f640 2023-11-28T17:09:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.354+0000 7fe9f8f43640 1 -- 172.21.15.116:0/377186462 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e4031700 con 0x7fe9f413f640 2023-11-28T17:09:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.354+0000 7fe9f8f43640 1 -- 172.21.15.116:0/377186462 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4031a90 con 0x7fe9f413f640 2023-11-28T17:09:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.355+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/377186462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f413f640 msgr2=0x7fe9f413fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.355+0000 7fe9fc1a9640 1 --2- 172.21.15.116:0/377186462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f413f640 0x7fe9f413fab0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40097a0 tx=0x7fe9e402b7a0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.355+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/377186462 shutdown_connections 2023-11-28T17:09:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.355+0000 7fe9fc1a9640 1 --2- 172.21.15.116:0/377186462 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9f4071d20 0x7fe9f4072130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.355+0000 7fe9fc1a9640 1 --2- 172.21.15.116:0/377186462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f413f640 0x7fe9f413fab0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.355+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/377186462 >> 172.21.15.116:0/377186462 conn(0x7fe9f40b0cf0 msgr2=0x7fe9f40b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.355+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/377186462 shutdown_connections 2023-11-28T17:09:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.355+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/377186462 wait complete. 2023-11-28T17:09:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.356+0000 7fe9fc1a9640 1 Processor -- start 2023-11-28T17:09:33.358 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.357+0000 7fe9fc1a9640 1 -- start start 2023-11-28T17:09:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9fc1a9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9f4071d20 0x7fe9f413c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9fc1a9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9f413cee0 0x7fe9f4136730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9fc1a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f413f640 0x7fe9f40b27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9fc1a9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f413b160 con 0x7fe9f413f640 2023-11-28T17:09:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9fc1a9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f413bce0 con 0x7fe9f4071d20 2023-11-28T17:09:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9fc1a9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f413b3b0 con 0x7fe9f413cee0 2023-11-28T17:09:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9fa746640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f413f640 0x7fe9f40b27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9fa746640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f413f640 0x7fe9f40b27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32768/0 (socket says 172.21.15.116:32768) 2023-11-28T17:09:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9fa746640 1 -- 172.21.15.116:0/2940094197 learned_addr learned my addr 172.21.15.116:0/2940094197 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9f9744640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9f413cee0 0x7fe9f4136730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9fa746640 1 -- 172.21.15.116:0/2940094197 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9f413cee0 msgr2=0x7fe9f4136730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9fa746640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9f413cee0 0x7fe9f4136730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9fa746640 1 -- 172.21.15.116:0/2940094197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9f4071d20 msgr2=0x7fe9f413c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9f9f45640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9f4071d20 0x7fe9f413c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9fa746640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9f4071d20 0x7fe9f413c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9fa746640 1 -- 172.21.15.116:0/2940094197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e4009400 con 0x7fe9f413f640 2023-11-28T17:09:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.358+0000 7fe9f9744640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9f413cee0 0x7fe9f4136730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.359+0000 7fe9f9f45640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9f4071d20 0x7fe9f413c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.359+0000 7fe9fa746640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f413f640 0x7fe9f40b27c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec00cc60 tx=0x7fe9ec00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.359+0000 7fe9ea7fc640 1 -- 172.21.15.116:0/2940094197 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec00ce70 con 0x7fe9f413f640 2023-11-28T17:09:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.359+0000 7fe9ea7fc640 1 -- 172.21.15.116:0/2940094197 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ec003990 con 0x7fe9f413f640 2023-11-28T17:09:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.359+0000 7fe9ea7fc640 1 -- 172.21.15.116:0/2940094197 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec003d20 con 0x7fe9f413f640 2023-11-28T17:09:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.359+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/2940094197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f413b690 con 0x7fe9f413f640 2023-11-28T17:09:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.359+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/2940094197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f4136c70 con 0x7fe9f413f640 2023-11-28T17:09:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.360+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/2940094197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f41391a0 con 0x7fe9f413f640 2023-11-28T17:09:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.361+0000 7fe9ea7fc640 1 -- 172.21.15.116:0/2940094197 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe9ec003e80 con 0x7fe9f413f640 2023-11-28T17:09:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.362+0000 7fe9ea7fc640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe9d806b2a0 0x7fe9d806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.362+0000 7fe9ea7fc640 1 -- 172.21.15.116:0/2940094197 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9ec089f30 con 0x7fe9f413f640 2023-11-28T17:09:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.362+0000 7fe9f9f45640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe9d806b2a0 0x7fe9d806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.363+0000 7fe9f9f45640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe9d806b2a0 0x7fe9d806d760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe9f00011e0 tx=0x7fe9f0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:33.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.367+0000 7fe9ea7fc640 1 -- 172.21.15.116:0/2940094197 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9ec054dd0 con 0x7fe9f413f640 2023-11-28T17:09:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.541+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/2940094197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe9f4069360 con 0x7fe9f413f640 2023-11-28T17:09:33.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.542+0000 7fe9ea7fc640 1 -- 172.21.15.116:0/2940094197 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe9ec01d020 con 0x7fe9f413f640 2023-11-28T17:09:33.544 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:09:33.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.545+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/2940094197 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe9d806b2a0 msgr2=0x7fe9d806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:33.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.545+0000 7fe9fc1a9640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe9d806b2a0 0x7fe9d806d760 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe9f00011e0 tx=0x7fe9f0009040 comp rx=0 tx=0).stop 2023-11-28T17:09:33.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.545+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/2940094197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f413f640 msgr2=0x7fe9f40b27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:33.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.545+0000 7fe9fc1a9640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f413f640 0x7fe9f40b27c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec00cc60 tx=0x7fe9ec00a5c0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.545+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/2940094197 shutdown_connections 2023-11-28T17:09:33.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.545+0000 7fe9fc1a9640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe9d806b2a0 0x7fe9d806d760 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.545+0000 7fe9fc1a9640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9f413f640 0x7fe9f40b27c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.545+0000 7fe9fc1a9640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9f413cee0 0x7fe9f4136730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.545+0000 7fe9fc1a9640 1 --2- 172.21.15.116:0/2940094197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9f4071d20 0x7fe9f413c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.545+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/2940094197 >> 172.21.15.116:0/2940094197 conn(0x7fe9f40b0cf0 msgr2=0x7fe9f406c580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:33.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.545+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/2940094197 shutdown_connections 2023-11-28T17:09:33.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.545+0000 7fe9fc1a9640 1 -- 172.21.15.116:0/2940094197 wait complete. 2023-11-28T17:09:33.564 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:09:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.772+0000 7fb1dab1d640 1 Processor -- start 2023-11-28T17:09:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.772+0000 7fb1dab1d640 1 -- start start 2023-11-28T17:09:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.773+0000 7fb1dab1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1d415b0c0 0x7fb1d415ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.773+0000 7fb1dab1d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb1d40ce940 0x7fb1d40cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.774+0000 7fb1dab1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d40cd320 con 0x7fb1d4071b40 2023-11-28T17:09:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.774+0000 7fb1dab1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d40ced80 con 0x7fb1d415b0c0 2023-11-28T17:09:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.774+0000 7fb1dab1d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d40cbee0 con 0x7fb1d40ce940 2023-11-28T17:09:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.774+0000 7fb1d90ba640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb1d4071b40 0x7fb1d415b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42464/0 (socket says 172.21.15.116:42464) 2023-11-28T17:09:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.774+0000 7fb1d90ba640 1 -- 172.21.15.116:0/3135477185 learned_addr learned my addr 172.21.15.116:0/3135477185 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.774+0000 7fb1d88b9640 1 --2- 172.21.15.116:0/3135477185 >> v2:172.21.15.116:3300/0 conn(0x7fb1d415b0c0 0x7fb1d415ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.774+0000 7fb1d3fff640 1 --2- 172.21.15.116:0/3135477185 >> v2:172.21.15.156:3300/0 conn(0x7fb1d40ce940 0x7fb1d40cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.774+0000 7fb1d37fe640 1 -- 172.21.15.116:0/3135477185 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317488430 0 0) 0x7fb1d40cd320 con 0x7fb1d4071b40 2023-11-28T17:09:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.774+0000 7fb1d37fe640 1 -- 172.21.15.116:0/3135477185 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1c4003520 con 0x7fb1d4071b40 2023-11-28T17:09:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.774+0000 7fb1d88b9640 1 -- 172.21.15.116:0/3135477185 >> v2:172.21.15.156:3300/0 conn(0x7fb1d40ce940 msgr2=0x7fb1d40cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.774+0000 7fb1d88b9640 1 --2- 172.21.15.116:0/3135477185 >> v2:172.21.15.156:3300/0 conn(0x7fb1d40ce940 0x7fb1d40cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.774+0000 7fb1d88b9640 1 -- 172.21.15.116:0/3135477185 >> v1:172.21.15.116:6789/0 conn(0x7fb1d4071b40 legacy=0x7fb1d415b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.775+0000 7fb1d88b9640 1 -- 172.21.15.116:0/3135477185 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d40cc710 con 0x7fb1d415b0c0 2023-11-28T17:09:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.776+0000 7fb1d88b9640 1 --2- 172.21.15.116:0/3135477185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d415b0c0 0x7fb1d415ed10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb1d406bf90 tx=0x7fb1c802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79d9b503a0be04e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.776+0000 7fb1d37fe640 1 -- 172.21.15.116:0/3135477185 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c802e070 con 0x7fb1d415b0c0 2023-11-28T17:09:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.776+0000 7fb1d37fe640 1 -- 172.21.15.116:0/3135477185 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c80315a0 con 0x7fb1d415b0c0 2023-11-28T17:09:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.776+0000 7fb1d37fe640 1 -- 172.21.15.116:0/3135477185 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8031930 con 0x7fb1d415b0c0 2023-11-28T17:09:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.777+0000 7fb1dab1d640 1 -- 172.21.15.116:0/3135477185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d415b0c0 msgr2=0x7fb1d415ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.777+0000 7fb1dab1d640 1 --2- 172.21.15.116:0/3135477185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d415b0c0 0x7fb1d415ed10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb1d406bf90 tx=0x7fb1c802b840 comp rx=0 tx=0).stop 2023-11-28T17:09:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.777+0000 7fb1dab1d640 1 -- 172.21.15.116:0/3135477185 shutdown_connections 2023-11-28T17:09:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.777+0000 7fb1dab1d640 1 --2- 172.21.15.116:0/3135477185 >> v2:172.21.15.156:3300/0 conn(0x7fb1d40ce940 0x7fb1d40cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.777+0000 7fb1dab1d640 1 --2- 172.21.15.116:0/3135477185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d415b0c0 0x7fb1d415ed10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.777+0000 7fb1dab1d640 1 -- 172.21.15.116:0/3135477185 >> 172.21.15.116:0/3135477185 conn(0x7fb1d40b0cf0 msgr2=0x7fb1d40b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:33.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.777+0000 7fb1dab1d640 1 -- 172.21.15.116:0/3135477185 shutdown_connections 2023-11-28T17:09:33.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.777+0000 7fb1dab1d640 1 -- 172.21.15.116:0/3135477185 wait complete. 2023-11-28T17:09:33.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.779+0000 7fb1dab1d640 1 Processor -- start 2023-11-28T17:09:33.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.779+0000 7fb1dab1d640 1 -- start start 2023-11-28T17:09:33.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.780+0000 7fb1dab1d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1d4071b40 0x7fb1d4155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:33.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.780+0000 7fb1dab1d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb1d40ce940 0x7fb1d40c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:33.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.780+0000 7fb1dab1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d415b0c0 0x7fb1d40c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:33.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.780+0000 7fb1dab1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d40c54f0 con 0x7fb1d415b0c0 2023-11-28T17:09:33.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.780+0000 7fb1dab1d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d40c4970 con 0x7fb1d4071b40 2023-11-28T17:09:33.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.780+0000 7fb1dab1d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d40c4c50 con 0x7fb1d40ce940 2023-11-28T17:09:33.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.780+0000 7fb1d90ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d415b0c0 0x7fb1d40c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:33.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.780+0000 7fb1d90ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d415b0c0 0x7fb1d40c5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32796/0 (socket says 172.21.15.116:32796) 2023-11-28T17:09:33.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.780+0000 7fb1d90ba640 1 -- 172.21.15.116:0/637929854 learned_addr learned my addr 172.21.15.116:0/637929854 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:33.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.780+0000 7fb1d88b9640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1d4071b40 0x7fb1d4155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:33.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.780+0000 7fb1d3fff640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb1d40ce940 0x7fb1d40c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:33.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.780+0000 7fb1d90ba640 1 -- 172.21.15.116:0/637929854 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb1d40ce940 msgr2=0x7fb1d40c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:33.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.781+0000 7fb1d90ba640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb1d40ce940 0x7fb1d40c5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.781+0000 7fb1d90ba640 1 -- 172.21.15.116:0/637929854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1d4071b40 msgr2=0x7fb1d4155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:33.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.781+0000 7fb1d90ba640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1d4071b40 0x7fb1d4155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.781+0000 7fb1d90ba640 1 -- 172.21.15.116:0/637929854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c8009400 con 0x7fb1d415b0c0 2023-11-28T17:09:33.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.781+0000 7fb1d88b9640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1d4071b40 0x7fb1d4155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:33.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.781+0000 7fb1d90ba640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d415b0c0 0x7fb1d40c5e80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc00caa0 tx=0x7fb1cc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:33.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.781+0000 7fb1d17fa640 1 -- 172.21.15.116:0/637929854 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc00ac00 con 0x7fb1d415b0c0 2023-11-28T17:09:33.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.781+0000 7fb1d17fa640 1 -- 172.21.15.116:0/637929854 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1cc00ad60 con 0x7fb1d415b0c0 2023-11-28T17:09:33.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.781+0000 7fb1d17fa640 1 -- 172.21.15.116:0/637929854 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc012620 con 0x7fb1d415b0c0 2023-11-28T17:09:33.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.781+0000 7fb1dab1d640 1 -- 172.21.15.116:0/637929854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d40c4f30 con 0x7fb1d415b0c0 2023-11-28T17:09:33.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.781+0000 7fb1dab1d640 1 -- 172.21.15.116:0/637929854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d40c7980 con 0x7fb1d415b0c0 2023-11-28T17:09:33.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.782+0000 7fb1dab1d640 1 -- 172.21.15.116:0/637929854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d40c77c0 con 0x7fb1d415b0c0 2023-11-28T17:09:33.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.783+0000 7fb1d17fa640 1 -- 172.21.15.116:0/637929854 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb1cc007070 con 0x7fb1d415b0c0 2023-11-28T17:09:33.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.785+0000 7fb1d17fa640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb1b806b2a0 0x7fb1b806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:33.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.785+0000 7fb1d17fa640 1 -- 172.21.15.116:0/637929854 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1cc089be0 con 0x7fb1d415b0c0 2023-11-28T17:09:33.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.785+0000 7fb1d88b9640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb1b806b2a0 0x7fb1b806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:33.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.786+0000 7fb1d88b9640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb1b806b2a0 0x7fb1b806d760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8036000 tx=0x7fb1c803d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:33.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.789+0000 7fb1d17fa640 1 -- 172.21.15.116:0/637929854 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1cc054a30 con 0x7fb1d415b0c0 2023-11-28T17:09:33.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.978+0000 7fb1dab1d640 1 -- 172.21.15.116:0/637929854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb1d4069360 con 0x7fb1d415b0c0 2023-11-28T17:09:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.978+0000 7fb1d17fa640 1 -- 172.21.15.116:0/637929854 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb1cc017020 con 0x7fb1d415b0c0 2023-11-28T17:09:33.980 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:09:33.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.981+0000 7fb1dab1d640 1 -- 172.21.15.116:0/637929854 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb1b806b2a0 msgr2=0x7fb1b806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:33.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.981+0000 7fb1dab1d640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb1b806b2a0 0x7fb1b806d760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8036000 tx=0x7fb1c803d040 comp rx=0 tx=0).stop 2023-11-28T17:09:33.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.981+0000 7fb1dab1d640 1 -- 172.21.15.116:0/637929854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d415b0c0 msgr2=0x7fb1d40c5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:33.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.981+0000 7fb1dab1d640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d415b0c0 0x7fb1d40c5e80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc00caa0 tx=0x7fb1cc00cf60 comp rx=0 tx=0).stop 2023-11-28T17:09:33.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.981+0000 7fb1dab1d640 1 -- 172.21.15.116:0/637929854 shutdown_connections 2023-11-28T17:09:33.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.981+0000 7fb1dab1d640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb1b806b2a0 0x7fb1b806d760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.981+0000 7fb1dab1d640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1d415b0c0 0x7fb1d40c5e80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.981+0000 7fb1dab1d640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb1d40ce940 0x7fb1d40c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.981+0000 7fb1dab1d640 1 --2- 172.21.15.116:0/637929854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1d4071b40 0x7fb1d4155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:33.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.981+0000 7fb1dab1d640 1 -- 172.21.15.116:0/637929854 >> 172.21.15.116:0/637929854 conn(0x7fb1d40b0cf0 msgr2=0x7fb1d40af450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:33.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.981+0000 7fb1dab1d640 1 -- 172.21.15.116:0/637929854 shutdown_connections 2023-11-28T17:09:33.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:33.981+0000 7fb1dab1d640 1 -- 172.21.15.116:0/637929854 wait complete. 2023-11-28T17:09:34.001 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:34.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.207+0000 7f2869fc9640 1 Processor -- start 2023-11-28T17:09:34.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.207+0000 7f2869fc9640 1 -- start start 2023-11-28T17:09:34.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.209+0000 7f2869fc9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2864159f30 0x7f28640701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:34.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.209+0000 7f2869fc9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640cf2a0 0x7f28640ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:34.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.209+0000 7f2869fc9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28640cd8e0 con 0x7f286415ae90 2023-11-28T17:09:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.209+0000 7f2869fc9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2864070730 con 0x7f2864159f30 2023-11-28T17:09:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.209+0000 7f2869fc9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2864070ee0 con 0x7f28640cf2a0 2023-11-28T17:09:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.209+0000 7f2868fc7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2864159f30 0x7f28640701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.209+0000 7f2863fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f286415ae90 0x7f28640ceb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42478/0 (socket says 172.21.15.116:42478) 2023-11-28T17:09:34.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.209+0000 7f2863fff640 1 -- 172.21.15.116:0/3141697837 learned_addr learned my addr 172.21.15.116:0/3141697837 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:34.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.210+0000 7f28697c8640 1 --2- 172.21.15.116:0/3141697837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640cf2a0 0x7f28640ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:34.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.210+0000 7f2868fc7640 1 -- 172.21.15.116:0/3141697837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640cf2a0 msgr2=0x7f28640ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:34.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.210+0000 7f2868fc7640 1 --2- 172.21.15.116:0/3141697837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640cf2a0 0x7f28640ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.210+0000 7f2868fc7640 1 -- 172.21.15.116:0/3141697837 >> v1:172.21.15.116:6789/0 conn(0x7f286415ae90 legacy=0x7f28640ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:34.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.210+0000 7f2868fc7640 1 -- 172.21.15.116:0/3141697837 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286415b2f0 con 0x7f2864159f30 2023-11-28T17:09:34.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.210+0000 7f28697c8640 1 --2- 172.21.15.116:0/3141697837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640cf2a0 0x7f28640ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:34.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.210+0000 7f28637fe640 1 -- 172.21.15.116:0/3141697837 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001392798 0 0) 0x7f28640cd8e0 con 0x7f286415ae90 2023-11-28T17:09:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.211+0000 7f2868fc7640 1 --2- 172.21.15.116:0/3141697837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2864159f30 0x7f28640701f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f285800ba40 tx=0x7f285802e080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4765f761d2e5b7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.211+0000 7f28637fe640 1 -- 172.21.15.116:0/3141697837 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285802ee30 con 0x7f2864159f30 2023-11-28T17:09:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.211+0000 7f28637fe640 1 -- 172.21.15.116:0/3141697837 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2858036440 con 0x7f2864159f30 2023-11-28T17:09:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.211+0000 7f28637fe640 1 -- 172.21.15.116:0/3141697837 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28580367d0 con 0x7f2864159f30 2023-11-28T17:09:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.212+0000 7f2869fc9640 1 -- 172.21.15.116:0/3141697837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2864159f30 msgr2=0x7f28640701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.212+0000 7f2869fc9640 1 --2- 172.21.15.116:0/3141697837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2864159f30 0x7f28640701f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f285800ba40 tx=0x7f285802e080 comp rx=0 tx=0).stop 2023-11-28T17:09:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.212+0000 7f2869fc9640 1 -- 172.21.15.116:0/3141697837 shutdown_connections 2023-11-28T17:09:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.212+0000 7f2869fc9640 1 --2- 172.21.15.116:0/3141697837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28640cf2a0 0x7f28640ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.212+0000 7f2869fc9640 1 --2- 172.21.15.116:0/3141697837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2864159f30 0x7f28640701f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.212+0000 7f2869fc9640 1 -- 172.21.15.116:0/3141697837 >> 172.21.15.116:0/3141697837 conn(0x7f28640b0950 msgr2=0x7f28640b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.212+0000 7f2869fc9640 1 -- 172.21.15.116:0/3141697837 shutdown_connections 2023-11-28T17:09:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.212+0000 7f2869fc9640 1 -- 172.21.15.116:0/3141697837 wait complete. 2023-11-28T17:09:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.214+0000 7f2869fc9640 1 Processor -- start 2023-11-28T17:09:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.214+0000 7f2869fc9640 1 -- start start 2023-11-28T17:09:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.215+0000 7f2869fc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28640cf2a0 0x7f2864154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.215+0000 7f2869fc9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2864159f30 0x7f286406d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.215+0000 7f2869fc9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f286415ae90 0x7f286406d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.215+0000 7f2869fc9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286409a9d0 con 0x7f28640cf2a0 2023-11-28T17:09:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.215+0000 7f2869fc9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2864099a00 con 0x7f286415ae90 2023-11-28T17:09:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.215+0000 7f2869fc9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28641545a0 con 0x7f2864159f30 2023-11-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.215+0000 7f2868fc7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28640cf2a0 0x7f2864154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.215+0000 7f2868fc7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28640cf2a0 0x7f2864154060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32800/0 (socket says 172.21.15.116:32800) 2023-11-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.215+0000 7f2868fc7640 1 -- 172.21.15.116:0/19479126 learned_addr learned my addr 172.21.15.116:0/19479126 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.216+0000 7f2868fc7640 1 -- 172.21.15.116:0/19479126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2864159f30 msgr2=0x7f286406d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.216+0000 7f2863fff640 1 --2- 172.21.15.116:0/19479126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2864159f30 0x7f286406d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.216+0000 7f28697c8640 1 --2- 172.21.15.116:0/19479126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f286415ae90 0x7f286406d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.216+0000 7f2868fc7640 1 --2- 172.21.15.116:0/19479126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2864159f30 0x7f286406d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.216+0000 7f2868fc7640 1 -- 172.21.15.116:0/19479126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f286415ae90 msgr2=0x7f286406d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.216+0000 7f2868fc7640 1 --2- 172.21.15.116:0/19479126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f286415ae90 0x7f286406d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.216+0000 7f2868fc7640 1 -- 172.21.15.116:0/19479126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285800b6a0 con 0x7f28640cf2a0 2023-11-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.216+0000 7f2868fc7640 1 --2- 172.21.15.116:0/19479126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28640cf2a0 0x7f2864154060 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2858039040 tx=0x7f28580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.216+0000 7f28617fa640 1 -- 172.21.15.116:0/19479126 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858034970 con 0x7f28640cf2a0 2023-11-28T17:09:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.216+0000 7f28617fa640 1 -- 172.21.15.116:0/19479126 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2858034b10 con 0x7f28640cf2a0 2023-11-28T17:09:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.216+0000 7f28617fa640 1 -- 172.21.15.116:0/19479126 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285803f440 con 0x7f28640cf2a0 2023-11-28T17:09:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.216+0000 7f2869fc9640 1 -- 172.21.15.116:0/19479126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2864099c10 con 0x7f28640cf2a0 2023-11-28T17:09:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.216+0000 7f2869fc9640 1 -- 172.21.15.116:0/19479126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286409a1d0 con 0x7f28640cf2a0 2023-11-28T17:09:34.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.218+0000 7f2869fc9640 1 -- 172.21.15.116:0/19479126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286409d0b0 con 0x7f28640cf2a0 2023-11-28T17:09:34.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.218+0000 7f28617fa640 1 -- 172.21.15.116:0/19479126 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f285803d070 con 0x7f28640cf2a0 2023-11-28T17:09:34.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.219+0000 7f28617fa640 1 --2- 172.21.15.116:0/19479126 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f284806b2a0 0x7f284806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:34.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.219+0000 7f28617fa640 1 -- 172.21.15.116:0/19479126 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28580ac470 con 0x7f28640cf2a0 2023-11-28T17:09:34.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.220+0000 7f2863fff640 1 --2- 172.21.15.116:0/19479126 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f284806b2a0 0x7f284806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:34.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.220+0000 7f2863fff640 1 --2- 172.21.15.116:0/19479126 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f284806b2a0 0x7f284806d760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2850007fd0 tx=0x7f2850009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:34.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.224+0000 7f28617fa640 1 -- 172.21.15.116:0/19479126 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2858077370 con 0x7f28640cf2a0 2023-11-28T17:09:34.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.417+0000 7f2869fc9640 1 -- 172.21.15.116:0/19479126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f286404e960 con 0x7f28640cf2a0 2023-11-28T17:09:34.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.420+0000 7f28617fa640 1 -- 172.21.15.116:0/19479126 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16521 (secure 0 0 0) 0x7f285807a990 con 0x7f28640cf2a0 2023-11-28T17:09:34.422 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:34.422 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:01.970957+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:34.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.423+0000 7f2869fc9640 1 -- 172.21.15.116:0/19479126 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f284806b2a0 msgr2=0x7f284806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:34.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.423+0000 7f2869fc9640 1 --2- 172.21.15.116:0/19479126 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f284806b2a0 0x7f284806d760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2850007fd0 tx=0x7f2850009040 comp rx=0 tx=0).stop 2023-11-28T17:09:34.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.423+0000 7f2869fc9640 1 -- 172.21.15.116:0/19479126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28640cf2a0 msgr2=0x7f2864154060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:34.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.423+0000 7f2869fc9640 1 --2- 172.21.15.116:0/19479126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28640cf2a0 0x7f2864154060 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2858039040 tx=0x7f28580046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.423+0000 7f2869fc9640 1 -- 172.21.15.116:0/19479126 shutdown_connections 2023-11-28T17:09:34.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.423+0000 7f2869fc9640 1 --2- 172.21.15.116:0/19479126 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f284806b2a0 0x7f284806d760 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.423+0000 7f2869fc9640 1 --2- 172.21.15.116:0/19479126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f286415ae90 0x7f286406d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.423+0000 7f2869fc9640 1 --2- 172.21.15.116:0/19479126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2864159f30 0x7f286406d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.423+0000 7f2869fc9640 1 --2- 172.21.15.116:0/19479126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28640cf2a0 0x7f2864154060 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.423+0000 7f2869fc9640 1 -- 172.21.15.116:0/19479126 >> 172.21.15.116:0/19479126 conn(0x7f28640b0950 msgr2=0x7f28640acd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:34.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.424+0000 7f2869fc9640 1 -- 172.21.15.116:0/19479126 shutdown_connections 2023-11-28T17:09:34.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.424+0000 7f2869fc9640 1 -- 172.21.15.116:0/19479126 wait complete. 2023-11-28T17:09:34.443 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:09:34.443 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:09:34.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.660+0000 7f50a2ba9640 1 Processor -- start 2023-11-28T17:09:34.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.660+0000 7f50a2ba9640 1 -- start start 2023-11-28T17:09:34.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.662+0000 7f50a2ba9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f509c13cee0 0x7f509c13d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.662+0000 7f50a2ba9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f509c13f640 0x7f509c13fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.662+0000 7f50a2ba9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f509c071d20 0x7f509c072130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.662+0000 7f50a2ba9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509c072670 con 0x7f509c13cee0 2023-11-28T17:09:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.662+0000 7f50a2ba9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509c13d2c0 con 0x7f509c13f640 2023-11-28T17:09:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.662+0000 7f50a2ba9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509c1400e0 con 0x7f509c071d20 2023-11-28T17:09:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.662+0000 7f50a0945640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f509c13cee0 0x7f509c13d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.663+0000 7f50a0945640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f509c13cee0 0x7f509c13d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32810/0 (socket says 172.21.15.116:32810) 2023-11-28T17:09:34.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.663+0000 7f50a0945640 1 -- 172.21.15.116:0/2530618615 learned_addr learned my addr 172.21.15.116:0/2530618615 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:34.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.663+0000 7f509bfff640 1 --2- 172.21.15.116:0/2530618615 >> v2:172.21.15.156:3300/0 conn(0x7f509c13f640 0x7f509c13fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:34.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.663+0000 7f50a1146640 1 --2- 172.21.15.116:0/2530618615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f509c071d20 0x7f509c072130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:34.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.663+0000 7f50a0945640 1 -- 172.21.15.116:0/2530618615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f509c071d20 msgr2=0x7f509c072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:34.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.663+0000 7f50a0945640 1 --2- 172.21.15.116:0/2530618615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f509c071d20 0x7f509c072130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.663+0000 7f50a0945640 1 -- 172.21.15.116:0/2530618615 >> v2:172.21.15.156:3300/0 conn(0x7f509c13f640 msgr2=0x7f509c13fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:34.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.663+0000 7f50a0945640 1 --2- 172.21.15.116:0/2530618615 >> v2:172.21.15.156:3300/0 conn(0x7f509c13f640 0x7f509c13fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.663+0000 7f50a0945640 1 -- 172.21.15.116:0/2530618615 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509c1403c0 con 0x7f509c13cee0 2023-11-28T17:09:34.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.664+0000 7f50a0945640 1 --2- 172.21.15.116:0/2530618615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f509c13cee0 0x7f509c13d600 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f509c143c70 tx=0x7f509002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2744dd36171b3f30 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:34.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.664+0000 7f509b7fe640 1 -- 172.21.15.116:0/2530618615 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5090031750 con 0x7f509c13cee0 2023-11-28T17:09:34.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.665+0000 7f509b7fe640 1 -- 172.21.15.116:0/2530618615 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50900318b0 con 0x7f509c13cee0 2023-11-28T17:09:34.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.665+0000 7f509b7fe640 1 -- 172.21.15.116:0/2530618615 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5090031c40 con 0x7f509c13cee0 2023-11-28T17:09:34.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.665+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2530618615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f509c13cee0 msgr2=0x7f509c13d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:34.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.665+0000 7f50a2ba9640 1 --2- 172.21.15.116:0/2530618615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f509c13cee0 0x7f509c13d600 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f509c143c70 tx=0x7f509002b7f0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.665+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2530618615 shutdown_connections 2023-11-28T17:09:34.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.665+0000 7f50a2ba9640 1 --2- 172.21.15.116:0/2530618615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f509c071d20 0x7f509c072130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.665+0000 7f50a2ba9640 1 --2- 172.21.15.116:0/2530618615 >> v2:172.21.15.156:3300/0 conn(0x7f509c13f640 0x7f509c13fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.665+0000 7f50a2ba9640 1 --2- 172.21.15.116:0/2530618615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f509c13cee0 0x7f509c13d600 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.665+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2530618615 >> 172.21.15.116:0/2530618615 conn(0x7f509c0b0cf0 msgr2=0x7f509c0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:34.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.665+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2530618615 shutdown_connections 2023-11-28T17:09:34.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.665+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2530618615 wait complete. 2023-11-28T17:09:34.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.667+0000 7f50a2ba9640 1 Processor -- start 2023-11-28T17:09:34.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.667+0000 7f50a2ba9640 1 -- start start 2023-11-28T17:09:34.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.668+0000 7f50a2ba9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f509c071d20 0x7f509c13baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:34.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.668+0000 7f50a2ba9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f509c13cee0 0x7f509c134500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:34.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.668+0000 7f50a2ba9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f509c13f640 0x7f509c134a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:34.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.668+0000 7f50a2ba9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c0adbd0 con 0x7f509c13f640 2023-11-28T17:09:34.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.668+0000 7f50a2ba9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c142680 con 0x7f509c13cee0 2023-11-28T17:09:34.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.668+0000 7f50a2ba9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509c166a10 con 0x7f509c071d20 2023-11-28T17:09:34.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.668+0000 7f50a1146640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f509c13f640 0x7f509c134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:34.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.668+0000 7f50a1146640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f509c13f640 0x7f509c134a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32812/0 (socket says 172.21.15.116:32812) 2023-11-28T17:09:34.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.668+0000 7f50a1146640 1 -- 172.21.15.116:0/2971624581 learned_addr learned my addr 172.21.15.116:0/2971624581 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.669+0000 7f50a1146640 1 -- 172.21.15.116:0/2971624581 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f509c071d20 msgr2=0x7f509c13baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.669+0000 7f50a1146640 1 --2- 172.21.15.116:0/2971624581 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f509c071d20 0x7f509c13baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.669+0000 7f509bfff640 1 --2- 172.21.15.116:0/2971624581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f509c13cee0 0x7f509c134500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.669+0000 7f50a1146640 1 -- 172.21.15.116:0/2971624581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f509c13cee0 msgr2=0x7f509c134500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.669+0000 7f50a1146640 1 --2- 172.21.15.116:0/2971624581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f509c13cee0 0x7f509c134500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.669+0000 7f50a1146640 1 -- 172.21.15.116:0/2971624581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50900094d0 con 0x7f509c13f640 2023-11-28T17:09:34.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.669+0000 7f509bfff640 1 --2- 172.21.15.116:0/2971624581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f509c13cee0 0x7f509c134500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.669+0000 7f50a1146640 1 --2- 172.21.15.116:0/2971624581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f509c13f640 0x7f509c134a40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5094007e40 tx=0x7f509400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.669+0000 7f50997fa640 1 -- 172.21.15.116:0/2971624581 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094014620 con 0x7f509c13f640 2023-11-28T17:09:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.669+0000 7f50997fa640 1 -- 172.21.15.116:0/2971624581 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509400ee90 con 0x7f509c13f640 2023-11-28T17:09:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.669+0000 7f50997fa640 1 -- 172.21.15.116:0/2971624581 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50940149e0 con 0x7f509c13f640 2023-11-28T17:09:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.669+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2971624581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509c166cf0 con 0x7f509c13f640 2023-11-28T17:09:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.670+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2971624581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509c166630 con 0x7f509c13f640 2023-11-28T17:09:34.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.671+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2971624581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509c0add10 con 0x7f509c13f640 2023-11-28T17:09:34.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.671+0000 7f50997fa640 1 -- 172.21.15.116:0/2971624581 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f5094014b40 con 0x7f509c13f640 2023-11-28T17:09:34.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.672+0000 7f50997fa640 1 --2- 172.21.15.116:0/2971624581 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f508006b2a0 0x7f508006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:34.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.673+0000 7f50997fa640 1 -- 172.21.15.116:0/2971624581 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f509408a600 con 0x7f509c13f640 2023-11-28T17:09:34.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.673+0000 7f50a0945640 1 --2- 172.21.15.116:0/2971624581 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f508006b2a0 0x7f508006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:34.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.673+0000 7f50a0945640 1 --2- 172.21.15.116:0/2971624581 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f508006b2a0 0x7f508006d760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5090035000 tx=0x7f5090030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:34.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.677+0000 7f50997fa640 1 -- 172.21.15.116:0/2971624581 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f509c0add10 con 0x7f509c13f640 2023-11-28T17:09:34.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.863+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2971624581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f509c0add10 con 0x7f509c13f640 2023-11-28T17:09:34.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.865+0000 7f50997fa640 1 -- 172.21.15.116:0/2971624581 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f509c0add10 con 0x7f509c13f640 2023-11-28T17:09:34.866 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:09:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.867+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2971624581 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f508006b2a0 msgr2=0x7f508006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.867+0000 7f50a2ba9640 1 --2- 172.21.15.116:0/2971624581 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f508006b2a0 0x7f508006d760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5090035000 tx=0x7f5090030000 comp rx=0 tx=0).stop 2023-11-28T17:09:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.867+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2971624581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f509c13f640 msgr2=0x7f509c134a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.867+0000 7f50a2ba9640 1 --2- 172.21.15.116:0/2971624581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f509c13f640 0x7f509c134a40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5094007e40 tx=0x7f509400e4b0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.867+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2971624581 shutdown_connections 2023-11-28T17:09:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.867+0000 7f50a2ba9640 1 --2- 172.21.15.116:0/2971624581 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f508006b2a0 0x7f508006d760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.867+0000 7f50a2ba9640 1 --2- 172.21.15.116:0/2971624581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f509c13f640 0x7f509c134a40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.867+0000 7f50a2ba9640 1 --2- 172.21.15.116:0/2971624581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f509c13cee0 0x7f509c134500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.867+0000 7f50a2ba9640 1 --2- 172.21.15.116:0/2971624581 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f509c071d20 0x7f509c13baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:34.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.867+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2971624581 >> 172.21.15.116:0/2971624581 conn(0x7f509c0b0cf0 msgr2=0x7f509c0b0230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:34.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.867+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2971624581 shutdown_connections 2023-11-28T17:09:34.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:34.867+0000 7f50a2ba9640 1 -- 172.21.15.116:0/2971624581 wait complete. 2023-11-28T17:09:34.885 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:09:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.100+0000 7fc00b59e640 1 Processor -- start 2023-11-28T17:09:35.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.101+0000 7fc00b59e640 1 -- start start 2023-11-28T17:09:35.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.102+0000 7fc00b59e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc00c0d0700 0x7fc00c0d1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.102+0000 7fc00b59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc00c0b4010 0x7fc00c0c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.102+0000 7fc00b59e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00c0d0da0 con 0x7fc00c160b80 2023-11-28T17:09:35.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.102+0000 7fc00b59e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc00c160f60 con 0x7fc00c0b4010 2023-11-28T17:09:35.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.102+0000 7fc00b59e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc00c0d1080 con 0x7fc00c0d0700 2023-11-28T17:09:35.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.102+0000 7fc00a59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc00c160b80 0x7fc00c0d1360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42494/0 (socket says 172.21.15.116:42494) 2023-11-28T17:09:35.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.102+0000 7fc00a59c640 1 -- 172.21.15.116:0/738101124 learned_addr learned my addr 172.21.15.116:0/738101124 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:35.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.103+0000 7fc00ad9d640 1 --2- 172.21.15.116:0/738101124 >> v2:172.21.15.116:3300/0 conn(0x7fc00c0b4010 0x7fc00c0c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.103+0000 7fc009d9b640 1 --2- 172.21.15.116:0/738101124 >> v2:172.21.15.156:3300/0 conn(0x7fc00c0d0700 0x7fc00c0d1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.103+0000 7fc00959a640 1 -- 172.21.15.116:0/738101124 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227024618 0 0) 0x7fc00c0d0da0 con 0x7fc00c160b80 2023-11-28T17:09:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.103+0000 7fc00ad9d640 1 -- 172.21.15.116:0/738101124 >> v2:172.21.15.156:3300/0 conn(0x7fc00c0d0700 msgr2=0x7fc00c0d1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.103+0000 7fc00ad9d640 1 --2- 172.21.15.116:0/738101124 >> v2:172.21.15.156:3300/0 conn(0x7fc00c0d0700 0x7fc00c0d1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.103+0000 7fc00ad9d640 1 -- 172.21.15.116:0/738101124 >> v1:172.21.15.116:6789/0 conn(0x7fc00c160b80 legacy=0x7fc00c0d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.103+0000 7fc00ad9d640 1 -- 172.21.15.116:0/738101124 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c0ce780 con 0x7fc00c0b4010 2023-11-28T17:09:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.104+0000 7fc00ad9d640 1 --2- 172.21.15.116:0/738101124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc00c0b4010 0x7fc00c0c6c10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc00c08be50 tx=0x7fc00402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8a651efbc1d091 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.104+0000 7fc00959a640 1 -- 172.21.15.116:0/738101124 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004031830 con 0x7fc00c0b4010 2023-11-28T17:09:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.104+0000 7fc00959a640 1 -- 172.21.15.116:0/738101124 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc004031990 con 0x7fc00c0b4010 2023-11-28T17:09:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.104+0000 7fc00959a640 1 -- 172.21.15.116:0/738101124 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004031d20 con 0x7fc00c0b4010 2023-11-28T17:09:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.105+0000 7fc00b59e640 1 -- 172.21.15.116:0/738101124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc00c0b4010 msgr2=0x7fc00c0c6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.105+0000 7fc00b59e640 1 --2- 172.21.15.116:0/738101124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc00c0b4010 0x7fc00c0c6c10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc00c08be50 tx=0x7fc00402b8d0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.105+0000 7fc00b59e640 1 -- 172.21.15.116:0/738101124 shutdown_connections 2023-11-28T17:09:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.105+0000 7fc00b59e640 1 --2- 172.21.15.116:0/738101124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc00c0b4010 0x7fc00c0c6c10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.105+0000 7fc00b59e640 1 --2- 172.21.15.116:0/738101124 >> v2:172.21.15.156:3300/0 conn(0x7fc00c0d0700 0x7fc00c0d1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.105+0000 7fc00b59e640 1 -- 172.21.15.116:0/738101124 >> 172.21.15.116:0/738101124 conn(0x7fc00c0d2fc0 msgr2=0x7fc00c0d33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.105+0000 7fc00b59e640 1 -- 172.21.15.116:0/738101124 shutdown_connections 2023-11-28T17:09:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.105+0000 7fc00b59e640 1 -- 172.21.15.116:0/738101124 wait complete. 2023-11-28T17:09:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.107+0000 7fc00b59e640 1 Processor -- start 2023-11-28T17:09:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.107+0000 7fc00b59e640 1 -- start start 2023-11-28T17:09:35.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.108+0000 7fc00b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc00c0b4010 0x7fc00c15b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.108+0000 7fc00b59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc00c160b80 0x7fc00c09e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.108+0000 7fc00b59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc00c09e950 0x7fc00c0ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.108+0000 7fc00b59e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c09edf0 con 0x7fc00c0b4010 2023-11-28T17:09:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.108+0000 7fc00b59e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c09f0d0 con 0x7fc00c09e950 2023-11-28T17:09:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.108+0000 7fc00b59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c09f3b0 con 0x7fc00c160b80 2023-11-28T17:09:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.108+0000 7fc00a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc00c0b4010 0x7fc00c15b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.108+0000 7fc00a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc00c0b4010 0x7fc00c15b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32824/0 (socket says 172.21.15.116:32824) 2023-11-28T17:09:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.108+0000 7fc00a59c640 1 -- 172.21.15.116:0/169445565 learned_addr learned my addr 172.21.15.116:0/169445565 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.109+0000 7fc00a59c640 1 -- 172.21.15.116:0/169445565 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc00c160b80 msgr2=0x7fc00c09e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.109+0000 7fc00a59c640 1 --2- 172.21.15.116:0/169445565 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc00c160b80 0x7fc00c09e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.109+0000 7fc00a59c640 1 -- 172.21.15.116:0/169445565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc00c09e950 msgr2=0x7fc00c0ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.109+0000 7fc00a59c640 1 --2- 172.21.15.116:0/169445565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc00c09e950 0x7fc00c0ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.109+0000 7fc00a59c640 1 -- 172.21.15.116:0/169445565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc004009400 con 0x7fc00c0b4010 2023-11-28T17:09:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.109+0000 7fc00a59c640 1 --2- 172.21.15.116:0/169445565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc00c0b4010 0x7fc00c15b870 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbffc00c7c0 tx=0x7fbffc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.109+0000 7fc002ffd640 1 -- 172.21.15.116:0/169445565 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc00a640 con 0x7fc00c0b4010 2023-11-28T17:09:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.109+0000 7fc002ffd640 1 -- 172.21.15.116:0/169445565 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbffc00a7c0 con 0x7fc00c0b4010 2023-11-28T17:09:35.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.109+0000 7fc002ffd640 1 -- 172.21.15.116:0/169445565 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc00ab70 con 0x7fc00c0b4010 2023-11-28T17:09:35.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.109+0000 7fc00b59e640 1 -- 172.21.15.116:0/169445565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc00c09f690 con 0x7fc00c0b4010 2023-11-28T17:09:35.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.109+0000 7fc00b59e640 1 -- 172.21.15.116:0/169445565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc00c0b86e0 con 0x7fc00c0b4010 2023-11-28T17:09:35.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.110+0000 7fc00b59e640 1 -- 172.21.15.116:0/169445565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00c067960 con 0x7fc00c0b4010 2023-11-28T17:09:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.111+0000 7fc002ffd640 1 -- 172.21.15.116:0/169445565 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fbffc00acd0 con 0x7fc00c0b4010 2023-11-28T17:09:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.112+0000 7fc002ffd640 1 --2- 172.21.15.116:0/169445565 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbfec06b2a0 0x7fbfec06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.112+0000 7fc002ffd640 1 -- 172.21.15.116:0/169445565 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbffc0891e0 con 0x7fc00c0b4010 2023-11-28T17:09:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.113+0000 7fc009d9b640 1 --2- 172.21.15.116:0/169445565 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbfec06b2a0 0x7fbfec06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.113+0000 7fc009d9b640 1 --2- 172.21.15.116:0/169445565 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbfec06b2a0 0x7fbfec06d760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0b53a0 tx=0x7fbff4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:35.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.116+0000 7fc002ffd640 1 -- 172.21.15.116:0/169445565 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbffc053fb0 con 0x7fc00c0b4010 2023-11-28T17:09:35.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.306+0000 7fc00b59e640 1 -- 172.21.15.116:0/169445565 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc00c0a0eb0 con 0x7fc00c0b4010 2023-11-28T17:09:35.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.306+0000 7fc002ffd640 1 -- 172.21.15.116:0/169445565 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbffc018020 con 0x7fc00c0b4010 2023-11-28T17:09:35.308 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:09:35.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.309+0000 7fc00b59e640 1 -- 172.21.15.116:0/169445565 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbfec06b2a0 msgr2=0x7fbfec06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.309+0000 7fc00b59e640 1 --2- 172.21.15.116:0/169445565 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbfec06b2a0 0x7fbfec06d760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0b53a0 tx=0x7fbff4008040 comp rx=0 tx=0).stop 2023-11-28T17:09:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.309+0000 7fc00b59e640 1 -- 172.21.15.116:0/169445565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc00c0b4010 msgr2=0x7fc00c15b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.309+0000 7fc00b59e640 1 --2- 172.21.15.116:0/169445565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc00c0b4010 0x7fc00c15b870 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbffc00c7c0 tx=0x7fbffc00cc80 comp rx=0 tx=0).stop 2023-11-28T17:09:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.309+0000 7fc00b59e640 1 -- 172.21.15.116:0/169445565 shutdown_connections 2023-11-28T17:09:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.309+0000 7fc00b59e640 1 --2- 172.21.15.116:0/169445565 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbfec06b2a0 0x7fbfec06d760 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.309+0000 7fc00b59e640 1 --2- 172.21.15.116:0/169445565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc00c09e950 0x7fc00c0ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.309+0000 7fc00b59e640 1 --2- 172.21.15.116:0/169445565 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc00c160b80 0x7fc00c09e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.309+0000 7fc00b59e640 1 --2- 172.21.15.116:0/169445565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc00c0b4010 0x7fc00c15b870 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.309+0000 7fc00b59e640 1 -- 172.21.15.116:0/169445565 >> 172.21.15.116:0/169445565 conn(0x7fc00c0d2fc0 msgr2=0x7fc00c15f800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.309+0000 7fc00b59e640 1 -- 172.21.15.116:0/169445565 shutdown_connections 2023-11-28T17:09:35.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.309+0000 7fc00b59e640 1 -- 172.21.15.116:0/169445565 wait complete. 2023-11-28T17:09:35.328 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:09:35.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.543+0000 7f9cda7a5640 1 Processor -- start 2023-11-28T17:09:35.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.543+0000 7f9cda7a5640 1 -- start start 2023-11-28T17:09:35.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.545+0000 7f9cda7a5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd4146540 0x7f9cd4160e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.545+0000 7f9cda7a5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9cd4146a90 0x7f9cd4161350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.545+0000 7f9cda7a5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd4161890 con 0x7f9cd4145ee0 2023-11-28T17:09:35.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.545+0000 7f9cda7a5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd4161dc0 con 0x7f9cd4146a90 2023-11-28T17:09:35.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.545+0000 7f9cda7a5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd40b5110 con 0x7f9cd4146540 2023-11-28T17:09:35.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.545+0000 7f9cd3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9cd4145ee0 0x7f9cd40ac870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42510/0 (socket says 172.21.15.116:42510) 2023-11-28T17:09:35.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.545+0000 7f9cd3fff640 1 -- 172.21.15.116:0/2259719719 learned_addr learned my addr 172.21.15.116:0/2259719719 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd8d42640 1 --2- 172.21.15.116:0/2259719719 >> v2:172.21.15.156:3300/0 conn(0x7f9cd4146a90 0x7f9cd4161350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9ccbfff640 1 --2- 172.21.15.116:0/2259719719 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd4146540 0x7f9cd4160e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 -- 172.21.15.116:0/2259719719 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182568924 0 0) 0x7f9cd4161890 con 0x7f9cd4145ee0 2023-11-28T17:09:35.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 -- 172.21.15.116:0/2259719719 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cc0003520 con 0x7f9cd4145ee0 2023-11-28T17:09:35.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 -- 172.21.15.116:0/2259719719 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1592553003 0 0) 0x7f9cc0003520 con 0x7f9cd4145ee0 2023-11-28T17:09:35.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 -- 172.21.15.116:0/2259719719 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd4146540 msgr2=0x7f9cd4160e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:35.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 --2- 172.21.15.116:0/2259719719 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd4146540 0x7f9cd4160e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 -- 172.21.15.116:0/2259719719 >> v2:172.21.15.156:3300/0 conn(0x7f9cd4146a90 msgr2=0x7f9cd4161350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:35.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 --2- 172.21.15.116:0/2259719719 >> v2:172.21.15.156:3300/0 conn(0x7f9cd4146a90 0x7f9cd4161350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 -- 172.21.15.116:0/2259719719 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd40b2190 con 0x7f9cd4145ee0 2023-11-28T17:09:35.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 -- 172.21.15.116:0/2259719719 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f9cc4001670 con 0x7f9cd4145ee0 2023-11-28T17:09:35.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 -- 172.21.15.116:0/2259719719 >> v1:172.21.15.116:6789/0 conn(0x7f9cd4145ee0 legacy=0x7f9cd40ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 --2- 172.21.15.116:0/2259719719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cc0003970 0x7f9cc0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 --2- 172.21.15.116:0/2259719719 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd4146540 0x7f9cc00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 --2- 172.21.15.116:0/2259719719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cc0004d00 0x7f9cc0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 -- 172.21.15.116:0/2259719719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd4161dc0 con 0x7f9cc0004d00 2023-11-28T17:09:35.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 -- 172.21.15.116:0/2259719719 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd40b5110 con 0x7f9cc0003970 2023-11-28T17:09:35.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.546+0000 7f9cd37fe640 1 -- 172.21.15.116:0/2259719719 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc000b980 con 0x7f9cd4146540 2023-11-28T17:09:35.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.547+0000 7f9cd8d42640 1 --2- 172.21.15.116:0/2259719719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cc0004d00 0x7f9cc0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.547+0000 7f9ccbfff640 1 --2- 172.21.15.116:0/2259719719 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd4146540 0x7f9cc00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.547+0000 7f9cd8d42640 1 -- 172.21.15.116:0/2259719719 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd4146540 msgr2=0x7f9cc00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:35.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.547+0000 7f9cd3fff640 1 --2- 172.21.15.116:0/2259719719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cc0003970 0x7f9cc0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.547+0000 7f9cd8d42640 1 --2- 172.21.15.116:0/2259719719 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd4146540 0x7f9cc00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.547+0000 7f9cd8d42640 1 -- 172.21.15.116:0/2259719719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cc0003970 msgr2=0x7f9cc0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:35.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.547+0000 7f9cd8d42640 1 --2- 172.21.15.116:0/2259719719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cc0003970 0x7f9cc0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.547+0000 7f9cd8d42640 1 -- 172.21.15.116:0/2259719719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc0003520 con 0x7f9cc0004d00 2023-11-28T17:09:35.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.547+0000 7f9cd3fff640 1 --2- 172.21.15.116:0/2259719719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cc0003970 0x7f9cc0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:35.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.548+0000 7f9cd8d42640 1 --2- 172.21.15.116:0/2259719719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cc0004d00 0x7f9cc0005180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009fc0 tx=0x7f9cc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50db03c872400afe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:35.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.549+0000 7f9cd37fe640 1 -- 172.21.15.116:0/2259719719 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc02cc60 con 0x7f9cc0004d00 2023-11-28T17:09:35.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.549+0000 7f9cd37fe640 1 -- 172.21.15.116:0/2259719719 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ccc00f040 con 0x7f9cc0004d00 2023-11-28T17:09:35.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.549+0000 7f9cd37fe640 1 -- 172.21.15.116:0/2259719719 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc034440 con 0x7f9cc0004d00 2023-11-28T17:09:35.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.549+0000 7f9cda7a5640 1 -- 172.21.15.116:0/2259719719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cc0004d00 msgr2=0x7f9cc0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.549+0000 7f9cda7a5640 1 --2- 172.21.15.116:0/2259719719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cc0004d00 0x7f9cc0005180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009fc0 tx=0x7f9cc00057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.549+0000 7f9cd8d42640 1 -- 172.21.15.116:0/2259719719 reap_dead start 2023-11-28T17:09:35.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.549+0000 7f9cda7a5640 1 -- 172.21.15.116:0/2259719719 shutdown_connections 2023-11-28T17:09:35.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.549+0000 7f9cda7a5640 1 -- 172.21.15.116:0/2259719719 >> 172.21.15.116:0/2259719719 conn(0x7f9cd40b0cf0 msgr2=0x7f9cd40b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:35.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.549+0000 7f9cda7a5640 1 -- 172.21.15.116:0/2259719719 shutdown_connections 2023-11-28T17:09:35.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.549+0000 7f9cda7a5640 1 -- 172.21.15.116:0/2259719719 wait complete. 2023-11-28T17:09:35.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.551+0000 7f9cda7a5640 1 Processor -- start 2023-11-28T17:09:35.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.551+0000 7f9cda7a5640 1 -- start start 2023-11-28T17:09:35.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.552+0000 7f9cda7a5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cc0004d00 0x7f9cd413f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.552+0000 7f9cda7a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd41f58f0 0x7f9cd40c3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.552+0000 7f9cda7a5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cd40c44a0 0x7f9cd41f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.552+0000 7f9cda7a5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd40c4950 con 0x7f9cd41f58f0 2023-11-28T17:09:35.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.552+0000 7f9cda7a5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd41f5460 con 0x7f9cd40c44a0 2023-11-28T17:09:35.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.552+0000 7f9cda7a5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd41f4850 con 0x7f9cc0004d00 2023-11-28T17:09:35.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.553+0000 7f9ccbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd41f58f0 0x7f9cd40c3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.553+0000 7f9ccbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd41f58f0 0x7f9cd40c3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32848/0 (socket says 172.21.15.116:32848) 2023-11-28T17:09:35.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.553+0000 7f9ccbfff640 1 -- 172.21.15.116:0/1202041912 learned_addr learned my addr 172.21.15.116:0/1202041912 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:35.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.553+0000 7f9cd3fff640 1 --2- 172.21.15.116:0/1202041912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cc0004d00 0x7f9cd413f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.553+0000 7f9ccbfff640 1 -- 172.21.15.116:0/1202041912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cc0004d00 msgr2=0x7f9cd413f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.553+0000 7f9ccbfff640 1 --2- 172.21.15.116:0/1202041912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cc0004d00 0x7f9cd413f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.553+0000 7f9ccbfff640 1 -- 172.21.15.116:0/1202041912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cd40c44a0 msgr2=0x7f9cd41f3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.553+0000 7f9ccbfff640 1 --2- 172.21.15.116:0/1202041912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cd40c44a0 0x7f9cd41f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.553+0000 7f9ccbfff640 1 -- 172.21.15.116:0/1202041912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd41f4b30 con 0x7f9cd41f58f0 2023-11-28T17:09:35.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.553+0000 7f9ccbfff640 1 --2- 172.21.15.116:0/1202041912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd41f58f0 0x7f9cd40c3f60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc010890 tx=0x7f9cbc010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:35.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.553+0000 7f9cd17fa640 1 -- 172.21.15.116:0/1202041912 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc009610 con 0x7f9cd41f58f0 2023-11-28T17:09:35.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.553+0000 7f9cd17fa640 1 -- 172.21.15.116:0/1202041912 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cbc00eae0 con 0x7f9cd41f58f0 2023-11-28T17:09:35.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.554+0000 7f9cda7a5640 1 -- 172.21.15.116:0/1202041912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd41f4e10 con 0x7f9cd41f58f0 2023-11-28T17:09:35.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.554+0000 7f9cd17fa640 1 -- 172.21.15.116:0/1202041912 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc00ee70 con 0x7f9cd41f58f0 2023-11-28T17:09:35.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.554+0000 7f9cda7a5640 1 -- 172.21.15.116:0/1202041912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd413fad0 con 0x7f9cd41f58f0 2023-11-28T17:09:35.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.555+0000 7f9cda7a5640 1 -- 172.21.15.116:0/1202041912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd4069360 con 0x7f9cd41f58f0 2023-11-28T17:09:35.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.556+0000 7f9cd17fa640 1 -- 172.21.15.116:0/1202041912 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f9cbc01c400 con 0x7f9cd41f58f0 2023-11-28T17:09:35.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.556+0000 7f9cd17fa640 1 --2- 172.21.15.116:0/1202041912 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9cb406b2d0 0x7f9cb406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.557+0000 7f9cd17fa640 1 -- 172.21.15.116:0/1202041912 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cbc08a160 con 0x7f9cd41f58f0 2023-11-28T17:09:35.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.557+0000 7f9cd3fff640 1 --2- 172.21.15.116:0/1202041912 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9cb406b2d0 0x7f9cb406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.557+0000 7f9cd3fff640 1 --2- 172.21.15.116:0/1202041912 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9cb406b2d0 0x7f9cb406d790 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4000c00 tx=0x7f9cc4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:35.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.561+0000 7f9cd17fa640 1 -- 172.21.15.116:0/1202041912 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9cbc054fe0 con 0x7f9cd41f58f0 2023-11-28T17:09:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.743+0000 7f9cda7a5640 1 -- 172.21.15.116:0/1202041912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9cd41f55a0 con 0x7f9cd41f58f0 2023-11-28T17:09:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.743+0000 7f9cd17fa640 1 -- 172.21.15.116:0/1202041912 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9cbc058600 con 0x7f9cd41f58f0 2023-11-28T17:09:35.745 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:09:35.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.746+0000 7f9cda7a5640 1 -- 172.21.15.116:0/1202041912 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9cb406b2d0 msgr2=0x7f9cb406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.746+0000 7f9cda7a5640 1 --2- 172.21.15.116:0/1202041912 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9cb406b2d0 0x7f9cb406d790 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4000c00 tx=0x7f9cc4009040 comp rx=0 tx=0).stop 2023-11-28T17:09:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.746+0000 7f9cda7a5640 1 -- 172.21.15.116:0/1202041912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd41f58f0 msgr2=0x7f9cd40c3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.746+0000 7f9cda7a5640 1 --2- 172.21.15.116:0/1202041912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd41f58f0 0x7f9cd40c3f60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc010890 tx=0x7f9cbc010d50 comp rx=0 tx=0).stop 2023-11-28T17:09:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.746+0000 7f9cda7a5640 1 -- 172.21.15.116:0/1202041912 shutdown_connections 2023-11-28T17:09:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.746+0000 7f9cda7a5640 1 --2- 172.21.15.116:0/1202041912 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9cb406b2d0 0x7f9cb406d790 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.746+0000 7f9cda7a5640 1 --2- 172.21.15.116:0/1202041912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cd40c44a0 0x7f9cd41f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.746+0000 7f9cda7a5640 1 --2- 172.21.15.116:0/1202041912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd41f58f0 0x7f9cd40c3f60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.746+0000 7f9cda7a5640 1 --2- 172.21.15.116:0/1202041912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cc0004d00 0x7f9cd413f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.746+0000 7f9cda7a5640 1 -- 172.21.15.116:0/1202041912 >> 172.21.15.116:0/1202041912 conn(0x7f9cd40b0cf0 msgr2=0x7f9cd40acec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.746+0000 7f9cda7a5640 1 -- 172.21.15.116:0/1202041912 shutdown_connections 2023-11-28T17:09:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.746+0000 7f9cda7a5640 1 -- 172.21.15.116:0/1202041912 wait complete. 2023-11-28T17:09:35.766 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:35.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.967+0000 7f288fda9640 1 Processor -- start 2023-11-28T17:09:35.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.967+0000 7f288fda9640 1 -- start start 2023-11-28T17:09:35.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.968+0000 7f288fda9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f288815f7b0 0x7f2888071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.968+0000 7f288fda9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28880721b0 0x7f288806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.968+0000 7f288fda9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f288815fc10 con 0x7f2888144430 2023-11-28T17:09:35.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.968+0000 7f288fda9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2888144810 con 0x7f288815f7b0 2023-11-28T17:09:35.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.968+0000 7f288fda9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2888070bd0 con 0x7f28880721b0 2023-11-28T17:09:35.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.969+0000 7f288db45640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2888144430 0x7f28880703f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42526/0 (socket says 172.21.15.116:42526) 2023-11-28T17:09:35.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.969+0000 7f288db45640 1 -- 172.21.15.116:0/136146660 learned_addr learned my addr 172.21.15.116:0/136146660 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.969+0000 7f288cb43640 1 -- 172.21.15.116:0/136146660 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394344769 0 0) 0x7f288815fc10 con 0x7f2888144430 2023-11-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.969+0000 7f288e346640 1 --2- 172.21.15.116:0/136146660 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28880721b0 0x7f288806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.969+0000 7f288cb43640 1 -- 172.21.15.116:0/136146660 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2874003520 con 0x7f2888144430 2023-11-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.969+0000 7f288d344640 1 --2- 172.21.15.116:0/136146660 >> v2:172.21.15.156:3300/0 conn(0x7f288815f7b0 0x7f2888071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.969+0000 7f288cb43640 1 -- 172.21.15.116:0/136146660 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1566468514 0 0) 0x7f2874003520 con 0x7f2888144430 2023-11-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.969+0000 7f288cb43640 1 -- 172.21.15.116:0/136146660 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28880721b0 msgr2=0x7f288806c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.969+0000 7f288cb43640 1 --2- 172.21.15.116:0/136146660 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28880721b0 0x7f288806c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288cb43640 1 -- 172.21.15.116:0/136146660 >> v2:172.21.15.156:3300/0 conn(0x7f288815f7b0 msgr2=0x7f2888071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288cb43640 1 --2- 172.21.15.116:0/136146660 >> v2:172.21.15.156:3300/0 conn(0x7f288815f7b0 0x7f2888071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288cb43640 1 -- 172.21.15.116:0/136146660 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2888070eb0 con 0x7f2888144430 2023-11-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288cb43640 1 -- 172.21.15.116:0/136146660 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f2884001670 con 0x7f2888144430 2023-11-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288cb43640 1 -- 172.21.15.116:0/136146660 >> v1:172.21.15.116:6789/0 conn(0x7f2888144430 legacy=0x7f28880703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288cb43640 1 --2- 172.21.15.116:0/136146660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2874003970 0x7f2874003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288cb43640 1 --2- 172.21.15.116:0/136146660 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28880721b0 0x7f28740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288cb43640 1 --2- 172.21.15.116:0/136146660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2874004d00 0x7f2874005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288cb43640 1 -- 172.21.15.116:0/136146660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2888144810 con 0x7f2874003970 2023-11-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288cb43640 1 -- 172.21.15.116:0/136146660 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2888070bd0 con 0x7f2874004d00 2023-11-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288cb43640 1 -- 172.21.15.116:0/136146660 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287400bae0 con 0x7f28880721b0 2023-11-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288db45640 1 --2- 172.21.15.116:0/136146660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2874003970 0x7f2874003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288d344640 1 --2- 172.21.15.116:0/136146660 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28880721b0 0x7f28740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288db45640 1 -- 172.21.15.116:0/136146660 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28880721b0 msgr2=0x7f28740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288db45640 1 --2- 172.21.15.116:0/136146660 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28880721b0 0x7f28740047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288e346640 1 --2- 172.21.15.116:0/136146660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2874004d00 0x7f2874005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288db45640 1 -- 172.21.15.116:0/136146660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2874004d00 msgr2=0x7f2874005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288db45640 1 --2- 172.21.15.116:0/136146660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2874004d00 0x7f2874005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.970+0000 7f288db45640 1 -- 172.21.15.116:0/136146660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2874003520 con 0x7f2874003970 2023-11-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.971+0000 7f288db45640 1 --2- 172.21.15.116:0/136146660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2874003970 0x7f2874003d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f288400c110 tx=0x7f28740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47470003ef895cde server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:35.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.972+0000 7f288cb43640 1 -- 172.21.15.116:0/136146660 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2884005ad0 con 0x7f2874003970 2023-11-28T17:09:35.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.972+0000 7f288cb43640 1 -- 172.21.15.116:0/136146660 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2884005c50 con 0x7f2874003970 2023-11-28T17:09:35.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.972+0000 7f288cb43640 1 -- 172.21.15.116:0/136146660 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288402c830 con 0x7f2874003970 2023-11-28T17:09:35.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.972+0000 7f288fda9640 1 -- 172.21.15.116:0/136146660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2874003970 msgr2=0x7f2874003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.972+0000 7f288fda9640 1 --2- 172.21.15.116:0/136146660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2874003970 0x7f2874003d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f288400c110 tx=0x7f28740057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.972+0000 7f288e346640 1 -- 172.21.15.116:0/136146660 reap_dead start 2023-11-28T17:09:35.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.972+0000 7f288fda9640 1 -- 172.21.15.116:0/136146660 shutdown_connections 2023-11-28T17:09:35.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.972+0000 7f288fda9640 1 -- 172.21.15.116:0/136146660 >> 172.21.15.116:0/136146660 conn(0x7f288806a4c0 msgr2=0x7f288806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:35.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.972+0000 7f288fda9640 1 -- 172.21.15.116:0/136146660 shutdown_connections 2023-11-28T17:09:35.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.972+0000 7f288fda9640 1 -- 172.21.15.116:0/136146660 wait complete. 2023-11-28T17:09:35.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.974+0000 7f288fda9640 1 Processor -- start 2023-11-28T17:09:35.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.974+0000 7f288fda9640 1 -- start start 2023-11-28T17:09:35.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.975+0000 7f288fda9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2874003970 0x7f28880c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.975+0000 7f288fda9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2888133520 0x7f28880a86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.975+0000 7f288fda9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28881efb90 0x7f28881f0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.975+0000 7f288fda9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28880c67c0 con 0x7f28881efb90 2023-11-28T17:09:35.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.975+0000 7f288fda9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28880c6aa0 con 0x7f2874003970 2023-11-28T17:09:35.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.975+0000 7f288fda9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28880c6d80 con 0x7f2888133520 2023-11-28T17:09:35.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.975+0000 7f288e346640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28881efb90 0x7f28881f0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.975+0000 7f288e346640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28881efb90 0x7f28881f0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32864/0 (socket says 172.21.15.116:32864) 2023-11-28T17:09:35.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.975+0000 7f288e346640 1 -- 172.21.15.116:0/2827383706 learned_addr learned my addr 172.21.15.116:0/2827383706 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:35.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.976+0000 7f288e346640 1 -- 172.21.15.116:0/2827383706 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2888133520 msgr2=0x7f28880a86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.976+0000 7f288e346640 1 --2- 172.21.15.116:0/2827383706 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2888133520 0x7f28880a86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.976+0000 7f288e346640 1 -- 172.21.15.116:0/2827383706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2874003970 msgr2=0x7f28880c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:35.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.976+0000 7f288e346640 1 --2- 172.21.15.116:0/2827383706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2874003970 0x7f28880c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:35.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.976+0000 7f288e346640 1 -- 172.21.15.116:0/2827383706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28880c7060 con 0x7f28881efb90 2023-11-28T17:09:35.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.976+0000 7f288e346640 1 --2- 172.21.15.116:0/2827383706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28881efb90 0x7f28881f0010 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f28880cc1d0 tx=0x7f2880009d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:35.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.976+0000 7f287e7fc640 1 -- 172.21.15.116:0/2827383706 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2880007cf0 con 0x7f28881efb90 2023-11-28T17:09:35.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.976+0000 7f287e7fc640 1 -- 172.21.15.116:0/2827383706 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2880007e70 con 0x7f28881efb90 2023-11-28T17:09:35.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.976+0000 7f287e7fc640 1 -- 172.21.15.116:0/2827383706 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2880014620 con 0x7f28881efb90 2023-11-28T17:09:35.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.976+0000 7f288fda9640 1 -- 172.21.15.116:0/2827383706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28880c7340 con 0x7f28881efb90 2023-11-28T17:09:35.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.976+0000 7f288fda9640 1 -- 172.21.15.116:0/2827383706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28881f0550 con 0x7f28881efb90 2023-11-28T17:09:35.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.977+0000 7f288fda9640 1 -- 172.21.15.116:0/2827383706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28880c7530 con 0x7f28881efb90 2023-11-28T17:09:35.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.978+0000 7f287e7fc640 1 -- 172.21.15.116:0/2827383706 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f288001d3b0 con 0x7f28881efb90 2023-11-28T17:09:35.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.979+0000 7f287e7fc640 1 --2- 172.21.15.116:0/2827383706 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f286c06b2a0 0x7f286c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:35.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.979+0000 7f287e7fc640 1 -- 172.21.15.116:0/2827383706 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2880089930 con 0x7f28881efb90 2023-11-28T17:09:35.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.979+0000 7f288db45640 1 --2- 172.21.15.116:0/2827383706 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f286c06b2a0 0x7f286c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:35.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.980+0000 7f288db45640 1 --2- 172.21.15.116:0/2827383706 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f286c06b2a0 0x7f286c06d760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2884034000 tx=0x7f288402e000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:35.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:35.984+0000 7f287e7fc640 1 -- 172.21.15.116:0/2827383706 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2880054790 con 0x7f28881efb90 2023-11-28T17:09:36.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.176+0000 7f288fda9640 1 -- 172.21.15.116:0/2827383706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28880693d0 con 0x7f28881efb90 2023-11-28T17:09:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.177+0000 7f287e7fc640 1 -- 172.21.15.116:0/2827383706 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16521 (secure 0 0 0) 0x7f2880003bc0 con 0x7f28881efb90 2023-11-28T17:09:36.179 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:36.179 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:01.970957+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:36.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.180+0000 7f288fda9640 1 -- 172.21.15.116:0/2827383706 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f286c06b2a0 msgr2=0x7f286c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:36.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.180+0000 7f288fda9640 1 --2- 172.21.15.116:0/2827383706 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f286c06b2a0 0x7f286c06d760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2884034000 tx=0x7f288402e000 comp rx=0 tx=0).stop 2023-11-28T17:09:36.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.180+0000 7f288fda9640 1 -- 172.21.15.116:0/2827383706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28881efb90 msgr2=0x7f28881f0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:36.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.180+0000 7f288fda9640 1 --2- 172.21.15.116:0/2827383706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28881efb90 0x7f28881f0010 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f28880cc1d0 tx=0x7f2880009d50 comp rx=0 tx=0).stop 2023-11-28T17:09:36.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.180+0000 7f288fda9640 1 -- 172.21.15.116:0/2827383706 shutdown_connections 2023-11-28T17:09:36.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.180+0000 7f288fda9640 1 --2- 172.21.15.116:0/2827383706 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f286c06b2a0 0x7f286c06d760 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.180+0000 7f288fda9640 1 --2- 172.21.15.116:0/2827383706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28881efb90 0x7f28881f0010 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.180+0000 7f288fda9640 1 --2- 172.21.15.116:0/2827383706 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2888133520 0x7f28880a86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.180+0000 7f288fda9640 1 --2- 172.21.15.116:0/2827383706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2874003970 0x7f28880c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.180+0000 7f288fda9640 1 -- 172.21.15.116:0/2827383706 >> 172.21.15.116:0/2827383706 conn(0x7f288806a4c0 msgr2=0x7f288815ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:36.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.180+0000 7f288fda9640 1 -- 172.21.15.116:0/2827383706 shutdown_connections 2023-11-28T17:09:36.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.180+0000 7f288fda9640 1 -- 172.21.15.116:0/2827383706 wait complete. 2023-11-28T17:09:36.200 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:09:36.201 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:09:36.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.413+0000 7f8a866d3640 1 Processor -- start 2023-11-28T17:09:36.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.414+0000 7f8a866d3640 1 -- start start 2023-11-28T17:09:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.415+0000 7f8a866d3640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a80146540 0x7f8a80160e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.415+0000 7f8a866d3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8a80146a90 0x7f8a80161350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.415+0000 7f8a866d3640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a80161890 con 0x7f8a80145ee0 2023-11-28T17:09:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.415+0000 7f8a866d3640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a80161dc0 con 0x7f8a80146a90 2023-11-28T17:09:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.415+0000 7f8a866d3640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800b5110 con 0x7f8a80146540 2023-11-28T17:09:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.416+0000 7f8a84c70640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8a80146a90 0x7f8a80161350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:36.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.416+0000 7f8a84c70640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8a80146a90 0x7f8a80161350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32876/0 (socket says 172.21.15.116:32876) 2023-11-28T17:09:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.416+0000 7f8a84c70640 1 -- 172.21.15.116:0/1108454305 learned_addr learned my addr 172.21.15.116:0/1108454305 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.416+0000 7f8a77fff640 1 --2- 172.21.15.116:0/1108454305 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a80146540 0x7f8a80160e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:36.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.416+0000 7f8a84c70640 1 -- 172.21.15.116:0/1108454305 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a80146540 msgr2=0x7f8a80160e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.416+0000 7f8a84c70640 1 --2- 172.21.15.116:0/1108454305 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a80146540 0x7f8a80160e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.416+0000 7f8a84c70640 1 -- 172.21.15.116:0/1108454305 >> v1:172.21.15.156:6789/0 conn(0x7f8a80145ee0 legacy=0x7f8a800ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.416+0000 7f8a84c70640 1 -- 172.21.15.116:0/1108454305 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a800b2190 con 0x7f8a80146a90 2023-11-28T17:09:36.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.417+0000 7f8a84c70640 1 --2- 172.21.15.116:0/1108454305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80146a90 0x7f8a80161350 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009e50 tx=0x7f8a7802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4aa2f49ef67e39c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.418+0000 7f8a7f7fe640 1 -- 172.21.15.116:0/1108454305 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7802ca70 con 0x7f8a80146a90 2023-11-28T17:09:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.418+0000 7f8a7f7fe640 1 -- 172.21.15.116:0/1108454305 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7802cbd0 con 0x7f8a80146a90 2023-11-28T17:09:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.418+0000 7f8a7f7fe640 1 -- 172.21.15.116:0/1108454305 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a780343f0 con 0x7f8a80146a90 2023-11-28T17:09:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.418+0000 7f8a866d3640 1 -- 172.21.15.116:0/1108454305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80146a90 msgr2=0x7f8a80161350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.418+0000 7f8a866d3640 1 --2- 172.21.15.116:0/1108454305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80146a90 0x7f8a80161350 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009e50 tx=0x7f8a7802bed0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.418+0000 7f8a866d3640 1 -- 172.21.15.116:0/1108454305 shutdown_connections 2023-11-28T17:09:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.418+0000 7f8a866d3640 1 --2- 172.21.15.116:0/1108454305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80146a90 0x7f8a80161350 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.418+0000 7f8a866d3640 1 --2- 172.21.15.116:0/1108454305 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a80146540 0x7f8a80160e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.418+0000 7f8a866d3640 1 -- 172.21.15.116:0/1108454305 >> 172.21.15.116:0/1108454305 conn(0x7f8a800b0cf0 msgr2=0x7f8a800b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.418+0000 7f8a866d3640 1 -- 172.21.15.116:0/1108454305 shutdown_connections 2023-11-28T17:09:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.419+0000 7f8a866d3640 1 -- 172.21.15.116:0/1108454305 wait complete. 2023-11-28T17:09:36.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.421+0000 7f8a866d3640 1 Processor -- start 2023-11-28T17:09:36.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.421+0000 7f8a866d3640 1 -- start start 2023-11-28T17:09:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.422+0000 7f8a866d3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a80145ee0 0x7f8a80071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.422+0000 7f8a866d3640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a80146540 0x7f8a801f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.422+0000 7f8a866d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80146a90 0x7f8a801f3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.422+0000 7f8a866d3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a80071ea0 con 0x7f8a80146a90 2023-11-28T17:09:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.422+0000 7f8a866d3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a801f4170 con 0x7f8a80145ee0 2023-11-28T17:09:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.422+0000 7f8a866d3640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a801f4410 con 0x7f8a80146540 2023-11-28T17:09:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.422+0000 7f8a84c70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80146a90 0x7f8a801f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.422+0000 7f8a84c70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80146a90 0x7f8a801f3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32892/0 (socket says 172.21.15.116:32892) 2023-11-28T17:09:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.422+0000 7f8a84c70640 1 -- 172.21.15.116:0/2150019106 learned_addr learned my addr 172.21.15.116:0/2150019106 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.423+0000 7f8a7ffff640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a80145ee0 0x7f8a80071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.423+0000 7f8a84c70640 1 -- 172.21.15.116:0/2150019106 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a80146540 msgr2=0x7f8a801f2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.423+0000 7f8a77fff640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a80146540 0x7f8a801f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.423+0000 7f8a84c70640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a80146540 0x7f8a801f2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.423+0000 7f8a84c70640 1 -- 172.21.15.116:0/2150019106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a80145ee0 msgr2=0x7f8a80071960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.423+0000 7f8a84c70640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a80145ee0 0x7f8a80071960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.423+0000 7f8a84c70640 1 -- 172.21.15.116:0/2150019106 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a78009b30 con 0x7f8a80146a90 2023-11-28T17:09:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.423+0000 7f8a7ffff640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a80145ee0 0x7f8a80071960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.423+0000 7f8a84c70640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80146a90 0x7f8a801f3400 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009e50 tx=0x7f8a78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:36.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.423+0000 7f8a7d7fa640 1 -- 172.21.15.116:0/2150019106 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a78009510 con 0x7f8a80146a90 2023-11-28T17:09:36.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.424+0000 7f8a7d7fa640 1 -- 172.21.15.116:0/2150019106 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7803e450 con 0x7f8a80146a90 2023-11-28T17:09:36.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.424+0000 7f8a7d7fa640 1 -- 172.21.15.116:0/2150019106 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7803e800 con 0x7f8a80146a90 2023-11-28T17:09:36.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.424+0000 7f8a866d3640 1 -- 172.21.15.116:0/2150019106 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a801f46f0 con 0x7f8a80146a90 2023-11-28T17:09:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.424+0000 7f8a866d3640 1 -- 172.21.15.116:0/2150019106 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a801454a0 con 0x7f8a80146a90 2023-11-28T17:09:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.425+0000 7f8a866d3640 1 -- 172.21.15.116:0/2150019106 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a80145720 con 0x7f8a80146a90 2023-11-28T17:09:36.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.426+0000 7f8a7d7fa640 1 -- 172.21.15.116:0/2150019106 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f8a7803e960 con 0x7f8a80146a90 2023-11-28T17:09:36.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.427+0000 7f8a7d7fa640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8a6006b2a0 0x7f8a6006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:36.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.427+0000 7f8a7d7fa640 1 -- 172.21.15.116:0/2150019106 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a780acb00 con 0x7f8a80146a90 2023-11-28T17:09:36.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.427+0000 7f8a7ffff640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8a6006b2a0 0x7f8a6006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:36.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.428+0000 7f8a7ffff640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8a6006b2a0 0x7f8a6006d760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8a800ceb60 tx=0x7f8a70008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:36.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.431+0000 7f8a7d7fa640 1 -- 172.21.15.116:0/2150019106 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a780aee60 con 0x7f8a80146a90 2023-11-28T17:09:36.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.623+0000 7f8a866d3640 1 -- 172.21.15.116:0/2150019106 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8a80069360 con 0x7f8a80146a90 2023-11-28T17:09:36.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.623+0000 7f8a7d7fa640 1 -- 172.21.15.116:0/2150019106 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8a78077980 con 0x7f8a80146a90 2023-11-28T17:09:36.625 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:09:36.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.626+0000 7f8a866d3640 1 -- 172.21.15.116:0/2150019106 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8a6006b2a0 msgr2=0x7f8a6006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.626+0000 7f8a866d3640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8a6006b2a0 0x7f8a6006d760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8a800ceb60 tx=0x7f8a70008040 comp rx=0 tx=0).stop 2023-11-28T17:09:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.626+0000 7f8a866d3640 1 -- 172.21.15.116:0/2150019106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80146a90 msgr2=0x7f8a801f3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.626+0000 7f8a866d3640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80146a90 0x7f8a801f3400 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009e50 tx=0x7f8a78004a40 comp rx=0 tx=0).stop 2023-11-28T17:09:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.626+0000 7f8a866d3640 1 -- 172.21.15.116:0/2150019106 shutdown_connections 2023-11-28T17:09:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.626+0000 7f8a866d3640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8a6006b2a0 0x7f8a6006d760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.626+0000 7f8a866d3640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a80146a90 0x7f8a801f3400 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.626+0000 7f8a866d3640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a80146540 0x7f8a801f2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.626+0000 7f8a866d3640 1 --2- 172.21.15.116:0/2150019106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a80145ee0 0x7f8a80071960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.626+0000 7f8a866d3640 1 -- 172.21.15.116:0/2150019106 >> 172.21.15.116:0/2150019106 conn(0x7f8a800b0cf0 msgr2=0x7f8a800ada70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:36.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.626+0000 7f8a866d3640 1 -- 172.21.15.116:0/2150019106 shutdown_connections 2023-11-28T17:09:36.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.626+0000 7f8a866d3640 1 -- 172.21.15.116:0/2150019106 wait complete. 2023-11-28T17:09:36.646 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:09:36.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.858+0000 7f7c85dc4640 1 Processor -- start 2023-11-28T17:09:36.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.859+0000 7f7c85dc4640 1 -- start start 2023-11-28T17:09:36.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.860+0000 7f7c85dc4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7c8015ed10 0x7f7c800ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:36.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.860+0000 7f7c85dc4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c800cd320 con 0x7f7c8015b0c0 2023-11-28T17:09:36.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.860+0000 7f7c85dc4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c800cc710 con 0x7f7c80072250 2023-11-28T17:09:36.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.860+0000 7f7c85dc4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8015f1a0 con 0x7f7c8015ed10 2023-11-28T17:09:36.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.860+0000 7f7c7f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7c8015b0c0 0x7f7c80071b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42542/0 (socket says 172.21.15.116:42542) 2023-11-28T17:09:36.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.860+0000 7f7c7f7fe640 1 -- 172.21.15.116:0/3652109005 learned_addr learned my addr 172.21.15.116:0/3652109005 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:36.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.861+0000 7f7c7ffff640 1 --2- 172.21.15.116:0/3652109005 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7c8015ed10 0x7f7c800ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:36.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.861+0000 7f7c7e7fc640 1 -- 172.21.15.116:0/3652109005 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161983746 0 0) 0x7f7c800cd320 con 0x7f7c8015b0c0 2023-11-28T17:09:36.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.861+0000 7f7c7e7fc640 1 -- 172.21.15.116:0/3652109005 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c68003520 con 0x7f7c8015b0c0 2023-11-28T17:09:36.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.861+0000 7f7c7e7fc640 1 -- 172.21.15.116:0/3652109005 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 848355351 0 0) 0x7f7c68003520 con 0x7f7c8015b0c0 2023-11-28T17:09:36.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.861+0000 7f7c7e7fc640 1 -- 172.21.15.116:0/3652109005 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7c8015ed10 msgr2=0x7f7c800ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:36.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.861+0000 7f7c7e7fc640 1 --2- 172.21.15.116:0/3652109005 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7c8015ed10 0x7f7c800ce940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.861+0000 7f7c7e7fc640 1 -- 172.21.15.116:0/3652109005 >> v1:172.21.15.156:6789/0 conn(0x7f7c80072250 legacy=0x7f7c8015b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:36.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.861+0000 7f7c7e7fc640 1 -- 172.21.15.116:0/3652109005 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c800cc9f0 con 0x7f7c8015b0c0 2023-11-28T17:09:36.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.861+0000 7f7c7e7fc640 1 -- 172.21.15.116:0/3652109005 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f7c70001670 con 0x7f7c8015b0c0 2023-11-28T17:09:36.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.861+0000 7f7c7e7fc640 1 -- 172.21.15.116:0/3652109005 >> v1:172.21.15.116:6789/0 conn(0x7f7c8015b0c0 legacy=0x7f7c80071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:36.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.861+0000 7f7c7e7fc640 1 --2- 172.21.15.116:0/3652109005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c680039a0 0x7f7c68003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:36.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.861+0000 7f7c7e7fc640 1 --2- 172.21.15.116:0/3652109005 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7c8015ed10 0x7f7c680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:36.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.862+0000 7f7c7e7fc640 1 --2- 172.21.15.116:0/3652109005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c68004d00 0x7f7c68005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:36.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.862+0000 7f7c7e7fc640 1 -- 172.21.15.116:0/3652109005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8015f1a0 con 0x7f7c68004d00 2023-11-28T17:09:36.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.862+0000 7f7c7e7fc640 1 -- 172.21.15.116:0/3652109005 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6800bab0 con 0x7f7c680039a0 2023-11-28T17:09:36.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.862+0000 7f7c7e7fc640 1 -- 172.21.15.116:0/3652109005 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6800bc70 con 0x7f7c8015ed10 2023-11-28T17:09:36.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.862+0000 7f7c7ffff640 1 --2- 172.21.15.116:0/3652109005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c68004d00 0x7f7c68005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:36.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.862+0000 7f7c7f7fe640 1 --2- 172.21.15.116:0/3652109005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c680039a0 0x7f7c68003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:36.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.862+0000 7f7c7ffff640 1 -- 172.21.15.116:0/3652109005 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7c8015ed10 msgr2=0x7f7c680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:36.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.862+0000 7f7c7ffff640 1 --2- 172.21.15.116:0/3652109005 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7c8015ed10 0x7f7c680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.862+0000 7f7c7ffff640 1 -- 172.21.15.116:0/3652109005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c680039a0 msgr2=0x7f7c68003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:36.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.862+0000 7f7c7ffff640 1 --2- 172.21.15.116:0/3652109005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c680039a0 0x7f7c68003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.862+0000 7f7c7ffff640 1 -- 172.21.15.116:0/3652109005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c68003520 con 0x7f7c68004d00 2023-11-28T17:09:36.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.863+0000 7f7c7ffff640 1 --2- 172.21.15.116:0/3652109005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c68004d00 0x7f7c68005180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7c78009fc0 tx=0x7f7c680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea4b88fa610714e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:36.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.863+0000 7f7c7e7fc640 1 -- 172.21.15.116:0/3652109005 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7802cc60 con 0x7f7c68004d00 2023-11-28T17:09:36.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.863+0000 7f7c7e7fc640 1 -- 172.21.15.116:0/3652109005 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7800f040 con 0x7f7c68004d00 2023-11-28T17:09:36.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.863+0000 7f7c7e7fc640 1 -- 172.21.15.116:0/3652109005 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c78034440 con 0x7f7c68004d00 2023-11-28T17:09:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.864+0000 7f7c85dc4640 1 -- 172.21.15.116:0/3652109005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c68004d00 msgr2=0x7f7c68005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.864+0000 7f7c85dc4640 1 --2- 172.21.15.116:0/3652109005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c68004d00 0x7f7c68005180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7c78009fc0 tx=0x7f7c680057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.864+0000 7f7c7ffff640 1 -- 172.21.15.116:0/3652109005 reap_dead start 2023-11-28T17:09:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.864+0000 7f7c85dc4640 1 -- 172.21.15.116:0/3652109005 shutdown_connections 2023-11-28T17:09:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.864+0000 7f7c85dc4640 1 -- 172.21.15.116:0/3652109005 >> 172.21.15.116:0/3652109005 conn(0x7f7c800b0cf0 msgr2=0x7f7c800b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.864+0000 7f7c85dc4640 1 -- 172.21.15.116:0/3652109005 shutdown_connections 2023-11-28T17:09:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.864+0000 7f7c85dc4640 1 -- 172.21.15.116:0/3652109005 wait complete. 2023-11-28T17:09:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.866+0000 7f7c85dc4640 1 Processor -- start 2023-11-28T17:09:36.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.866+0000 7f7c85dc4640 1 -- start start 2023-11-28T17:09:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.867+0000 7f7c85dc4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c68004d00 0x7f7c8014ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.867+0000 7f7c85dc4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7c8014a9c0 0x7f7c8014d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.867+0000 7f7c85dc4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c8014ae00 0x7f7c800707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.867+0000 7f7c85dc4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8014de60 con 0x7f7c68004d00 2023-11-28T17:09:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.867+0000 7f7c85dc4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8014e140 con 0x7f7c8014ae00 2023-11-28T17:09:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.867+0000 7f7c85dc4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8014e420 con 0x7f7c8014a9c0 2023-11-28T17:09:36.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.867+0000 7f7c7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c68004d00 0x7f7c8014ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.867+0000 7f7c7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c68004d00 0x7f7c8014ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32920/0 (socket says 172.21.15.116:32920) 2023-11-28T17:09:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.867+0000 7f7c7f7fe640 1 -- 172.21.15.116:0/2368134752 learned_addr learned my addr 172.21.15.116:0/2368134752 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.867+0000 7f7c7f7fe640 1 -- 172.21.15.116:0/2368134752 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7c8014a9c0 msgr2=0x7f7c8014d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.867+0000 7f7c7f7fe640 1 --2- 172.21.15.116:0/2368134752 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7c8014a9c0 0x7f7c8014d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.867+0000 7f7c7f7fe640 1 -- 172.21.15.116:0/2368134752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c8014ae00 msgr2=0x7f7c800707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.867+0000 7f7c7f7fe640 1 --2- 172.21.15.116:0/2368134752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c8014ae00 0x7f7c800707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.868+0000 7f7c7f7fe640 1 -- 172.21.15.116:0/2368134752 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8015b620 con 0x7f7c68004d00 2023-11-28T17:09:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.868+0000 7f7c7f7fe640 1 --2- 172.21.15.116:0/2368134752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c68004d00 0x7f7c8014ef00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000e8c0 tx=0x7f7c7000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.868+0000 7f7c67fff640 1 -- 172.21.15.116:0/2368134752 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7000c850 con 0x7f7c68004d00 2023-11-28T17:09:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.868+0000 7f7c67fff640 1 -- 172.21.15.116:0/2368134752 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7000c9b0 con 0x7f7c68004d00 2023-11-28T17:09:36.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.868+0000 7f7c67fff640 1 -- 172.21.15.116:0/2368134752 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7000cd60 con 0x7f7c68004d00 2023-11-28T17:09:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.868+0000 7f7c85dc4640 1 -- 172.21.15.116:0/2368134752 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c8014e700 con 0x7f7c68004d00 2023-11-28T17:09:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.868+0000 7f7c85dc4640 1 -- 172.21.15.116:0/2368134752 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8014f440 con 0x7f7c68004d00 2023-11-28T17:09:36.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.869+0000 7f7c85dc4640 1 -- 172.21.15.116:0/2368134752 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c80069360 con 0x7f7c68004d00 2023-11-28T17:09:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.870+0000 7f7c67fff640 1 -- 172.21.15.116:0/2368134752 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f7c7001d460 con 0x7f7c68004d00 2023-11-28T17:09:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.871+0000 7f7c67fff640 1 --2- 172.21.15.116:0/2368134752 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7c6c06b2d0 0x7f7c6c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.871+0000 7f7c67fff640 1 -- 172.21.15.116:0/2368134752 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c70089b40 con 0x7f7c68004d00 2023-11-28T17:09:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.871+0000 7f7c7effd640 1 --2- 172.21.15.116:0/2368134752 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7c6c06b2d0 0x7f7c6c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:36.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.872+0000 7f7c7effd640 1 --2- 172.21.15.116:0/2368134752 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7c6c06b2d0 0x7f7c6c06d790 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7c801532d0 tx=0x7f7c74007990 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:36.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:36.875+0000 7f7c67fff640 1 -- 172.21.15.116:0/2368134752 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c70054a40 con 0x7f7c68004d00 2023-11-28T17:09:37.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.052+0000 7f7c85dc4640 1 -- 172.21.15.116:0/2368134752 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7c800b1990 con 0x7f7c68004d00 2023-11-28T17:09:37.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.053+0000 7f7c67fff640 1 -- 172.21.15.116:0/2368134752 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7c70001610 con 0x7f7c68004d00 2023-11-28T17:09:37.055 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:09:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.056+0000 7f7c85dc4640 1 -- 172.21.15.116:0/2368134752 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7c6c06b2d0 msgr2=0x7f7c6c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:37.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.056+0000 7f7c85dc4640 1 --2- 172.21.15.116:0/2368134752 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7c6c06b2d0 0x7f7c6c06d790 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7c801532d0 tx=0x7f7c74007990 comp rx=0 tx=0).stop 2023-11-28T17:09:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.056+0000 7f7c85dc4640 1 -- 172.21.15.116:0/2368134752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c68004d00 msgr2=0x7f7c8014ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.056+0000 7f7c85dc4640 1 --2- 172.21.15.116:0/2368134752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c68004d00 0x7f7c8014ef00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000e8c0 tx=0x7f7c7000ed80 comp rx=0 tx=0).stop 2023-11-28T17:09:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.056+0000 7f7c85dc4640 1 -- 172.21.15.116:0/2368134752 shutdown_connections 2023-11-28T17:09:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.056+0000 7f7c85dc4640 1 --2- 172.21.15.116:0/2368134752 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7c6c06b2d0 0x7f7c6c06d790 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.056+0000 7f7c85dc4640 1 --2- 172.21.15.116:0/2368134752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c8014ae00 0x7f7c800707a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.056+0000 7f7c85dc4640 1 --2- 172.21.15.116:0/2368134752 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7c8014a9c0 0x7f7c8014d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.056+0000 7f7c85dc4640 1 --2- 172.21.15.116:0/2368134752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c68004d00 0x7f7c8014ef00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.056+0000 7f7c85dc4640 1 -- 172.21.15.116:0/2368134752 >> 172.21.15.116:0/2368134752 conn(0x7f7c800b0cf0 msgr2=0x7f7c8015e160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.056+0000 7f7c85dc4640 1 -- 172.21.15.116:0/2368134752 shutdown_connections 2023-11-28T17:09:37.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.056+0000 7f7c85dc4640 1 -- 172.21.15.116:0/2368134752 wait complete. 2023-11-28T17:09:37.074 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:09:37.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.284+0000 7f49ebcb4640 1 Processor -- start 2023-11-28T17:09:37.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.284+0000 7f49ebcb4640 1 -- start start 2023-11-28T17:09:37.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.286+0000 7f49ebcb4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f49e415b0c0 0x7f49e415ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:37.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.286+0000 7f49ebcb4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f49e40ce940 0x7f49e40cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:37.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.286+0000 7f49ebcb4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e40cd320 con 0x7f49e4071b40 2023-11-28T17:09:37.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.286+0000 7f49ebcb4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e40ced80 con 0x7f49e40ce940 2023-11-28T17:09:37.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.286+0000 7f49ebcb4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e40cbee0 con 0x7f49e415b0c0 2023-11-28T17:09:37.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.286+0000 7f49ea251640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f49e4071b40 0x7f49e415b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42550/0 (socket says 172.21.15.116:42550) 2023-11-28T17:09:37.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.286+0000 7f49ea251640 1 -- 172.21.15.116:0/2471893498 learned_addr learned my addr 172.21.15.116:0/2471893498 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:37.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.286+0000 7f49e924f640 1 --2- 172.21.15.116:0/2471893498 >> v2:172.21.15.116:3300/0 conn(0x7f49e40ce940 0x7f49e40cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:37.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.287+0000 7f49e9a50640 1 --2- 172.21.15.116:0/2471893498 >> v2:172.21.15.156:3300/0 conn(0x7f49e415b0c0 0x7f49e415ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:37.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.287+0000 7f49e924f640 1 -- 172.21.15.116:0/2471893498 >> v2:172.21.15.156:3300/0 conn(0x7f49e415b0c0 msgr2=0x7f49e415ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.287+0000 7f49e924f640 1 --2- 172.21.15.116:0/2471893498 >> v2:172.21.15.156:3300/0 conn(0x7f49e415b0c0 0x7f49e415ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.287+0000 7f49e8a4e640 1 -- 172.21.15.116:0/2471893498 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806007003 0 0) 0x7f49e40cd320 con 0x7f49e4071b40 2023-11-28T17:09:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.287+0000 7f49e924f640 1 -- 172.21.15.116:0/2471893498 >> v1:172.21.15.116:6789/0 conn(0x7f49e4071b40 legacy=0x7f49e415b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.287+0000 7f49e924f640 1 -- 172.21.15.116:0/2471893498 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e40cc710 con 0x7f49e40ce940 2023-11-28T17:09:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.287+0000 7f49e9a50640 1 --2- 172.21.15.116:0/2471893498 >> v2:172.21.15.156:3300/0 conn(0x7f49e415b0c0 0x7f49e415ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.288+0000 7f49e924f640 1 --2- 172.21.15.116:0/2471893498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49e40ce940 0x7f49e40cb9a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f49e406b470 tx=0x7f49d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c64f3047d89be712 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.288+0000 7f49e8a4e640 1 -- 172.21.15.116:0/2471893498 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d402e070 con 0x7f49e40ce940 2023-11-28T17:09:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.288+0000 7f49e8a4e640 1 -- 172.21.15.116:0/2471893498 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d40315a0 con 0x7f49e40ce940 2023-11-28T17:09:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.288+0000 7f49e8a4e640 1 -- 172.21.15.116:0/2471893498 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4031930 con 0x7f49e40ce940 2023-11-28T17:09:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.289+0000 7f49ebcb4640 1 -- 172.21.15.116:0/2471893498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49e40ce940 msgr2=0x7f49e40cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.289+0000 7f49ebcb4640 1 --2- 172.21.15.116:0/2471893498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49e40ce940 0x7f49e40cb9a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f49e406b470 tx=0x7f49d402b840 comp rx=0 tx=0).stop 2023-11-28T17:09:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.289+0000 7f49ebcb4640 1 -- 172.21.15.116:0/2471893498 shutdown_connections 2023-11-28T17:09:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.289+0000 7f49ebcb4640 1 --2- 172.21.15.116:0/2471893498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49e40ce940 0x7f49e40cb9a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.289+0000 7f49ebcb4640 1 --2- 172.21.15.116:0/2471893498 >> v2:172.21.15.156:3300/0 conn(0x7f49e415b0c0 0x7f49e415ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.289+0000 7f49ebcb4640 1 -- 172.21.15.116:0/2471893498 >> 172.21.15.116:0/2471893498 conn(0x7f49e40b0cf0 msgr2=0x7f49e40b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.289+0000 7f49ebcb4640 1 -- 172.21.15.116:0/2471893498 shutdown_connections 2023-11-28T17:09:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.289+0000 7f49ebcb4640 1 -- 172.21.15.116:0/2471893498 wait complete. 2023-11-28T17:09:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.291+0000 7f49ebcb4640 1 Processor -- start 2023-11-28T17:09:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.291+0000 7f49ebcb4640 1 -- start start 2023-11-28T17:09:37.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49ebcb4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49e4071b40 0x7f49e4155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49ebcb4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49e40ce940 0x7f49e41f1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49ebcb4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f49e415b0c0 0x7f49e40aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49ebcb4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e41f1bb0 con 0x7f49e40ce940 2023-11-28T17:09:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49ebcb4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e41f50b0 con 0x7f49e4071b40 2023-11-28T17:09:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49ebcb4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e41f4530 con 0x7f49e415b0c0 2023-11-28T17:09:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49e924f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49e40ce940 0x7f49e41f1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49e924f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49e40ce940 0x7f49e41f1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32930/0 (socket says 172.21.15.116:32930) 2023-11-28T17:09:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49e924f640 1 -- 172.21.15.116:0/1548940788 learned_addr learned my addr 172.21.15.116:0/1548940788 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49e9a50640 1 --2- 172.21.15.116:0/1548940788 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49e4071b40 0x7f49e4155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49e924f640 1 -- 172.21.15.116:0/1548940788 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f49e415b0c0 msgr2=0x7f49e40aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:37.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49e924f640 1 --2- 172.21.15.116:0/1548940788 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f49e415b0c0 0x7f49e40aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49e924f640 1 -- 172.21.15.116:0/1548940788 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49e4071b40 msgr2=0x7f49e4155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:37.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49e924f640 1 --2- 172.21.15.116:0/1548940788 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49e4071b40 0x7f49e4155130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.292+0000 7f49e924f640 1 -- 172.21.15.116:0/1548940788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d4009400 con 0x7f49e40ce940 2023-11-28T17:09:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.293+0000 7f49e924f640 1 --2- 172.21.15.116:0/1548940788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49e40ce940 0x7f49e41f1670 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f49d4009890 tx=0x7f49d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.293+0000 7f49da7fc640 1 -- 172.21.15.116:0/1548940788 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4032900 con 0x7f49e40ce940 2023-11-28T17:09:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.293+0000 7f49da7fc640 1 -- 172.21.15.116:0/1548940788 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d4032a60 con 0x7f49e40ce940 2023-11-28T17:09:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.293+0000 7f49da7fc640 1 -- 172.21.15.116:0/1548940788 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4032e10 con 0x7f49e40ce940 2023-11-28T17:09:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.293+0000 7f49ebcb4640 1 -- 172.21.15.116:0/1548940788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e41f4810 con 0x7f49e40ce940 2023-11-28T17:09:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.293+0000 7f49ebcb4640 1 -- 172.21.15.116:0/1548940788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e41f4dd0 con 0x7f49e40ce940 2023-11-28T17:09:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.294+0000 7f49ebcb4640 1 -- 172.21.15.116:0/1548940788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e4069360 con 0x7f49e40ce940 2023-11-28T17:09:37.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.295+0000 7f49da7fc640 1 -- 172.21.15.116:0/1548940788 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f49d403b3f0 con 0x7f49e40ce940 2023-11-28T17:09:37.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.296+0000 7f49da7fc640 1 --2- 172.21.15.116:0/1548940788 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f49c806b2a0 0x7f49c806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:37.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.296+0000 7f49da7fc640 1 -- 172.21.15.116:0/1548940788 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49d40abd60 con 0x7f49e40ce940 2023-11-28T17:09:37.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.296+0000 7f49e9a50640 1 --2- 172.21.15.116:0/1548940788 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f49c806b2a0 0x7f49c806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:37.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.297+0000 7f49e9a50640 1 --2- 172.21.15.116:0/1548940788 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f49c806b2a0 0x7f49c806d760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f49e4152260 tx=0x7f49e0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:37.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.301+0000 7f49da7fc640 1 -- 172.21.15.116:0/1548940788 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f49d4076bc0 con 0x7f49e40ce940 2023-11-28T17:09:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.491+0000 7f49ebcb4640 1 -- 172.21.15.116:0/1548940788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f49e4152360 con 0x7f49e40ce940 2023-11-28T17:09:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.492+0000 7f49da7fc640 1 -- 172.21.15.116:0/1548940788 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f49d407a1e0 con 0x7f49e40ce940 2023-11-28T17:09:37.493 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:09:37.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.494+0000 7f49ebcb4640 1 -- 172.21.15.116:0/1548940788 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f49c806b2a0 msgr2=0x7f49c806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:37.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.494+0000 7f49ebcb4640 1 --2- 172.21.15.116:0/1548940788 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f49c806b2a0 0x7f49c806d760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f49e4152260 tx=0x7f49e0008040 comp rx=0 tx=0).stop 2023-11-28T17:09:37.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.494+0000 7f49ebcb4640 1 -- 172.21.15.116:0/1548940788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49e40ce940 msgr2=0x7f49e41f1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:37.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.494+0000 7f49ebcb4640 1 --2- 172.21.15.116:0/1548940788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49e40ce940 0x7f49e41f1670 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f49d4009890 tx=0x7f49d40046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.494+0000 7f49ebcb4640 1 -- 172.21.15.116:0/1548940788 shutdown_connections 2023-11-28T17:09:37.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.494+0000 7f49ebcb4640 1 --2- 172.21.15.116:0/1548940788 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f49c806b2a0 0x7f49c806d760 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.494+0000 7f49ebcb4640 1 --2- 172.21.15.116:0/1548940788 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f49e415b0c0 0x7f49e40aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.494+0000 7f49ebcb4640 1 --2- 172.21.15.116:0/1548940788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49e40ce940 0x7f49e41f1670 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.494+0000 7f49ebcb4640 1 --2- 172.21.15.116:0/1548940788 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49e4071b40 0x7f49e4155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.494+0000 7f49ebcb4640 1 -- 172.21.15.116:0/1548940788 >> 172.21.15.116:0/1548940788 conn(0x7f49e40b0cf0 msgr2=0x7f49e4070820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.495+0000 7f49ebcb4640 1 -- 172.21.15.116:0/1548940788 shutdown_connections 2023-11-28T17:09:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.495+0000 7f49ebcb4640 1 -- 172.21.15.116:0/1548940788 wait complete. 2023-11-28T17:09:37.514 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:37.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.717+0000 7f5b00a94640 1 Processor -- start 2023-11-28T17:09:37.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.717+0000 7f5b00a94640 1 -- start start 2023-11-28T17:09:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.718+0000 7f5b00a94640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5afc144430 0x7f5afc15f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.718+0000 7f5b00a94640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5afc0703f0 0x7f5afc071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.718+0000 7f5b00a94640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc070bd0 con 0x7f5afc0721b0 2023-11-28T17:09:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.718+0000 7f5b00a94640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5afc144810 con 0x7f5afc0703f0 2023-11-28T17:09:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.718+0000 7f5b00a94640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc070eb0 con 0x7f5afc144430 2023-11-28T17:09:37.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.719+0000 7f5af9d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5afc0703f0 0x7f5afc071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:37.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.719+0000 7f5af9d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5afc0703f0 0x7f5afc071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32934/0 (socket says 172.21.15.116:32934) 2023-11-28T17:09:37.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.719+0000 7f5af9d9b640 1 -- 172.21.15.116:0/3848719154 learned_addr learned my addr 172.21.15.116:0/3848719154 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:37.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.719+0000 7f5afa59c640 1 --2- 172.21.15.116:0/3848719154 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5afc144430 0x7f5afc15f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:37.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.719+0000 7f5af9d9b640 1 -- 172.21.15.116:0/3848719154 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5afc144430 msgr2=0x7f5afc15f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:37.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.719+0000 7f5af9d9b640 1 --2- 172.21.15.116:0/3848719154 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5afc144430 0x7f5afc15f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.720+0000 7f5af9d9b640 1 -- 172.21.15.116:0/3848719154 >> v1:172.21.15.156:6789/0 conn(0x7f5afc0721b0 legacy=0x7f5afc06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:37.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.720+0000 7f5af9d9b640 1 -- 172.21.15.116:0/3848719154 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc071190 con 0x7f5afc0703f0 2023-11-28T17:09:37.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.721+0000 7f5af9d9b640 1 --2- 172.21.15.116:0/3848719154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc0703f0 0x7f5afc071c70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009870 tx=0x7f5af002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d09b96bbeeb5be server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:37.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.721+0000 7f5af959a640 1 -- 172.21.15.116:0/3848719154 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af0030840 con 0x7f5afc0703f0 2023-11-28T17:09:37.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.721+0000 7f5af959a640 1 -- 172.21.15.116:0/3848719154 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af00309a0 con 0x7f5afc0703f0 2023-11-28T17:09:37.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.721+0000 7f5af959a640 1 -- 172.21.15.116:0/3848719154 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af0030d30 con 0x7f5afc0703f0 2023-11-28T17:09:37.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.721+0000 7f5b00a94640 1 -- 172.21.15.116:0/3848719154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc0703f0 msgr2=0x7f5afc071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:37.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.721+0000 7f5b00a94640 1 --2- 172.21.15.116:0/3848719154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc0703f0 0x7f5afc071c70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009870 tx=0x7f5af002b980 comp rx=0 tx=0).stop 2023-11-28T17:09:37.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.722+0000 7f5b00a94640 1 -- 172.21.15.116:0/3848719154 shutdown_connections 2023-11-28T17:09:37.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.722+0000 7f5b00a94640 1 --2- 172.21.15.116:0/3848719154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc0703f0 0x7f5afc071c70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.722+0000 7f5b00a94640 1 --2- 172.21.15.116:0/3848719154 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5afc144430 0x7f5afc15f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.722+0000 7f5b00a94640 1 -- 172.21.15.116:0/3848719154 >> 172.21.15.116:0/3848719154 conn(0x7f5afc06a4c0 msgr2=0x7f5afc06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:37.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.722+0000 7f5b00a94640 1 -- 172.21.15.116:0/3848719154 shutdown_connections 2023-11-28T17:09:37.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.722+0000 7f5b00a94640 1 -- 172.21.15.116:0/3848719154 wait complete. 2023-11-28T17:09:37.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.723+0000 7f5b00a94640 1 Processor -- start 2023-11-28T17:09:37.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.724+0000 7f5b00a94640 1 -- start start 2023-11-28T17:09:37.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5b00a94640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5afc0703f0 0x7f5afc1e6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:37.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5b00a94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc0721b0 0x7f5afc13f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:37.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5b00a94640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5afc144430 0x7f5afc0ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:37.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5b00a94640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc13ed10 con 0x7f5afc0721b0 2023-11-28T17:09:37.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5b00a94640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc13e420 con 0x7f5afc144430 2023-11-28T17:09:37.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5b00a94640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc13e6c0 con 0x7f5afc0703f0 2023-11-28T17:09:37.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5af9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc0721b0 0x7f5afc13f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:37.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5af9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc0721b0 0x7f5afc13f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32948/0 (socket says 172.21.15.116:32948) 2023-11-28T17:09:37.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5af9d9b640 1 -- 172.21.15.116:0/278205946 learned_addr learned my addr 172.21.15.116:0/278205946 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:37.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5afa59c640 1 --2- 172.21.15.116:0/278205946 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5afc0703f0 0x7f5afc1e6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:37.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5af9d9b640 1 -- 172.21.15.116:0/278205946 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5afc0703f0 msgr2=0x7f5afc1e6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:37.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5afad9d640 1 --2- 172.21.15.116:0/278205946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5afc144430 0x7f5afc0ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:37.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5af9d9b640 1 --2- 172.21.15.116:0/278205946 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5afc0703f0 0x7f5afc1e6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5af9d9b640 1 -- 172.21.15.116:0/278205946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5afc144430 msgr2=0x7f5afc0ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:37.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5af9d9b640 1 --2- 172.21.15.116:0/278205946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5afc144430 0x7f5afc0ca050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.725+0000 7f5af9d9b640 1 -- 172.21.15.116:0/278205946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af00094d0 con 0x7f5afc0721b0 2023-11-28T17:09:37.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.726+0000 7f5af9d9b640 1 --2- 172.21.15.116:0/278205946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc0721b0 0x7f5afc13f1a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5af0000c00 tx=0x7f5af0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:37.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.726+0000 7f5aeaffd640 1 -- 172.21.15.116:0/278205946 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af0039c90 con 0x7f5afc0721b0 2023-11-28T17:09:37.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.726+0000 7f5aeaffd640 1 -- 172.21.15.116:0/278205946 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af0039df0 con 0x7f5afc0721b0 2023-11-28T17:09:37.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.726+0000 7f5aeaffd640 1 -- 172.21.15.116:0/278205946 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af0043640 con 0x7f5afc0721b0 2023-11-28T17:09:37.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.726+0000 7f5b00a94640 1 -- 172.21.15.116:0/278205946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc13e9a0 con 0x7f5afc0721b0 2023-11-28T17:09:37.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.726+0000 7f5b00a94640 1 -- 172.21.15.116:0/278205946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc0caaf0 con 0x7f5afc0721b0 2023-11-28T17:09:37.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.727+0000 7f5b00a94640 1 -- 172.21.15.116:0/278205946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5afc0693d0 con 0x7f5afc0721b0 2023-11-28T17:09:37.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.728+0000 7f5aeaffd640 1 -- 172.21.15.116:0/278205946 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f5af00437a0 con 0x7f5afc0721b0 2023-11-28T17:09:37.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.729+0000 7f5aeaffd640 1 --2- 172.21.15.116:0/278205946 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5adc06b2a0 0x7f5adc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:37.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.729+0000 7f5aeaffd640 1 -- 172.21.15.116:0/278205946 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5af003e070 con 0x7f5afc0721b0 2023-11-28T17:09:37.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.729+0000 7f5afa59c640 1 --2- 172.21.15.116:0/278205946 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5adc06b2a0 0x7f5adc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:37.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.730+0000 7f5afa59c640 1 --2- 172.21.15.116:0/278205946 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5adc06b2a0 0x7f5adc06d760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5aec006fd0 tx=0x7f5aec008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:37.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.733+0000 7f5aeaffd640 1 -- 172.21.15.116:0/278205946 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5af0077b30 con 0x7f5afc0721b0 2023-11-28T17:09:37.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.924+0000 7f5b00a94640 1 -- 172.21.15.116:0/278205946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5afc06d100 con 0x7f5afc0721b0 2023-11-28T17:09:37.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.926+0000 7f5aeaffd640 1 -- 172.21.15.116:0/278205946 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16521 (secure 0 0 0) 0x7f5af007b150 con 0x7f5afc0721b0 2023-11-28T17:09:37.928 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:37.928 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:01.970957+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.929+0000 7f5b00a94640 1 -- 172.21.15.116:0/278205946 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5adc06b2a0 msgr2=0x7f5adc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.929+0000 7f5b00a94640 1 --2- 172.21.15.116:0/278205946 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5adc06b2a0 0x7f5adc06d760 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5aec006fd0 tx=0x7f5aec008040 comp rx=0 tx=0).stop 2023-11-28T17:09:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.929+0000 7f5b00a94640 1 -- 172.21.15.116:0/278205946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc0721b0 msgr2=0x7f5afc13f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.929+0000 7f5b00a94640 1 --2- 172.21.15.116:0/278205946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc0721b0 0x7f5afc13f1a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5af0000c00 tx=0x7f5af0004730 comp rx=0 tx=0).stop 2023-11-28T17:09:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.929+0000 7f5b00a94640 1 -- 172.21.15.116:0/278205946 shutdown_connections 2023-11-28T17:09:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.929+0000 7f5b00a94640 1 --2- 172.21.15.116:0/278205946 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5adc06b2a0 0x7f5adc06d760 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.929+0000 7f5b00a94640 1 --2- 172.21.15.116:0/278205946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5afc144430 0x7f5afc0ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.929+0000 7f5b00a94640 1 --2- 172.21.15.116:0/278205946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5afc0721b0 0x7f5afc13f1a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.929+0000 7f5b00a94640 1 --2- 172.21.15.116:0/278205946 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5afc0703f0 0x7f5afc1e6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.929+0000 7f5b00a94640 1 -- 172.21.15.116:0/278205946 >> 172.21.15.116:0/278205946 conn(0x7f5afc06a4c0 msgr2=0x7f5afc143410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.930+0000 7f5b00a94640 1 -- 172.21.15.116:0/278205946 shutdown_connections 2023-11-28T17:09:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:37.930+0000 7f5b00a94640 1 -- 172.21.15.116:0/278205946 wait complete. 2023-11-28T17:09:37.950 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:09:37.950 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:09:38.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.162+0000 7f6f05887640 1 Processor -- start 2023-11-28T17:09:38.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.162+0000 7f6f05887640 1 -- start start 2023-11-28T17:09:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.164+0000 7f6f05887640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6f0015b0c0 0x7f6f0015ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.164+0000 7f6f05887640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f00072250 0x7f6f000cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:38.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.164+0000 7f6f05887640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f000cd320 con 0x7f6f000ce940 2023-11-28T17:09:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.164+0000 7f6f05887640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f000ced80 con 0x7f6f0015b0c0 2023-11-28T17:09:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.164+0000 7f6f05887640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f000cbee0 con 0x7f6f00072250 2023-11-28T17:09:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.165+0000 7f6eff7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f00072250 0x7f6f000cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.165+0000 7f6efeffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6f0015b0c0 0x7f6f0015ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.165+0000 7f6eff7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f00072250 0x7f6f000cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:47156/0 (socket says 172.21.15.116:47156) 2023-11-28T17:09:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.165+0000 7f6eff7fe640 1 -- 172.21.15.116:0/2386292444 learned_addr learned my addr 172.21.15.116:0/2386292444 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.165+0000 7f6efeffd640 1 -- 172.21.15.116:0/2386292444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f00072250 msgr2=0x7f6f000cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.165+0000 7f6efeffd640 1 --2- 172.21.15.116:0/2386292444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f00072250 0x7f6f000cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.165+0000 7f6efeffd640 1 -- 172.21.15.116:0/2386292444 >> v1:172.21.15.156:6789/0 conn(0x7f6f000ce940 legacy=0x7f6f00071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.165+0000 7f6efeffd640 1 -- 172.21.15.116:0/2386292444 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f000cc710 con 0x7f6f0015b0c0 2023-11-28T17:09:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.167+0000 7f6efeffd640 1 --2- 172.21.15.116:0/2386292444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f0015b0c0 0x7f6f0015ed10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6ef00097a0 tx=0x7f6ef002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6396648b3eeddf93 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.167+0000 7f6efdffb640 1 -- 172.21.15.116:0/2386292444 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef002e070 con 0x7f6f0015b0c0 2023-11-28T17:09:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.167+0000 7f6efdffb640 1 -- 172.21.15.116:0/2386292444 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef00315a0 con 0x7f6f0015b0c0 2023-11-28T17:09:38.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.167+0000 7f6efdffb640 1 -- 172.21.15.116:0/2386292444 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0031930 con 0x7f6f0015b0c0 2023-11-28T17:09:38.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.168+0000 7f6f05887640 1 -- 172.21.15.116:0/2386292444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f0015b0c0 msgr2=0x7f6f0015ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:38.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.168+0000 7f6f05887640 1 --2- 172.21.15.116:0/2386292444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f0015b0c0 0x7f6f0015ed10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6ef00097a0 tx=0x7f6ef002b840 comp rx=0 tx=0).stop 2023-11-28T17:09:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.168+0000 7f6f05887640 1 -- 172.21.15.116:0/2386292444 shutdown_connections 2023-11-28T17:09:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.168+0000 7f6f05887640 1 --2- 172.21.15.116:0/2386292444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f00072250 0x7f6f000cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.168+0000 7f6f05887640 1 --2- 172.21.15.116:0/2386292444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f0015b0c0 0x7f6f0015ed10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.168+0000 7f6f05887640 1 -- 172.21.15.116:0/2386292444 >> 172.21.15.116:0/2386292444 conn(0x7f6f000b0cf0 msgr2=0x7f6f000b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.168+0000 7f6f05887640 1 -- 172.21.15.116:0/2386292444 shutdown_connections 2023-11-28T17:09:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.168+0000 7f6f05887640 1 -- 172.21.15.116:0/2386292444 wait complete. 2023-11-28T17:09:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.170+0000 7f6f05887640 1 Processor -- start 2023-11-28T17:09:38.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.170+0000 7f6f05887640 1 -- start start 2023-11-28T17:09:38.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6f05887640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f00072250 0x7f6f000c4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:38.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6f05887640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f000ce940 0x7f6f001f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:38.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6f05887640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0015b0c0 0x7f6f00155c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:38.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6f05887640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f00158ba0 con 0x7f6f0015b0c0 2023-11-28T17:09:38.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6f05887640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f001f3510 con 0x7f6f000ce940 2023-11-28T17:09:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6f05887640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f00156180 con 0x7f6f00072250 2023-11-28T17:09:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6eff7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0015b0c0 0x7f6f00155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6eff7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0015b0c0 0x7f6f00155c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59768/0 (socket says 172.21.15.116:59768) 2023-11-28T17:09:38.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6eff7fe640 1 -- 172.21.15.116:0/1001932886 learned_addr learned my addr 172.21.15.116:0/1001932886 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6efeffd640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f00072250 0x7f6f000c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6efe7fc640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f000ce940 0x7f6f001f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6eff7fe640 1 -- 172.21.15.116:0/1001932886 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f00072250 msgr2=0x7f6f000c4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6eff7fe640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f00072250 0x7f6f000c4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6eff7fe640 1 -- 172.21.15.116:0/1001932886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f000ce940 msgr2=0x7f6f001f2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6eff7fe640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f000ce940 0x7f6f001f2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6eff7fe640 1 -- 172.21.15.116:0/1001932886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef0009400 con 0x7f6f0015b0c0 2023-11-28T17:09:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6efeffd640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f00072250 0x7f6f000c4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.171+0000 7f6efe7fc640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f000ce940 0x7f6f001f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.172+0000 7f6eff7fe640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0015b0c0 0x7f6f00155c40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800efc0 tx=0x7f6ef800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.172+0000 7f6ee7fff640 1 -- 172.21.15.116:0/1001932886 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef800cd20 con 0x7f6f0015b0c0 2023-11-28T17:09:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.172+0000 7f6ee7fff640 1 -- 172.21.15.116:0/1001932886 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef800f040 con 0x7f6f0015b0c0 2023-11-28T17:09:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.172+0000 7f6ee7fff640 1 -- 172.21.15.116:0/1001932886 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8003bd0 con 0x7f6f0015b0c0 2023-11-28T17:09:38.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.172+0000 7f6f05887640 1 -- 172.21.15.116:0/1001932886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f001f3770 con 0x7f6f0015b0c0 2023-11-28T17:09:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.172+0000 7f6f05887640 1 -- 172.21.15.116:0/1001932886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f001f3d30 con 0x7f6f0015b0c0 2023-11-28T17:09:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.173+0000 7f6f05887640 1 -- 172.21.15.116:0/1001932886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f00069360 con 0x7f6f0015b0c0 2023-11-28T17:09:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.174+0000 7f6ee7fff640 1 -- 172.21.15.116:0/1001932886 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6ef8003d30 con 0x7f6f0015b0c0 2023-11-28T17:09:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.175+0000 7f6ee7fff640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6eec06b2a0 0x7f6eec06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.175+0000 7f6ee7fff640 1 -- 172.21.15.116:0/1001932886 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ef808a240 con 0x7f6f0015b0c0 2023-11-28T17:09:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.175+0000 7f6efeffd640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6eec06b2a0 0x7f6eec06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.176+0000 7f6efeffd640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6eec06b2a0 0x7f6eec06d760 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0036000 tx=0x7f6ef003d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.180+0000 7f6ee7fff640 1 -- 172.21.15.116:0/1001932886 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ef80550a0 con 0x7f6f0015b0c0 2023-11-28T17:09:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.370+0000 7f6f05887640 1 -- 172.21.15.116:0/1001932886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6f000b1990 con 0x7f6f0015b0c0 2023-11-28T17:09:38.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.371+0000 7f6ee7fff640 1 -- 172.21.15.116:0/1001932886 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6ef80586c0 con 0x7f6f0015b0c0 2023-11-28T17:09:38.372 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:09:38.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.373+0000 7f6f05887640 1 -- 172.21.15.116:0/1001932886 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6eec06b2a0 msgr2=0x7f6eec06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:38.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.373+0000 7f6f05887640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6eec06b2a0 0x7f6eec06d760 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0036000 tx=0x7f6ef003d040 comp rx=0 tx=0).stop 2023-11-28T17:09:38.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.373+0000 7f6f05887640 1 -- 172.21.15.116:0/1001932886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0015b0c0 msgr2=0x7f6f00155c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:38.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.373+0000 7f6f05887640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0015b0c0 0x7f6f00155c40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800efc0 tx=0x7f6ef800c490 comp rx=0 tx=0).stop 2023-11-28T17:09:38.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.373+0000 7f6f05887640 1 -- 172.21.15.116:0/1001932886 shutdown_connections 2023-11-28T17:09:38.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.373+0000 7f6f05887640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6eec06b2a0 0x7f6eec06d760 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.373+0000 7f6f05887640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0015b0c0 0x7f6f00155c40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.373+0000 7f6f05887640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f000ce940 0x7f6f001f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.373+0000 7f6f05887640 1 --2- 172.21.15.116:0/1001932886 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f00072250 0x7f6f000c4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.373+0000 7f6f05887640 1 -- 172.21.15.116:0/1001932886 >> 172.21.15.116:0/1001932886 conn(0x7f6f000b0cf0 msgr2=0x7f6f000cb1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:38.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.373+0000 7f6f05887640 1 -- 172.21.15.116:0/1001932886 shutdown_connections 2023-11-28T17:09:38.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.373+0000 7f6f05887640 1 -- 172.21.15.116:0/1001932886 wait complete. 2023-11-28T17:09:38.393 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:09:38.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.607+0000 7fa88094c640 1 Processor -- start 2023-11-28T17:09:38.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.607+0000 7fa88094c640 1 -- start start 2023-11-28T17:09:38.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.608+0000 7fa88094c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa87c15b0c0 0x7fa87c15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:38.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.608+0000 7fa88094c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa87c0ce940 0x7fa87c0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:38.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.608+0000 7fa88094c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa87c071b40 0x7fa87c071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:38.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.608+0000 7fa88094c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa87c072510 con 0x7fa87c071b40 2023-11-28T17:09:38.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.608+0000 7fa88094c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa87c0cd320 con 0x7fa87c0ce940 2023-11-28T17:09:38.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.608+0000 7fa88094c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c0ced80 con 0x7fa87c15b0c0 2023-11-28T17:09:38.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.609+0000 7fa87ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa87c071b40 0x7fa87c071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:38.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.609+0000 7fa87ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa87c071b40 0x7fa87c071fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59778/0 (socket says 172.21.15.116:59778) 2023-11-28T17:09:38.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.609+0000 7fa87ad9d640 1 -- 172.21.15.116:0/1711920248 learned_addr learned my addr 172.21.15.116:0/1711920248 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:38.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.609+0000 7fa87a59c640 1 --2- 172.21.15.116:0/1711920248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa87c15b0c0 0x7fa87c15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:38.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.609+0000 7fa873fff640 1 --2- 172.21.15.116:0/1711920248 >> v2:172.21.15.156:3300/0 conn(0x7fa87c0ce940 0x7fa87c0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:38.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.609+0000 7fa87ad9d640 1 -- 172.21.15.116:0/1711920248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa87c15b0c0 msgr2=0x7fa87c15ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:38.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.609+0000 7fa87ad9d640 1 --2- 172.21.15.116:0/1711920248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa87c15b0c0 0x7fa87c15ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.609+0000 7fa87ad9d640 1 -- 172.21.15.116:0/1711920248 >> v2:172.21.15.156:3300/0 conn(0x7fa87c0ce940 msgr2=0x7fa87c0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:38.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.609+0000 7fa87ad9d640 1 --2- 172.21.15.116:0/1711920248 >> v2:172.21.15.156:3300/0 conn(0x7fa87c0ce940 0x7fa87c0cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.609+0000 7fa87ad9d640 1 -- 172.21.15.116:0/1711920248 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c0cbee0 con 0x7fa87c071b40 2023-11-28T17:09:38.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.611+0000 7fa87ad9d640 1 --2- 172.21.15.116:0/1711920248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c071b40 0x7fa87c071fd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa87c0b1db0 tx=0x7fa87402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f325988a05f0816f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:38.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.611+0000 7fa879d9b640 1 -- 172.21.15.116:0/1711920248 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87402c9e0 con 0x7fa87c071b40 2023-11-28T17:09:38.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.611+0000 7fa879d9b640 1 -- 172.21.15.116:0/1711920248 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa87402cb40 con 0x7fa87c071b40 2023-11-28T17:09:38.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.611+0000 7fa879d9b640 1 -- 172.21.15.116:0/1711920248 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8740343f0 con 0x7fa87c071b40 2023-11-28T17:09:38.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.611+0000 7fa88094c640 1 -- 172.21.15.116:0/1711920248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c071b40 msgr2=0x7fa87c071fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:38.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.611+0000 7fa88094c640 1 --2- 172.21.15.116:0/1711920248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c071b40 0x7fa87c071fd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa87c0b1db0 tx=0x7fa87402be40 comp rx=0 tx=0).stop 2023-11-28T17:09:38.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.612+0000 7fa88094c640 1 -- 172.21.15.116:0/1711920248 shutdown_connections 2023-11-28T17:09:38.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.612+0000 7fa88094c640 1 --2- 172.21.15.116:0/1711920248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c071b40 0x7fa87c071fd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.612+0000 7fa88094c640 1 --2- 172.21.15.116:0/1711920248 >> v2:172.21.15.156:3300/0 conn(0x7fa87c0ce940 0x7fa87c0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.612+0000 7fa88094c640 1 --2- 172.21.15.116:0/1711920248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa87c15b0c0 0x7fa87c15ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.612+0000 7fa88094c640 1 -- 172.21.15.116:0/1711920248 >> 172.21.15.116:0/1711920248 conn(0x7fa87c0b0cf0 msgr2=0x7fa87c0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:38.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.612+0000 7fa88094c640 1 -- 172.21.15.116:0/1711920248 shutdown_connections 2023-11-28T17:09:38.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.612+0000 7fa88094c640 1 -- 172.21.15.116:0/1711920248 wait complete. 2023-11-28T17:09:38.614 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.613+0000 7fa88094c640 1 Processor -- start 2023-11-28T17:09:38.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.614+0000 7fa88094c640 1 -- start start 2023-11-28T17:09:38.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.614+0000 7fa88094c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa87c071b40 0x7fa87c1e6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:38.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.614+0000 7fa88094c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa87c0ce940 0x7fa87c1f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:38.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.614+0000 7fa88094c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c15b0c0 0x7fa87c0ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:38.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.614+0000 7fa88094c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c0ac870 con 0x7fa87c15b0c0 2023-11-28T17:09:38.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.614+0000 7fa88094c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c15a820 con 0x7fa87c071b40 2023-11-28T17:09:38.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.614+0000 7fa88094c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c0acac0 con 0x7fa87c0ce940 2023-11-28T17:09:38.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.615+0000 7fa87ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c15b0c0 0x7fa87c0ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:38.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.615+0000 7fa87ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c15b0c0 0x7fa87c0ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59794/0 (socket says 172.21.15.116:59794) 2023-11-28T17:09:38.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.615+0000 7fa87ad9d640 1 -- 172.21.15.116:0/2208921553 learned_addr learned my addr 172.21.15.116:0/2208921553 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:38.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.615+0000 7fa87a59c640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa87c071b40 0x7fa87c1e6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:38.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.615+0000 7fa873fff640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa87c0ce940 0x7fa87c1f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:38.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.615+0000 7fa87ad9d640 1 -- 172.21.15.116:0/2208921553 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa87c0ce940 msgr2=0x7fa87c1f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:38.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.615+0000 7fa87ad9d640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa87c0ce940 0x7fa87c1f4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.615+0000 7fa87ad9d640 1 -- 172.21.15.116:0/2208921553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa87c071b40 msgr2=0x7fa87c1e6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:38.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.615+0000 7fa87ad9d640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa87c071b40 0x7fa87c1e6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.615+0000 7fa87ad9d640 1 -- 172.21.15.116:0/2208921553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa874009b30 con 0x7fa87c15b0c0 2023-11-28T17:09:38.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.615+0000 7fa873fff640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa87c0ce940 0x7fa87c1f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:38.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.616+0000 7fa87ad9d640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c15b0c0 0x7fa87c0ad880 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa874009f40 tx=0x7fa874004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:38.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.616+0000 7fa872ffd640 1 -- 172.21.15.116:0/2208921553 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa874009510 con 0x7fa87c15b0c0 2023-11-28T17:09:38.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.616+0000 7fa872ffd640 1 -- 172.21.15.116:0/2208921553 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa87403e450 con 0x7fa87c15b0c0 2023-11-28T17:09:38.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.616+0000 7fa88094c640 1 -- 172.21.15.116:0/2208921553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87c0acda0 con 0x7fa87c15b0c0 2023-11-28T17:09:38.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.616+0000 7fa872ffd640 1 -- 172.21.15.116:0/2208921553 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87403e7e0 con 0x7fa87c15b0c0 2023-11-28T17:09:38.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.616+0000 7fa88094c640 1 -- 172.21.15.116:0/2208921553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87c0ad360 con 0x7fa87c15b0c0 2023-11-28T17:09:38.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.617+0000 7fa88094c640 1 -- 172.21.15.116:0/2208921553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87c069360 con 0x7fa87c15b0c0 2023-11-28T17:09:38.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.618+0000 7fa872ffd640 1 -- 172.21.15.116:0/2208921553 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa87403e940 con 0x7fa87c15b0c0 2023-11-28T17:09:38.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.619+0000 7fa872ffd640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa85c06b2a0 0x7fa85c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:38.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.619+0000 7fa872ffd640 1 -- 172.21.15.116:0/2208921553 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8740aca80 con 0x7fa87c15b0c0 2023-11-28T17:09:38.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.619+0000 7fa87a59c640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa85c06b2a0 0x7fa85c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:38.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.620+0000 7fa87a59c640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa85c06b2a0 0x7fa85c06d760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa86c006fd0 tx=0x7fa86c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:38.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.623+0000 7fa872ffd640 1 -- 172.21.15.116:0/2208921553 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa874077850 con 0x7fa87c15b0c0 2023-11-28T17:09:38.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.809+0000 7fa88094c640 1 -- 172.21.15.116:0/2208921553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa87c0b1990 con 0x7fa87c15b0c0 2023-11-28T17:09:38.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.810+0000 7fa872ffd640 1 -- 172.21.15.116:0/2208921553 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa874043020 con 0x7fa87c15b0c0 2023-11-28T17:09:38.812 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:09:38.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.813+0000 7fa88094c640 1 -- 172.21.15.116:0/2208921553 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa85c06b2a0 msgr2=0x7fa85c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.813+0000 7fa88094c640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa85c06b2a0 0x7fa85c06d760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa86c006fd0 tx=0x7fa86c008040 comp rx=0 tx=0).stop 2023-11-28T17:09:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.813+0000 7fa88094c640 1 -- 172.21.15.116:0/2208921553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c15b0c0 msgr2=0x7fa87c0ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.813+0000 7fa88094c640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c15b0c0 0x7fa87c0ad880 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa874009f40 tx=0x7fa874004a40 comp rx=0 tx=0).stop 2023-11-28T17:09:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.813+0000 7fa88094c640 1 -- 172.21.15.116:0/2208921553 shutdown_connections 2023-11-28T17:09:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.813+0000 7fa88094c640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa85c06b2a0 0x7fa85c06d760 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.813+0000 7fa88094c640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c15b0c0 0x7fa87c0ad880 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.813+0000 7fa88094c640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa87c0ce940 0x7fa87c1f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.813+0000 7fa88094c640 1 --2- 172.21.15.116:0/2208921553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa87c071b40 0x7fa87c1e6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:38.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.813+0000 7fa88094c640 1 -- 172.21.15.116:0/2208921553 >> 172.21.15.116:0/2208921553 conn(0x7fa87c0b0cf0 msgr2=0x7fa87c15b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:38.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.813+0000 7fa88094c640 1 -- 172.21.15.116:0/2208921553 shutdown_connections 2023-11-28T17:09:38.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:38.813+0000 7fa88094c640 1 -- 172.21.15.116:0/2208921553 wait complete. 2023-11-28T17:09:38.833 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:09:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.042+0000 7f39655ba640 1 Processor -- start 2023-11-28T17:09:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.042+0000 7f39655ba640 1 -- start start 2023-11-28T17:09:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.044+0000 7f39655ba640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3960152050 0x7f396015bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.044+0000 7f39655ba640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f39600abd30 0x7f39600cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.044+0000 7f39655ba640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39600b2b50 con 0x7f39600b2fe0 2023-11-28T17:09:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.044+0000 7f39655ba640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f396015c520 con 0x7f3960152050 2023-11-28T17:09:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.044+0000 7f39655ba640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39600ac2d0 con 0x7f39600abd30 2023-11-28T17:09:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.044+0000 7f39577fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f39600b2fe0 0x7f39600ad930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41246/0 (socket says 172.21.15.116:41246) 2023-11-28T17:09:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.044+0000 7f39577fe640 1 -- 172.21.15.116:0/785500013 learned_addr learned my addr 172.21.15.116:0/785500013 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.044+0000 7f395ffff640 1 --2- 172.21.15.116:0/785500013 >> v2:172.21.15.156:3300/0 conn(0x7f3960152050 0x7f396015bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.044+0000 7f395f7fe640 1 -- 172.21.15.116:0/785500013 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686981933 0 0) 0x7f39600b2b50 con 0x7f39600b2fe0 2023-11-28T17:09:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.044+0000 7f395f7fe640 1 -- 172.21.15.116:0/785500013 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394c003520 con 0x7f39600b2fe0 2023-11-28T17:09:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.044+0000 7f3964db9640 1 --2- 172.21.15.116:0/785500013 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f39600abd30 0x7f39600cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395f7fe640 1 -- 172.21.15.116:0/785500013 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2795553301 0 0) 0x7f394c003520 con 0x7f39600b2fe0 2023-11-28T17:09:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395f7fe640 1 -- 172.21.15.116:0/785500013 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f39600abd30 msgr2=0x7f39600cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395f7fe640 1 --2- 172.21.15.116:0/785500013 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f39600abd30 0x7f39600cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395f7fe640 1 -- 172.21.15.116:0/785500013 >> v2:172.21.15.156:3300/0 conn(0x7f3960152050 msgr2=0x7f396015bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395f7fe640 1 --2- 172.21.15.116:0/785500013 >> v2:172.21.15.156:3300/0 conn(0x7f3960152050 0x7f396015bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395f7fe640 1 -- 172.21.15.116:0/785500013 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39600b2220 con 0x7f39600b2fe0 2023-11-28T17:09:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395f7fe640 1 -- 172.21.15.116:0/785500013 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f39480019f0 con 0x7f39600b2fe0 2023-11-28T17:09:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395f7fe640 1 -- 172.21.15.116:0/785500013 >> v1:172.21.15.116:6789/0 conn(0x7f39600b2fe0 legacy=0x7f39600ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395f7fe640 1 --2- 172.21.15.116:0/785500013 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f39600abd30 0x7f394c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395f7fe640 1 --2- 172.21.15.116:0/785500013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f394c003ff0 0x7f394c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395f7fe640 1 --2- 172.21.15.116:0/785500013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f394c004d00 0x7f394c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395f7fe640 1 -- 172.21.15.116:0/785500013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396015c520 con 0x7f394c003ff0 2023-11-28T17:09:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395f7fe640 1 -- 172.21.15.116:0/785500013 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39600ac2d0 con 0x7f394c004d00 2023-11-28T17:09:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395f7fe640 1 -- 172.21.15.116:0/785500013 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394c00bae0 con 0x7f39600abd30 2023-11-28T17:09:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f395ffff640 1 --2- 172.21.15.116:0/785500013 >> v2:172.21.15.156:3300/0 conn(0x7f3960152050 0x7f396015bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:09:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.045+0000 7f39577fe640 1 --2- 172.21.15.116:0/785500013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f394c003ff0 0x7f394c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.046+0000 7f39577fe640 1 -- 172.21.15.116:0/785500013 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f39600abd30 msgr2=0x7f394c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.046+0000 7f39577fe640 1 --2- 172.21.15.116:0/785500013 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f39600abd30 0x7f394c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.046+0000 7f39577fe640 1 -- 172.21.15.116:0/785500013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f394c004d00 msgr2=0x7f394c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.046+0000 7f39577fe640 1 --2- 172.21.15.116:0/785500013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f394c004d00 0x7f394c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.046+0000 7f39577fe640 1 -- 172.21.15.116:0/785500013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394c003520 con 0x7f394c003ff0 2023-11-28T17:09:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.047+0000 7f39577fe640 1 --2- 172.21.15.116:0/785500013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f394c003ff0 0x7f394c0047c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f394800a790 tx=0x7f394c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b4a1431cbef3b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.047+0000 7f395f7fe640 1 -- 172.21.15.116:0/785500013 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3948034630 con 0x7f394c003ff0 2023-11-28T17:09:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.047+0000 7f395f7fe640 1 -- 172.21.15.116:0/785500013 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f394802f070 con 0x7f394c003ff0 2023-11-28T17:09:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.047+0000 7f395f7fe640 1 -- 172.21.15.116:0/785500013 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3948034a70 con 0x7f394c003ff0 2023-11-28T17:09:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.047+0000 7f39655ba640 1 -- 172.21.15.116:0/785500013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f394c003ff0 msgr2=0x7f394c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.047+0000 7f39655ba640 1 --2- 172.21.15.116:0/785500013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f394c003ff0 0x7f394c0047c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f394800a790 tx=0x7f394c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.047+0000 7f3964db9640 1 -- 172.21.15.116:0/785500013 reap_dead start 2023-11-28T17:09:39.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.048+0000 7f39655ba640 1 -- 172.21.15.116:0/785500013 shutdown_connections 2023-11-28T17:09:39.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.048+0000 7f39655ba640 1 -- 172.21.15.116:0/785500013 >> 172.21.15.116:0/785500013 conn(0x7f39600b0950 msgr2=0x7f39600b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:39.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.048+0000 7f39655ba640 1 -- 172.21.15.116:0/785500013 shutdown_connections 2023-11-28T17:09:39.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.048+0000 7f39655ba640 1 -- 172.21.15.116:0/785500013 wait complete. 2023-11-28T17:09:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.049+0000 7f39655ba640 1 Processor -- start 2023-11-28T17:09:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.050+0000 7f39655ba640 1 -- start start 2023-11-28T17:09:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f39655ba640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f394c003ff0 0x7f396009a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f39655ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39601569c0 0x7f3960141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f39655ba640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3960157f60 0x7f39601583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f39655ba640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3960156ec0 con 0x7f39601569c0 2023-11-28T17:09:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f39655ba640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39601571a0 con 0x7f3960157f60 2023-11-28T17:09:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f39655ba640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3960157480 con 0x7f394c003ff0 2023-11-28T17:09:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f39577fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39601569c0 0x7f3960141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f39577fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39601569c0 0x7f3960141400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59822/0 (socket says 172.21.15.116:59822) 2023-11-28T17:09:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f39577fe640 1 -- 172.21.15.116:0/1254556955 learned_addr learned my addr 172.21.15.116:0/1254556955 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f395ffff640 1 --2- 172.21.15.116:0/1254556955 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f394c003ff0 0x7f396009a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f3964db9640 1 --2- 172.21.15.116:0/1254556955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3960157f60 0x7f39601583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f39577fe640 1 -- 172.21.15.116:0/1254556955 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f394c003ff0 msgr2=0x7f396009a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f39577fe640 1 --2- 172.21.15.116:0/1254556955 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f394c003ff0 0x7f396009a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f39577fe640 1 -- 172.21.15.116:0/1254556955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3960157f60 msgr2=0x7f39601583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.051+0000 7f39577fe640 1 --2- 172.21.15.116:0/1254556955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3960157f60 0x7f39601583f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.052+0000 7f39577fe640 1 -- 172.21.15.116:0/1254556955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3960157760 con 0x7f39601569c0 2023-11-28T17:09:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.052+0000 7f39577fe640 1 --2- 172.21.15.116:0/1254556955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39601569c0 0x7f3960141400 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3948001350 tx=0x7f3948001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.052+0000 7f395d7fa640 1 -- 172.21.15.116:0/1254556955 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3948034980 con 0x7f39601569c0 2023-11-28T17:09:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.052+0000 7f395d7fa640 1 -- 172.21.15.116:0/1254556955 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3948034ae0 con 0x7f39601569c0 2023-11-28T17:09:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.052+0000 7f395d7fa640 1 -- 172.21.15.116:0/1254556955 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3948034e90 con 0x7f39601569c0 2023-11-28T17:09:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.052+0000 7f39655ba640 1 -- 172.21.15.116:0/1254556955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3960157a40 con 0x7f39601569c0 2023-11-28T17:09:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.052+0000 7f39655ba640 1 -- 172.21.15.116:0/1254556955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3960141940 con 0x7f39601569c0 2023-11-28T17:09:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.053+0000 7f39655ba640 1 -- 172.21.15.116:0/1254556955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396009d0b0 con 0x7f39601569c0 2023-11-28T17:09:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.055+0000 7f395d7fa640 1 -- 172.21.15.116:0/1254556955 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3948031030 con 0x7f39601569c0 2023-11-28T17:09:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.056+0000 7f395d7fa640 1 --2- 172.21.15.116:0/1254556955 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f394006b2d0 0x7f394006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.056+0000 7f395d7fa640 1 -- 172.21.15.116:0/1254556955 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39480ac2e0 con 0x7f39601569c0 2023-11-28T17:09:39.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.056+0000 7f395ffff640 1 --2- 172.21.15.116:0/1254556955 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f394006b2d0 0x7f394006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.057+0000 7f395ffff640 1 --2- 172.21.15.116:0/1254556955 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f394006b2d0 0x7f394006d790 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3950006a90 tx=0x7f3950008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:39.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.060+0000 7f395d7fa640 1 -- 172.21.15.116:0/1254556955 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3948077140 con 0x7f39601569c0 2023-11-28T17:09:39.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.250+0000 7f39655ba640 1 -- 172.21.15.116:0/1254556955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f396009d0b0 con 0x7f39601569c0 2023-11-28T17:09:39.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.251+0000 7f395d7fa640 1 -- 172.21.15.116:0/1254556955 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f394807a760 con 0x7f39601569c0 2023-11-28T17:09:39.252 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:09:39.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.253+0000 7f39655ba640 1 -- 172.21.15.116:0/1254556955 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f394006b2d0 msgr2=0x7f394006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.253+0000 7f39655ba640 1 --2- 172.21.15.116:0/1254556955 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f394006b2d0 0x7f394006d790 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3950006a90 tx=0x7f3950008040 comp rx=0 tx=0).stop 2023-11-28T17:09:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.253+0000 7f39655ba640 1 -- 172.21.15.116:0/1254556955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39601569c0 msgr2=0x7f3960141400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.253+0000 7f39655ba640 1 --2- 172.21.15.116:0/1254556955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39601569c0 0x7f3960141400 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3948001350 tx=0x7f3948001cb0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.253+0000 7f39655ba640 1 -- 172.21.15.116:0/1254556955 shutdown_connections 2023-11-28T17:09:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.253+0000 7f39655ba640 1 --2- 172.21.15.116:0/1254556955 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f394006b2d0 0x7f394006d790 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.253+0000 7f39655ba640 1 --2- 172.21.15.116:0/1254556955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3960157f60 0x7f39601583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.253+0000 7f39655ba640 1 --2- 172.21.15.116:0/1254556955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39601569c0 0x7f3960141400 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.253+0000 7f39655ba640 1 --2- 172.21.15.116:0/1254556955 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f394c003ff0 0x7f396009a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.253+0000 7f39655ba640 1 -- 172.21.15.116:0/1254556955 >> 172.21.15.116:0/1254556955 conn(0x7f39600b0950 msgr2=0x7f396015df80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.253+0000 7f39655ba640 1 -- 172.21.15.116:0/1254556955 shutdown_connections 2023-11-28T17:09:39.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.253+0000 7f39655ba640 1 -- 172.21.15.116:0/1254556955 wait complete. 2023-11-28T17:09:39.282 INFO:tasks.cephfs_test_runner:Starting test: test_mount_conn_close (tasks.cephfs.test_sessionmap.TestSessionMap) That when a client unmounts, the thread count on the MDS goes back 2023-11-28T17:09:39.282 INFO:tasks.cephfs_test_runner:test_mount_conn_close (tasks.cephfs.test_sessionmap.TestSessionMap) 2023-11-28T17:09:39.283 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:39.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.490+0000 7f5a45543640 1 Processor -- start 2023-11-28T17:09:39.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.491+0000 7f5a45543640 1 -- start start 2023-11-28T17:09:39.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.492+0000 7f5a45543640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a40071ab0 0x7f5a40071f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.492+0000 7f5a45543640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a40072450 con 0x7f5a4015cb80 2023-11-28T17:09:39.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.492+0000 7f5a45543640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a40070490 con 0x7f5a400ccb40 2023-11-28T17:09:39.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.492+0000 7f5a45543640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400b3f60 con 0x7f5a40071ab0 2023-11-28T17:09:39.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.492+0000 7f5a3ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5a4015cb80 0x7f5a400b2610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41248/0 (socket says 172.21.15.116:41248) 2023-11-28T17:09:39.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.492+0000 7f5a3ffff640 1 -- 172.21.15.116:0/615042021 learned_addr learned my addr 172.21.15.116:0/615042021 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:39.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a37fff640 1 --2- 172.21.15.116:0/615042021 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a40071ab0 0x7f5a40071f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 -- 172.21.15.116:0/615042021 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843064942 0 0) 0x7f5a40072450 con 0x7f5a4015cb80 2023-11-28T17:09:39.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 -- 172.21.15.116:0/615042021 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a2c003440 con 0x7f5a4015cb80 2023-11-28T17:09:39.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 -- 172.21.15.116:0/615042021 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 755074488 0 0) 0x7f5a2c003440 con 0x7f5a4015cb80 2023-11-28T17:09:39.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 -- 172.21.15.116:0/615042021 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a40071ab0 msgr2=0x7f5a40071f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:39.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 --2- 172.21.15.116:0/615042021 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a40071ab0 0x7f5a40071f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 -- 172.21.15.116:0/615042021 >> v1:172.21.15.156:6789/0 conn(0x7f5a400ccb40 legacy=0x7f5a400ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 -- 172.21.15.116:0/615042021 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a400b33e0 con 0x7f5a4015cb80 2023-11-28T17:09:39.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 -- 172.21.15.116:0/615042021 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f5a30001670 con 0x7f5a4015cb80 2023-11-28T17:09:39.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 -- 172.21.15.116:0/615042021 >> v1:172.21.15.116:6789/0 conn(0x7f5a4015cb80 legacy=0x7f5a400b2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 --2- 172.21.15.116:0/615042021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2c003a90 0x7f5a2c003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 --2- 172.21.15.116:0/615042021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a2c008850 0x7f5a2c00ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 --2- 172.21.15.116:0/615042021 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a40071ab0 0x7f5a2c00d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 -- 172.21.15.116:0/615042021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400b3f60 con 0x7f5a2c003a90 2023-11-28T17:09:39.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 -- 172.21.15.116:0/615042021 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c00dcb0 con 0x7f5a2c008850 2023-11-28T17:09:39.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3f7fe640 1 -- 172.21.15.116:0/615042021 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c00de70 con 0x7f5a40071ab0 2023-11-28T17:09:39.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.493+0000 7f5a3ffff640 1 --2- 172.21.15.116:0/615042021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2c003a90 0x7f5a2c003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.494+0000 7f5a37fff640 1 --2- 172.21.15.116:0/615042021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a2c008850 0x7f5a2c00ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.494+0000 7f5a3ffff640 1 -- 172.21.15.116:0/615042021 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a40071ab0 msgr2=0x7f5a2c00d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:39.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.494+0000 7f5a3ffff640 1 --2- 172.21.15.116:0/615042021 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a40071ab0 0x7f5a2c00d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.494+0000 7f5a3ffff640 1 -- 172.21.15.116:0/615042021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a2c008850 msgr2=0x7f5a2c00ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:39.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.494+0000 7f5a3ffff640 1 --2- 172.21.15.116:0/615042021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a2c008850 0x7f5a2c00ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.494+0000 7f5a3ffff640 1 -- 172.21.15.116:0/615042021 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c003440 con 0x7f5a2c003a90 2023-11-28T17:09:39.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.494+0000 7f5a37fff640 1 --2- 172.21.15.116:0/615042021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a2c008850 0x7f5a2c00ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:39.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.495+0000 7f5a3ffff640 1 --2- 172.21.15.116:0/615042021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2c003a90 0x7f5a2c003eb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000c240 tx=0x7f5a2c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b33d0cd00d3f1df server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:39.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.495+0000 7f5a3f7fe640 1 -- 172.21.15.116:0/615042021 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a30005ad0 con 0x7f5a2c003a90 2023-11-28T17:09:39.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.495+0000 7f5a3f7fe640 1 -- 172.21.15.116:0/615042021 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a30005c30 con 0x7f5a2c003a90 2023-11-28T17:09:39.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.495+0000 7f5a3f7fe640 1 -- 172.21.15.116:0/615042021 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3002c8b0 con 0x7f5a2c003a90 2023-11-28T17:09:39.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.496+0000 7f5a45543640 1 -- 172.21.15.116:0/615042021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2c003a90 msgr2=0x7f5a2c003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.496+0000 7f5a45543640 1 --2- 172.21.15.116:0/615042021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a2c003a90 0x7f5a2c003eb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5a3000c240 tx=0x7f5a2c0036f0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.496+0000 7f5a44d42640 1 -- 172.21.15.116:0/615042021 reap_dead start 2023-11-28T17:09:39.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.496+0000 7f5a45543640 1 -- 172.21.15.116:0/615042021 shutdown_connections 2023-11-28T17:09:39.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.496+0000 7f5a45543640 1 -- 172.21.15.116:0/615042021 >> 172.21.15.116:0/615042021 conn(0x7f5a400b0950 msgr2=0x7f5a400b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:39.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.496+0000 7f5a45543640 1 -- 172.21.15.116:0/615042021 shutdown_connections 2023-11-28T17:09:39.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.496+0000 7f5a45543640 1 -- 172.21.15.116:0/615042021 wait complete. 2023-11-28T17:09:39.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.498+0000 7f5a45543640 1 Processor -- start 2023-11-28T17:09:39.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.498+0000 7f5a45543640 1 -- start start 2023-11-28T17:09:39.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.499+0000 7f5a45543640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a2c003a90 0x7f5a4014bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.499+0000 7f5a45543640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a4014b290 0x7f5a401516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.499+0000 7f5a45543640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a40151c00 0x7f5a4015b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.499+0000 7f5a45543640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a401520b0 con 0x7f5a40151c00 2023-11-28T17:09:39.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.499+0000 7f5a45543640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4009aaf0 con 0x7f5a4014b290 2023-11-28T17:09:39.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.499+0000 7f5a45543640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4009add0 con 0x7f5a2c003a90 2023-11-28T17:09:39.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.499+0000 7f5a44d42640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a40151c00 0x7f5a4015b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.499+0000 7f5a44d42640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a40151c00 0x7f5a4015b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59850/0 (socket says 172.21.15.116:59850) 2023-11-28T17:09:39.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.499+0000 7f5a44d42640 1 -- 172.21.15.116:0/4144597815 learned_addr learned my addr 172.21.15.116:0/4144597815 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:39.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.499+0000 7f5a37fff640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a4014b290 0x7f5a401516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.499+0000 7f5a44d42640 1 -- 172.21.15.116:0/4144597815 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a2c003a90 msgr2=0x7f5a4014bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.499+0000 7f5a3ffff640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a2c003a90 0x7f5a4014bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.500+0000 7f5a44d42640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a2c003a90 0x7f5a4014bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.500+0000 7f5a44d42640 1 -- 172.21.15.116:0/4144597815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a4014b290 msgr2=0x7f5a401516c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.500+0000 7f5a44d42640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a4014b290 0x7f5a401516c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.500+0000 7f5a44d42640 1 -- 172.21.15.116:0/4144597815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4009b0b0 con 0x7f5a40151c00 2023-11-28T17:09:39.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.500+0000 7f5a3ffff640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a2c003a90 0x7f5a4014bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:39.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.500+0000 7f5a37fff640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a4014b290 0x7f5a401516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:39.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.500+0000 7f5a44d42640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a40151c00 0x7f5a4015b630 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5a400b19d0 tx=0x7f5a3800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:39.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.500+0000 7f5a3d7fa640 1 -- 172.21.15.116:0/4144597815 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38005d40 con 0x7f5a40151c00 2023-11-28T17:09:39.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.500+0000 7f5a3d7fa640 1 -- 172.21.15.116:0/4144597815 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3800ced0 con 0x7f5a40151c00 2023-11-28T17:09:39.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.500+0000 7f5a45543640 1 -- 172.21.15.116:0/4144597815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4009b390 con 0x7f5a40151c00 2023-11-28T17:09:39.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.500+0000 7f5a3d7fa640 1 -- 172.21.15.116:0/4144597815 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3800a850 con 0x7f5a40151c00 2023-11-28T17:09:39.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.500+0000 7f5a45543640 1 -- 172.21.15.116:0/4144597815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4015c320 con 0x7f5a40151c00 2023-11-28T17:09:39.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.502+0000 7f5a45543640 1 -- 172.21.15.116:0/4144597815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4009d0b0 con 0x7f5a40151c00 2023-11-28T17:09:39.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.503+0000 7f5a3d7fa640 1 -- 172.21.15.116:0/4144597815 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f5a3800a9b0 con 0x7f5a40151c00 2023-11-28T17:09:39.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.504+0000 7f5a3d7fa640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5a2006b2d0 0x7f5a2006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.504+0000 7f5a3d7fa640 1 -- 172.21.15.116:0/4144597815 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a38088a80 con 0x7f5a40151c00 2023-11-28T17:09:39.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.504+0000 7f5a3ffff640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5a2006b2d0 0x7f5a2006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.505+0000 7f5a3ffff640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5a2006b2d0 0x7f5a2006d790 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5a30034000 tx=0x7f5a30009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:39.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.508+0000 7f5a3d7fa640 1 -- 172.21.15.116:0/4144597815 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a380539b0 con 0x7f5a40151c00 2023-11-28T17:09:39.702 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.701+0000 7f5a45543640 1 -- 172.21.15.116:0/4144597815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a400ce7d0 con 0x7f5a40151c00 2023-11-28T17:09:39.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.702+0000 7f5a3d7fa640 1 -- 172.21.15.116:0/4144597815 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16521 (secure 0 0 0) 0x7f5a38017090 con 0x7f5a40151c00 2023-11-28T17:09:39.705 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:39.705 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:01.970957+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:39.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.705+0000 7f5a45543640 1 -- 172.21.15.116:0/4144597815 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5a2006b2d0 msgr2=0x7f5a2006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.705+0000 7f5a45543640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5a2006b2d0 0x7f5a2006d790 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5a30034000 tx=0x7f5a30009030 comp rx=0 tx=0).stop 2023-11-28T17:09:39.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.705+0000 7f5a45543640 1 -- 172.21.15.116:0/4144597815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a40151c00 msgr2=0x7f5a4015b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.705+0000 7f5a45543640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a40151c00 0x7f5a4015b630 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5a400b19d0 tx=0x7f5a3800cb90 comp rx=0 tx=0).stop 2023-11-28T17:09:39.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.705+0000 7f5a45543640 1 -- 172.21.15.116:0/4144597815 shutdown_connections 2023-11-28T17:09:39.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.705+0000 7f5a45543640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5a2006b2d0 0x7f5a2006d790 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.705+0000 7f5a45543640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a40151c00 0x7f5a4015b630 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.705+0000 7f5a45543640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a4014b290 0x7f5a401516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.705+0000 7f5a45543640 1 --2- 172.21.15.116:0/4144597815 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5a2c003a90 0x7f5a4014bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.705+0000 7f5a45543640 1 -- 172.21.15.116:0/4144597815 >> 172.21.15.116:0/4144597815 conn(0x7f5a400b0950 msgr2=0x7f5a4006d480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:39.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.705+0000 7f5a45543640 1 -- 172.21.15.116:0/4144597815 shutdown_connections 2023-11-28T17:09:39.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.705+0000 7f5a45543640 1 -- 172.21.15.116:0/4144597815 wait complete. 2023-11-28T17:09:39.724 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:09:39.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:09:39.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.932+0000 7fc23c1be640 1 Processor -- start 2023-11-28T17:09:39.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.932+0000 7fc23c1be640 1 -- start start 2023-11-28T17:09:39.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.933+0000 7fc23c1be640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23413f640 0x7fc23413fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.933+0000 7fc23c1be640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc234071d20 0x7fc234072170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.933+0000 7fc23c1be640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2340726b0 con 0x7fc23413cee0 2023-11-28T17:09:39.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.933+0000 7fc23c1be640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23413dd10 con 0x7fc234071d20 2023-11-28T17:09:39.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.933+0000 7fc23c1be640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23413d2c0 con 0x7fc23413f640 2023-11-28T17:09:39.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.934+0000 7fc239f5a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc23413cee0 0x7fc23413d600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41264/0 (socket says 172.21.15.116:41264) 2023-11-28T17:09:39.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.934+0000 7fc239f5a640 1 -- 172.21.15.116:0/3710545282 learned_addr learned my addr 172.21.15.116:0/3710545282 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:39.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.934+0000 7fc239759640 1 --2- 172.21.15.116:0/3710545282 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23413f640 0x7fc23413fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.934+0000 7fc23a75b640 1 --2- 172.21.15.116:0/3710545282 >> v2:172.21.15.156:3300/0 conn(0x7fc234071d20 0x7fc234072170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.935+0000 7fc238f58640 1 -- 172.21.15.116:0/3710545282 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496509027 0 0) 0x7fc2340726b0 con 0x7fc23413cee0 2023-11-28T17:09:39.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.935+0000 7fc238f58640 1 -- 172.21.15.116:0/3710545282 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc220003520 con 0x7fc23413cee0 2023-11-28T17:09:39.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.935+0000 7fc239759640 1 -- 172.21.15.116:0/3710545282 >> v2:172.21.15.156:3300/0 conn(0x7fc234071d20 msgr2=0x7fc234072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:39.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.935+0000 7fc239759640 1 --2- 172.21.15.116:0/3710545282 >> v2:172.21.15.156:3300/0 conn(0x7fc234071d20 0x7fc234072170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.935+0000 7fc239759640 1 -- 172.21.15.116:0/3710545282 >> v1:172.21.15.116:6789/0 conn(0x7fc23413cee0 legacy=0x7fc23413d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.935+0000 7fc239759640 1 -- 172.21.15.116:0/3710545282 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2341400e0 con 0x7fc23413f640 2023-11-28T17:09:39.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.936+0000 7fc239759640 1 --2- 172.21.15.116:0/3710545282 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23413f640 0x7fc23413fab0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc2240097a0 tx=0x7fc22402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=900c6e057fe6fd64 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:39.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.937+0000 7fc238f58640 1 -- 172.21.15.116:0/3710545282 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2240316f0 con 0x7fc23413f640 2023-11-28T17:09:39.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.937+0000 7fc238f58640 1 -- 172.21.15.116:0/3710545282 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc224031850 con 0x7fc23413f640 2023-11-28T17:09:39.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.937+0000 7fc238f58640 1 -- 172.21.15.116:0/3710545282 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc224031be0 con 0x7fc23413f640 2023-11-28T17:09:39.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.937+0000 7fc23c1be640 1 -- 172.21.15.116:0/3710545282 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23413f640 msgr2=0x7fc23413fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.937+0000 7fc23c1be640 1 --2- 172.21.15.116:0/3710545282 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23413f640 0x7fc23413fab0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc2240097a0 tx=0x7fc22402b790 comp rx=0 tx=0).stop 2023-11-28T17:09:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.937+0000 7fc23c1be640 1 -- 172.21.15.116:0/3710545282 shutdown_connections 2023-11-28T17:09:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.937+0000 7fc23c1be640 1 --2- 172.21.15.116:0/3710545282 >> v2:172.21.15.156:3300/0 conn(0x7fc234071d20 0x7fc234072170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.937+0000 7fc23c1be640 1 --2- 172.21.15.116:0/3710545282 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23413f640 0x7fc23413fab0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.937+0000 7fc23c1be640 1 -- 172.21.15.116:0/3710545282 >> 172.21.15.116:0/3710545282 conn(0x7fc2340b0cf0 msgr2=0x7fc2340b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.937+0000 7fc23c1be640 1 -- 172.21.15.116:0/3710545282 shutdown_connections 2023-11-28T17:09:39.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.937+0000 7fc23c1be640 1 -- 172.21.15.116:0/3710545282 wait complete. 2023-11-28T17:09:39.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.939+0000 7fc23c1be640 1 Processor -- start 2023-11-28T17:09:39.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.939+0000 7fc23c1be640 1 -- start start 2023-11-28T17:09:39.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.940+0000 7fc23c1be640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc234071d20 0x7fc234136520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.940+0000 7fc23c1be640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23413cee0 0x7fc2341f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.940+0000 7fc23c1be640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23413f640 0x7fc234137090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.940+0000 7fc23c1be640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc234152240 con 0x7fc234071d20 2023-11-28T17:09:39.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.940+0000 7fc23c1be640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2341375d0 con 0x7fc23413cee0 2023-11-28T17:09:39.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.940+0000 7fc23c1be640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2341515c0 con 0x7fc23413f640 2023-11-28T17:09:39.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.941+0000 7fc239f5a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc234071d20 0x7fc234136520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.941+0000 7fc239f5a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc234071d20 0x7fc234136520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59854/0 (socket says 172.21.15.116:59854) 2023-11-28T17:09:39.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.941+0000 7fc239f5a640 1 -- 172.21.15.116:0/649504579 learned_addr learned my addr 172.21.15.116:0/649504579 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:39.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.941+0000 7fc23a75b640 1 --2- 172.21.15.116:0/649504579 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23413f640 0x7fc234137090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.941+0000 7fc239759640 1 --2- 172.21.15.116:0/649504579 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23413cee0 0x7fc2341f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.941+0000 7fc239f5a640 1 -- 172.21.15.116:0/649504579 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23413f640 msgr2=0x7fc234137090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.941+0000 7fc239f5a640 1 --2- 172.21.15.116:0/649504579 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23413f640 0x7fc234137090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.941+0000 7fc239f5a640 1 -- 172.21.15.116:0/649504579 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23413cee0 msgr2=0x7fc2341f28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:39.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.941+0000 7fc239f5a640 1 --2- 172.21.15.116:0/649504579 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23413cee0 0x7fc2341f28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:39.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.941+0000 7fc239f5a640 1 -- 172.21.15.116:0/649504579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc224009400 con 0x7fc234071d20 2023-11-28T17:09:39.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.941+0000 7fc239f5a640 1 --2- 172.21.15.116:0/649504579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc234071d20 0x7fc234136520 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc23000c410 tx=0x7fc23000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:39.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.942+0000 7fc22a7fc640 1 -- 172.21.15.116:0/649504579 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc230005ce0 con 0x7fc234071d20 2023-11-28T17:09:39.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.942+0000 7fc22a7fc640 1 -- 172.21.15.116:0/649504579 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc230005e40 con 0x7fc234071d20 2023-11-28T17:09:39.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.942+0000 7fc22a7fc640 1 -- 172.21.15.116:0/649504579 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2300072f0 con 0x7fc234071d20 2023-11-28T17:09:39.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.942+0000 7fc23c1be640 1 -- 172.21.15.116:0/649504579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2341518a0 con 0x7fc234071d20 2023-11-28T17:09:39.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.942+0000 7fc23c1be640 1 -- 172.21.15.116:0/649504579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc234151e60 con 0x7fc234071d20 2023-11-28T17:09:39.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.943+0000 7fc23c1be640 1 -- 172.21.15.116:0/649504579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23413c720 con 0x7fc234071d20 2023-11-28T17:09:39.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.944+0000 7fc22a7fc640 1 -- 172.21.15.116:0/649504579 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fc230007510 con 0x7fc234071d20 2023-11-28T17:09:39.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.945+0000 7fc22a7fc640 1 --2- 172.21.15.116:0/649504579 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc21806b2a0 0x7fc21806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:39.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.945+0000 7fc22a7fc640 1 -- 172.21.15.116:0/649504579 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc230089340 con 0x7fc234071d20 2023-11-28T17:09:39.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.945+0000 7fc239759640 1 --2- 172.21.15.116:0/649504579 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc21806b2a0 0x7fc21806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:39.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.946+0000 7fc239759640 1 --2- 172.21.15.116:0/649504579 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc21806b2a0 0x7fc21806d760 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc224036000 tx=0x7fc224008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:39.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:39.949+0000 7fc22a7fc640 1 -- 172.21.15.116:0/649504579 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2300541d0 con 0x7fc234071d20 2023-11-28T17:09:40.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.145+0000 7fc23c1be640 1 -- 172.21.15.116:0/649504579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc234151ff0 con 0x7fc234071d20 2023-11-28T17:09:40.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.146+0000 7fc22a7fc640 1 -- 172.21.15.116:0/649504579 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc2300577f0 con 0x7fc234071d20 2023-11-28T17:09:40.148 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:09:40.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.149+0000 7fc23c1be640 1 -- 172.21.15.116:0/649504579 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc21806b2a0 msgr2=0x7fc21806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:40.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.149+0000 7fc23c1be640 1 --2- 172.21.15.116:0/649504579 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc21806b2a0 0x7fc21806d760 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc224036000 tx=0x7fc224008030 comp rx=0 tx=0).stop 2023-11-28T17:09:40.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.149+0000 7fc23c1be640 1 -- 172.21.15.116:0/649504579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc234071d20 msgr2=0x7fc234136520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:40.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.149+0000 7fc23c1be640 1 --2- 172.21.15.116:0/649504579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc234071d20 0x7fc234136520 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc23000c410 tx=0x7fc23000c8d0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.149+0000 7fc23c1be640 1 -- 172.21.15.116:0/649504579 shutdown_connections 2023-11-28T17:09:40.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.149+0000 7fc23c1be640 1 --2- 172.21.15.116:0/649504579 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc21806b2a0 0x7fc21806d760 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.149+0000 7fc23c1be640 1 --2- 172.21.15.116:0/649504579 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc23413f640 0x7fc234137090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.149+0000 7fc23c1be640 1 --2- 172.21.15.116:0/649504579 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc23413cee0 0x7fc2341f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.149+0000 7fc23c1be640 1 --2- 172.21.15.116:0/649504579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc234071d20 0x7fc234136520 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.149+0000 7fc23c1be640 1 -- 172.21.15.116:0/649504579 >> 172.21.15.116:0/649504579 conn(0x7fc2340b0cf0 msgr2=0x7fc23413e980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:40.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.149+0000 7fc23c1be640 1 -- 172.21.15.116:0/649504579 shutdown_connections 2023-11-28T17:09:40.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.149+0000 7fc23c1be640 1 -- 172.21.15.116:0/649504579 wait complete. 2023-11-28T17:09:40.169 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:09:40.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.380+0000 7fb2d3589640 1 Processor -- start 2023-11-28T17:09:40.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.381+0000 7fb2d3589640 1 -- start start 2023-11-28T17:09:40.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.382+0000 7fb2d3589640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb2cc15b0c0 0x7fb2cc15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:40.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.382+0000 7fb2d3589640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2cc072250 0x7fb2cc0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:40.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.382+0000 7fb2d3589640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc0cd320 con 0x7fb2cc0ce940 2023-11-28T17:09:40.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.382+0000 7fb2d3589640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc0ced80 con 0x7fb2cc15b0c0 2023-11-28T17:09:40.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.382+0000 7fb2d3589640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc0cbee0 con 0x7fb2cc072250 2023-11-28T17:09:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.383+0000 7fb2d1325640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb2cc15b0c0 0x7fb2cc15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.383+0000 7fb2d1b26640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2cc072250 0x7fb2cc0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.383+0000 7fb2d0b24640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fb2cc0ce940 0x7fb2cc071b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.116:40558/0 (socket says 172.21.15.116:40558) 2023-11-28T17:09:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.383+0000 7fb2d0b24640 1 -- 172.21.15.116:0/3033067818 learned_addr learned my addr 172.21.15.116:0/3033067818 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.384+0000 7fb2d1325640 1 -- 172.21.15.116:0/3033067818 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2cc072250 msgr2=0x7fb2cc0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:40.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.384+0000 7fb2d1325640 1 --2- 172.21.15.116:0/3033067818 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2cc072250 0x7fb2cc0cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.384+0000 7fb2d1325640 1 -- 172.21.15.116:0/3033067818 >> v1:172.21.15.156:6789/0 conn(0x7fb2cc0ce940 legacy=0x7fb2cc071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:40.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.384+0000 7fb2d1325640 1 -- 172.21.15.116:0/3033067818 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc0cc710 con 0x7fb2cc15b0c0 2023-11-28T17:09:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.386+0000 7fb2d1325640 1 --2- 172.21.15.116:0/3033067818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2cc15b0c0 0x7fb2cc15ed10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80097a0 tx=0x7fb2c802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17f2512deb965b83 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.386+0000 7fb2c3fff640 1 -- 172.21.15.116:0/3033067818 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c802e070 con 0x7fb2cc15b0c0 2023-11-28T17:09:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.386+0000 7fb2c3fff640 1 -- 172.21.15.116:0/3033067818 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c80315a0 con 0x7fb2cc15b0c0 2023-11-28T17:09:40.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.386+0000 7fb2c3fff640 1 -- 172.21.15.116:0/3033067818 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8031930 con 0x7fb2cc15b0c0 2023-11-28T17:09:40.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.387+0000 7fb2d3589640 1 -- 172.21.15.116:0/3033067818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2cc15b0c0 msgr2=0x7fb2cc15ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:40.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.387+0000 7fb2d3589640 1 --2- 172.21.15.116:0/3033067818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2cc15b0c0 0x7fb2cc15ed10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80097a0 tx=0x7fb2c802b840 comp rx=0 tx=0).stop 2023-11-28T17:09:40.388 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.387+0000 7fb2d3589640 1 -- 172.21.15.116:0/3033067818 shutdown_connections 2023-11-28T17:09:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.387+0000 7fb2d3589640 1 --2- 172.21.15.116:0/3033067818 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2cc072250 0x7fb2cc0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.387+0000 7fb2d3589640 1 --2- 172.21.15.116:0/3033067818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2cc15b0c0 0x7fb2cc15ed10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.387+0000 7fb2d3589640 1 -- 172.21.15.116:0/3033067818 >> 172.21.15.116:0/3033067818 conn(0x7fb2cc0b0cf0 msgr2=0x7fb2cc0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.387+0000 7fb2d3589640 1 -- 172.21.15.116:0/3033067818 shutdown_connections 2023-11-28T17:09:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.387+0000 7fb2d3589640 1 -- 172.21.15.116:0/3033067818 wait complete. 2023-11-28T17:09:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.389+0000 7fb2d3589640 1 Processor -- start 2023-11-28T17:09:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.389+0000 7fb2d3589640 1 -- start start 2023-11-28T17:09:40.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d3589640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2cc072250 0x7fb2cc155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:40.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d3589640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2cc0ce940 0x7fb2cc157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:40.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d3589640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2cc15b0c0 0x7fb2cc0c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:40.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d3589640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc158c60 con 0x7fb2cc072250 2023-11-28T17:09:40.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d3589640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc1583c0 con 0x7fb2cc0ce940 2023-11-28T17:09:40.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d3589640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc1586a0 con 0x7fb2cc15b0c0 2023-11-28T17:09:40.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d1325640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2cc072250 0x7fb2cc155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:40.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d1325640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2cc072250 0x7fb2cc155a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59864/0 (socket says 172.21.15.116:59864) 2023-11-28T17:09:40.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d1325640 1 -- 172.21.15.116:0/1176000402 learned_addr learned my addr 172.21.15.116:0/1176000402 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:40.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d1b26640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2cc15b0c0 0x7fb2cc0c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:40.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d0b24640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2cc0ce940 0x7fb2cc157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:40.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d1325640 1 -- 172.21.15.116:0/1176000402 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2cc15b0c0 msgr2=0x7fb2cc0c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:40.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d1325640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2cc15b0c0 0x7fb2cc0c55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d1325640 1 -- 172.21.15.116:0/1176000402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2cc0ce940 msgr2=0x7fb2cc157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:40.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.390+0000 7fb2d1325640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2cc0ce940 0x7fb2cc157920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.391+0000 7fb2d1325640 1 -- 172.21.15.116:0/1176000402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c8009400 con 0x7fb2cc072250 2023-11-28T17:09:40.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.391+0000 7fb2d0b24640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2cc0ce940 0x7fb2cc157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:40.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.391+0000 7fb2d1325640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2cc072250 0x7fb2cc155a00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8034040 tx=0x7fb2c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:40.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.391+0000 7fb2c1ffb640 1 -- 172.21.15.116:0/1176000402 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8032840 con 0x7fb2cc072250 2023-11-28T17:09:40.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.391+0000 7fb2c1ffb640 1 -- 172.21.15.116:0/1176000402 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c8031f00 con 0x7fb2cc072250 2023-11-28T17:09:40.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.391+0000 7fb2c1ffb640 1 -- 172.21.15.116:0/1176000402 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8032be0 con 0x7fb2cc072250 2023-11-28T17:09:40.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.391+0000 7fb2d3589640 1 -- 172.21.15.116:0/1176000402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2cc158980 con 0x7fb2cc072250 2023-11-28T17:09:40.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.391+0000 7fb2d3589640 1 -- 172.21.15.116:0/1176000402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2cc0c51a0 con 0x7fb2cc072250 2023-11-28T17:09:40.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.392+0000 7fb2d3589640 1 -- 172.21.15.116:0/1176000402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2cc0ac970 con 0x7fb2cc072250 2023-11-28T17:09:40.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.393+0000 7fb2c1ffb640 1 -- 172.21.15.116:0/1176000402 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb2c8032d40 con 0x7fb2cc072250 2023-11-28T17:09:40.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.394+0000 7fb2c1ffb640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb2b006b2d0 0x7fb2b006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:40.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.394+0000 7fb2c1ffb640 1 -- 172.21.15.116:0/1176000402 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2c80abac0 con 0x7fb2cc072250 2023-11-28T17:09:40.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.395+0000 7fb2d0b24640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb2b006b2d0 0x7fb2b006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:40.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.395+0000 7fb2d0b24640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb2b006b2d0 0x7fb2b006d790 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc159e40 tx=0x7fb2bc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:40.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.399+0000 7fb2c1ffb640 1 -- 172.21.15.116:0/1176000402 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2c8076920 con 0x7fb2cc072250 2023-11-28T17:09:40.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.588+0000 7fb2d3589640 1 -- 172.21.15.116:0/1176000402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb2cc069360 con 0x7fb2cc072250 2023-11-28T17:09:40.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.589+0000 7fb2c1ffb640 1 -- 172.21.15.116:0/1176000402 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb2c8079f40 con 0x7fb2cc072250 2023-11-28T17:09:40.591 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:09:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.592+0000 7fb2d3589640 1 -- 172.21.15.116:0/1176000402 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb2b006b2d0 msgr2=0x7fb2b006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.592+0000 7fb2d3589640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb2b006b2d0 0x7fb2b006d790 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc159e40 tx=0x7fb2bc009040 comp rx=0 tx=0).stop 2023-11-28T17:09:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.592+0000 7fb2d3589640 1 -- 172.21.15.116:0/1176000402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2cc072250 msgr2=0x7fb2cc155a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.592+0000 7fb2d3589640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2cc072250 0x7fb2cc155a00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8034040 tx=0x7fb2c80046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.592+0000 7fb2d3589640 1 -- 172.21.15.116:0/1176000402 shutdown_connections 2023-11-28T17:09:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.592+0000 7fb2d3589640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb2b006b2d0 0x7fb2b006d790 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.592+0000 7fb2d3589640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2cc15b0c0 0x7fb2cc0c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.592+0000 7fb2d3589640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2cc0ce940 0x7fb2cc157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.592+0000 7fb2d3589640 1 --2- 172.21.15.116:0/1176000402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2cc072250 0x7fb2cc155a00 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.592+0000 7fb2d3589640 1 -- 172.21.15.116:0/1176000402 >> 172.21.15.116:0/1176000402 conn(0x7fb2cc0b0cf0 msgr2=0x7fb2cc0ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.593+0000 7fb2d3589640 1 -- 172.21.15.116:0/1176000402 shutdown_connections 2023-11-28T17:09:40.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.593+0000 7fb2d3589640 1 -- 172.21.15.116:0/1176000402 wait complete. 2023-11-28T17:09:40.612 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:09:40.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.829+0000 7f86059c9640 1 Processor -- start 2023-11-28T17:09:40.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.829+0000 7f86059c9640 1 -- start start 2023-11-28T17:09:40.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.831+0000 7f86059c9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f860015b0c0 0x7f860015ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:40.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.831+0000 7f86059c9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8600072250 0x7f86000cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:40.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.832+0000 7f86059c9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86000cd320 con 0x7f86000ce940 2023-11-28T17:09:40.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.832+0000 7f86059c9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86000ced80 con 0x7f860015b0c0 2023-11-28T17:09:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.832+0000 7f86059c9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86000cbee0 con 0x7f8600072250 2023-11-28T17:09:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.832+0000 7f85feffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f860015b0c0 0x7f860015ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.832+0000 7f85f7fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f86000ce940 0x7f8600071b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41270/0 (socket says 172.21.15.116:41270) 2023-11-28T17:09:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.832+0000 7f85f7fff640 1 -- 172.21.15.116:0/3452184796 learned_addr learned my addr 172.21.15.116:0/3452184796 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:40.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.832+0000 7f85ff7fe640 1 --2- 172.21.15.116:0/3452184796 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8600072250 0x7f86000cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:40.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.832+0000 7f85feffd640 1 -- 172.21.15.116:0/3452184796 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8600072250 msgr2=0x7f86000cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:40.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.832+0000 7f85feffd640 1 --2- 172.21.15.116:0/3452184796 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8600072250 0x7f86000cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.832+0000 7f85feffd640 1 -- 172.21.15.116:0/3452184796 >> v1:172.21.15.116:6789/0 conn(0x7f86000ce940 legacy=0x7f8600071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:40.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.832+0000 7f85feffd640 1 -- 172.21.15.116:0/3452184796 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86000cc710 con 0x7f860015b0c0 2023-11-28T17:09:40.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.832+0000 7f85ff7fe640 1 --2- 172.21.15.116:0/3452184796 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8600072250 0x7f86000cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:40.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.833+0000 7f85feffd640 1 --2- 172.21.15.116:0/3452184796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860015b0c0 0x7f860015ed10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f85f00097a0 tx=0x7f85f002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebbf3144fb7e50e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:40.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.834+0000 7f85fe7fc640 1 -- 172.21.15.116:0/3452184796 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0031780 con 0x7f860015b0c0 2023-11-28T17:09:40.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.834+0000 7f85fe7fc640 1 -- 172.21.15.116:0/3452184796 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f00318e0 con 0x7f860015b0c0 2023-11-28T17:09:40.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.834+0000 7f85fe7fc640 1 -- 172.21.15.116:0/3452184796 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0031c70 con 0x7f860015b0c0 2023-11-28T17:09:40.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.834+0000 7f86059c9640 1 -- 172.21.15.116:0/3452184796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860015b0c0 msgr2=0x7f860015ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:40.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.834+0000 7f86059c9640 1 --2- 172.21.15.116:0/3452184796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860015b0c0 0x7f860015ed10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f85f00097a0 tx=0x7f85f002b820 comp rx=0 tx=0).stop 2023-11-28T17:09:40.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.835+0000 7f86059c9640 1 -- 172.21.15.116:0/3452184796 shutdown_connections 2023-11-28T17:09:40.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.835+0000 7f86059c9640 1 --2- 172.21.15.116:0/3452184796 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8600072250 0x7f86000cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.835+0000 7f86059c9640 1 --2- 172.21.15.116:0/3452184796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860015b0c0 0x7f860015ed10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.835+0000 7f86059c9640 1 -- 172.21.15.116:0/3452184796 >> 172.21.15.116:0/3452184796 conn(0x7f86000b0cf0 msgr2=0x7f86000b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:40.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.835+0000 7f86059c9640 1 -- 172.21.15.116:0/3452184796 shutdown_connections 2023-11-28T17:09:40.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.835+0000 7f86059c9640 1 -- 172.21.15.116:0/3452184796 wait complete. 2023-11-28T17:09:40.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.836+0000 7f86059c9640 1 Processor -- start 2023-11-28T17:09:40.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.837+0000 7f86059c9640 1 -- start start 2023-11-28T17:09:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.837+0000 7f86059c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8600072250 0x7f86000c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.837+0000 7f86059c9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86000ce940 0x7f860014eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.837+0000 7f86059c9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f860015b0c0 0x7f86000c4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:40.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.837+0000 7f86059c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86000c7cc0 con 0x7f8600072250 2023-11-28T17:09:40.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.837+0000 7f86059c9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860014eff0 con 0x7f86000ce940 2023-11-28T17:09:40.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.837+0000 7f86059c9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86000c52c0 con 0x7f860015b0c0 2023-11-28T17:09:40.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.838+0000 7f85feffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8600072250 0x7f86000c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:40.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.838+0000 7f85feffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8600072250 0x7f86000c5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59870/0 (socket says 172.21.15.116:59870) 2023-11-28T17:09:40.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.838+0000 7f85feffd640 1 -- 172.21.15.116:0/3751045155 learned_addr learned my addr 172.21.15.116:0/3751045155 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:40.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.838+0000 7f85f7fff640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86000ce940 0x7f860014eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:40.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.838+0000 7f85ff7fe640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f860015b0c0 0x7f86000c4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:40.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.838+0000 7f85feffd640 1 -- 172.21.15.116:0/3751045155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f860015b0c0 msgr2=0x7f86000c4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:40.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.838+0000 7f85feffd640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f860015b0c0 0x7f86000c4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.838+0000 7f85feffd640 1 -- 172.21.15.116:0/3751045155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86000ce940 msgr2=0x7f860014eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:40.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.838+0000 7f85feffd640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86000ce940 0x7f860014eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:40.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.838+0000 7f85feffd640 1 -- 172.21.15.116:0/3751045155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f0009400 con 0x7f8600072250 2023-11-28T17:09:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.838+0000 7f85f7fff640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86000ce940 0x7f860014eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.838+0000 7f85feffd640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8600072250 0x7f86000c5aa0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f85f0034040 tx=0x7f85f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.839+0000 7f85fcff9640 1 -- 172.21.15.116:0/3751045155 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0032960 con 0x7f8600072250 2023-11-28T17:09:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.839+0000 7f85fcff9640 1 -- 172.21.15.116:0/3751045155 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f0032ac0 con 0x7f8600072250 2023-11-28T17:09:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.839+0000 7f85fcff9640 1 -- 172.21.15.116:0/3751045155 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f0032e70 con 0x7f8600072250 2023-11-28T17:09:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.839+0000 7f86059c9640 1 -- 172.21.15.116:0/3751045155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86000c60f0 con 0x7f8600072250 2023-11-28T17:09:40.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.839+0000 7f86059c9640 1 -- 172.21.15.116:0/3751045155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8600157060 con 0x7f8600072250 2023-11-28T17:09:40.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.840+0000 7f86059c9640 1 -- 172.21.15.116:0/3751045155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86000cd680 con 0x7f8600072250 2023-11-28T17:09:40.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.841+0000 7f85fcff9640 1 -- 172.21.15.116:0/3751045155 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f85f003b3f0 con 0x7f8600072250 2023-11-28T17:09:40.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.842+0000 7f85fcff9640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f85e006b2d0 0x7f85e006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:40.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.842+0000 7f85fcff9640 1 -- 172.21.15.116:0/3751045155 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85f00abfa0 con 0x7f8600072250 2023-11-28T17:09:40.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.842+0000 7f85f7fff640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f85e006b2d0 0x7f85e006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:40.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.843+0000 7f85f7fff640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f85e006b2d0 0x7f85e006d790 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f86000b0150 tx=0x7f85e8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:40.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:40.846+0000 7f85fcff9640 1 -- 172.21.15.116:0/3751045155 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f85f00dee90 con 0x7f8600072250 2023-11-28T17:09:41.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.045+0000 7f86059c9640 1 -- 172.21.15.116:0/3751045155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8600066f70 con 0x7f8600072250 2023-11-28T17:09:41.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.046+0000 7f85fcff9640 1 -- 172.21.15.116:0/3751045155 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f85f0076ea0 con 0x7f8600072250 2023-11-28T17:09:41.047 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:09:41.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.048+0000 7f86059c9640 1 -- 172.21.15.116:0/3751045155 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f85e006b2d0 msgr2=0x7f85e006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:41.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.048+0000 7f86059c9640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f85e006b2d0 0x7f85e006d790 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f86000b0150 tx=0x7f85e8009040 comp rx=0 tx=0).stop 2023-11-28T17:09:41.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.048+0000 7f86059c9640 1 -- 172.21.15.116:0/3751045155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8600072250 msgr2=0x7f86000c5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:41.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.048+0000 7f86059c9640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8600072250 0x7f86000c5aa0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f85f0034040 tx=0x7f85f00046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:41.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.048+0000 7f86059c9640 1 -- 172.21.15.116:0/3751045155 shutdown_connections 2023-11-28T17:09:41.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.048+0000 7f86059c9640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f85e006b2d0 0x7f85e006d790 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:41.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.048+0000 7f86059c9640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f860015b0c0 0x7f86000c4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:41.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.048+0000 7f86059c9640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86000ce940 0x7f860014eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:41.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.048+0000 7f86059c9640 1 --2- 172.21.15.116:0/3751045155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8600072250 0x7f86000c5aa0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:41.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.049+0000 7f86059c9640 1 -- 172.21.15.116:0/3751045155 >> 172.21.15.116:0/3751045155 conn(0x7f86000b0cf0 msgr2=0x7f86000ad8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:41.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.049+0000 7f86059c9640 1 -- 172.21.15.116:0/3751045155 shutdown_connections 2023-11-28T17:09:41.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.049+0000 7f86059c9640 1 -- 172.21.15.116:0/3751045155 wait complete. 2023-11-28T17:09:41.069 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_mount_conn_close' 2023-11-28T17:09:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.272+0000 7f6a9620e640 1 Processor -- start 2023-11-28T17:09:41.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.272+0000 7f6a9620e640 1 -- start start 2023-11-28T17:09:41.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.274+0000 7f6a9620e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6a90161a80 0x7f6a900b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:41.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.274+0000 7f6a9620e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6a900b30e0 0x7f6a9006be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:41.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.274+0000 7f6a9620e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a900cadf0 con 0x7f6a90146a50 2023-11-28T17:09:41.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.274+0000 7f6a9620e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a90161ee0 con 0x7f6a900b30e0 2023-11-28T17:09:41.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.274+0000 7f6a9620e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a900ce7c0 con 0x7f6a90161a80 2023-11-28T17:09:41.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.274+0000 7f6a8ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6a900b30e0 0x7f6a9006be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:41.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.274+0000 7f6a8ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6a900b30e0 0x7f6a9006be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59878/0 (socket says 172.21.15.116:59878) 2023-11-28T17:09:41.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.274+0000 7f6a8ffff640 1 -- 172.21.15.116:0/1836682164 learned_addr learned my addr 172.21.15.116:0/1836682164 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:41.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.275+0000 7f6a8effd640 1 --2- 172.21.15.116:0/1836682164 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6a90161a80 0x7f6a900b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:41.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.275+0000 7f6a8ffff640 1 -- 172.21.15.116:0/1836682164 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6a90161a80 msgr2=0x7f6a900b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:41.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.275+0000 7f6a8ffff640 1 --2- 172.21.15.116:0/1836682164 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6a90161a80 0x7f6a900b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:41.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.275+0000 7f6a8ffff640 1 -- 172.21.15.116:0/1836682164 >> v1:172.21.15.156:6789/0 conn(0x7f6a90146a50 legacy=0x7f6a900cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:41.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.275+0000 7f6a8ffff640 1 -- 172.21.15.116:0/1836682164 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a900b3510 con 0x7f6a900b30e0 2023-11-28T17:09:41.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.276+0000 7f6a8ffff640 1 --2- 172.21.15.116:0/1836682164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a900b30e0 0x7f6a9006be00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6a90162020 tx=0x7f6a8802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546f648068a22bda server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:41.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.276+0000 7f6a8e7fc640 1 -- 172.21.15.116:0/1836682164 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a880317c0 con 0x7f6a900b30e0 2023-11-28T17:09:41.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.276+0000 7f6a8e7fc640 1 -- 172.21.15.116:0/1836682164 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a88031920 con 0x7f6a900b30e0 2023-11-28T17:09:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.276+0000 7f6a8e7fc640 1 -- 172.21.15.116:0/1836682164 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a88031cb0 con 0x7f6a900b30e0 2023-11-28T17:09:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.277+0000 7f6a9620e640 1 -- 172.21.15.116:0/1836682164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a900b30e0 msgr2=0x7f6a9006be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.277+0000 7f6a9620e640 1 --2- 172.21.15.116:0/1836682164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a900b30e0 0x7f6a9006be00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6a90162020 tx=0x7f6a8802b860 comp rx=0 tx=0).stop 2023-11-28T17:09:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.277+0000 7f6a9620e640 1 -- 172.21.15.116:0/1836682164 shutdown_connections 2023-11-28T17:09:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.277+0000 7f6a9620e640 1 --2- 172.21.15.116:0/1836682164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a900b30e0 0x7f6a9006be00 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.277+0000 7f6a9620e640 1 --2- 172.21.15.116:0/1836682164 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6a90161a80 0x7f6a900b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.277+0000 7f6a9620e640 1 -- 172.21.15.116:0/1836682164 >> 172.21.15.116:0/1836682164 conn(0x7f6a9006a4c0 msgr2=0x7f6a9006a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.277+0000 7f6a9620e640 1 -- 172.21.15.116:0/1836682164 shutdown_connections 2023-11-28T17:09:41.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.277+0000 7f6a9620e640 1 -- 172.21.15.116:0/1836682164 wait complete. 2023-11-28T17:09:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.279+0000 7f6a9620e640 1 Processor -- start 2023-11-28T17:09:41.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.279+0000 7f6a9620e640 1 -- start start 2023-11-28T17:09:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.280+0000 7f6a9620e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a900b30e0 0x7f6a9015f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.280+0000 7f6a9620e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a90146a50 0x7f6a901402b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.280+0000 7f6a9620e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6a90161a80 0x7f6a901407f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.280+0000 7f6a9620e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a90140e20 con 0x7f6a900b30e0 2023-11-28T17:09:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.280+0000 7f6a9620e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a90141100 con 0x7f6a90146a50 2023-11-28T17:09:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.280+0000 7f6a9620e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a901413e0 con 0x7f6a90161a80 2023-11-28T17:09:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.281+0000 7f6a8f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a900b30e0 0x7f6a9015f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:41.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.281+0000 7f6a8f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a900b30e0 0x7f6a9015f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59894/0 (socket says 172.21.15.116:59894) 2023-11-28T17:09:41.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.281+0000 7f6a8f7fe640 1 -- 172.21.15.116:0/69309810 learned_addr learned my addr 172.21.15.116:0/69309810 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:41.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.281+0000 7f6a8f7fe640 1 -- 172.21.15.116:0/69309810 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6a90161a80 msgr2=0x7f6a901407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:41.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.281+0000 7f6a8f7fe640 1 --2- 172.21.15.116:0/69309810 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6a90161a80 0x7f6a901407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:41.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.281+0000 7f6a8f7fe640 1 -- 172.21.15.116:0/69309810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a90146a50 msgr2=0x7f6a901402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:41.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.281+0000 7f6a8f7fe640 1 --2- 172.21.15.116:0/69309810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a90146a50 0x7f6a901402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:41.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.281+0000 7f6a8f7fe640 1 -- 172.21.15.116:0/69309810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a880094d0 con 0x7f6a900b30e0 2023-11-28T17:09:41.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.281+0000 7f6a8f7fe640 1 --2- 172.21.15.116:0/69309810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a900b30e0 0x7f6a9015f450 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6a8000c7c0 tx=0x7f6a8000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:41.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.282+0000 7f6a87fff640 1 -- 172.21.15.116:0/69309810 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8000aa50 con 0x7f6a900b30e0 2023-11-28T17:09:41.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.282+0000 7f6a87fff640 1 -- 172.21.15.116:0/69309810 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8000abb0 con 0x7f6a900b30e0 2023-11-28T17:09:41.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.282+0000 7f6a87fff640 1 -- 172.21.15.116:0/69309810 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a80012420 con 0x7f6a900b30e0 2023-11-28T17:09:41.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.282+0000 7f6a9620e640 1 -- 172.21.15.116:0/69309810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a901416c0 con 0x7f6a900b30e0 2023-11-28T17:09:41.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.282+0000 7f6a9620e640 1 -- 172.21.15.116:0/69309810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a9015f990 con 0x7f6a900b30e0 2023-11-28T17:09:41.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.283+0000 7f6a9620e640 1 -- 172.21.15.116:0/69309810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a90141be0 con 0x7f6a900b30e0 2023-11-28T17:09:41.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.284+0000 7f6a87fff640 1 -- 172.21.15.116:0/69309810 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6a80012580 con 0x7f6a900b30e0 2023-11-28T17:09:41.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.285+0000 7f6a87fff640 1 --2- 172.21.15.116:0/69309810 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6a7006b2a0 0x7f6a7006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:41.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.285+0000 7f6a87fff640 1 -- 172.21.15.116:0/69309810 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a80089b90 con 0x7f6a900b30e0 2023-11-28T17:09:41.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.285+0000 7f6a8effd640 1 --2- 172.21.15.116:0/69309810 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6a7006b2a0 0x7f6a7006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:41.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.286+0000 7f6a8effd640 1 --2- 172.21.15.116:0/69309810 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6a7006b2a0 0x7f6a7006d760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6a78006fd0 tx=0x7f6a78008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:41.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.289+0000 7f6a87fff640 1 -- 172.21.15.116:0/69309810 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a80054a00 con 0x7f6a900b30e0 2023-11-28T17:09:41.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:41.550+0000 7f6a9620e640 1 -- 172.21.15.116:0/69309810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_mount_conn_close"]} v 0) v1 -- 0x7f6a900693d0 con 0x7f6a900b30e0 2023-11-28T17:09:42.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.049+0000 7f6a87fff640 1 -- 172.21.15.116:0/69309810 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_mount_conn_close"]}]=0 v59) v1 ==== 147+0+0 (secure 0 0 0) 0x7f6a800160d0 con 0x7f6a900b30e0 2023-11-28T17:09:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.053+0000 7f6a9620e640 1 -- 172.21.15.116:0/69309810 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6a7006b2a0 msgr2=0x7f6a7006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.053+0000 7f6a9620e640 1 --2- 172.21.15.116:0/69309810 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6a7006b2a0 0x7f6a7006d760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6a78006fd0 tx=0x7f6a78008040 comp rx=0 tx=0).stop 2023-11-28T17:09:42.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.053+0000 7f6a9620e640 1 -- 172.21.15.116:0/69309810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a900b30e0 msgr2=0x7f6a9015f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:42.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.053+0000 7f6a9620e640 1 --2- 172.21.15.116:0/69309810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a900b30e0 0x7f6a9015f450 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6a8000c7c0 tx=0x7f6a8000cc80 comp rx=0 tx=0).stop 2023-11-28T17:09:42.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.053+0000 7f6a9620e640 1 -- 172.21.15.116:0/69309810 shutdown_connections 2023-11-28T17:09:42.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.053+0000 7f6a9620e640 1 --2- 172.21.15.116:0/69309810 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6a7006b2a0 0x7f6a7006d760 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.053+0000 7f6a9620e640 1 --2- 172.21.15.116:0/69309810 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6a90161a80 0x7f6a901407f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.053+0000 7f6a9620e640 1 --2- 172.21.15.116:0/69309810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a90146a50 0x7f6a901402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.053+0000 7f6a9620e640 1 --2- 172.21.15.116:0/69309810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a900b30e0 0x7f6a9015f450 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.053+0000 7f6a9620e640 1 -- 172.21.15.116:0/69309810 >> 172.21.15.116:0/69309810 conn(0x7f6a9006a4c0 msgr2=0x7f6a900d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:42.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.053+0000 7f6a9620e640 1 -- 172.21.15.116:0/69309810 shutdown_connections 2023-11-28T17:09:42.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.053+0000 7f6a9620e640 1 -- 172.21.15.116:0/69309810 wait complete. 2023-11-28T17:09:42.075 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T17:09:42.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.288+0000 7f54697b0640 1 Processor -- start 2023-11-28T17:09:42.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.288+0000 7f54697b0640 1 -- start start 2023-11-28T17:09:42.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.290+0000 7f54697b0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54640cd0c0 0x7f546415b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:42.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.290+0000 7f54697b0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f546415bab0 0x7f546406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:42.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.290+0000 7f54697b0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54640709e0 con 0x7f546406c770 2023-11-28T17:09:42.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.290+0000 7f54697b0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546406cc00 con 0x7f546415bab0 2023-11-28T17:09:42.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.290+0000 7f54697b0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54640703d0 con 0x7f54640cd0c0 2023-11-28T17:09:42.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.290+0000 7f54627fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f546415bab0 0x7f546406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:42.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.290+0000 7f54637fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f546406c770 0x7f5464143f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41278/0 (socket says 172.21.15.116:41278) 2023-11-28T17:09:42.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.290+0000 7f54637fe640 1 -- 172.21.15.116:0/414726915 learned_addr learned my addr 172.21.15.116:0/414726915 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:42.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.291+0000 7f5462ffd640 1 --2- 172.21.15.116:0/414726915 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54640cd0c0 0x7f546415b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:42.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.291+0000 7f5461ffb640 1 -- 172.21.15.116:0/414726915 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427287728 0 0) 0x7f54640709e0 con 0x7f546406c770 2023-11-28T17:09:42.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.291+0000 7f54627fc640 1 -- 172.21.15.116:0/414726915 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54640cd0c0 msgr2=0x7f546415b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:42.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.291+0000 7f54627fc640 1 --2- 172.21.15.116:0/414726915 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54640cd0c0 0x7f546415b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.291+0000 7f54627fc640 1 -- 172.21.15.116:0/414726915 >> v1:172.21.15.116:6789/0 conn(0x7f546406c770 legacy=0x7f5464143f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:42.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.291+0000 7f54627fc640 1 -- 172.21.15.116:0/414726915 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54640cd4a0 con 0x7f546415bab0 2023-11-28T17:09:42.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.292+0000 7f54627fc640 1 --2- 172.21.15.116:0/414726915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546415bab0 0x7f546406c230 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f54580097a0 tx=0x7f545802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0e545c915275ea server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:42.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.292+0000 7f5461ffb640 1 -- 172.21.15.116:0/414726915 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545802d040 con 0x7f546415bab0 2023-11-28T17:09:42.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.292+0000 7f5461ffb640 1 -- 172.21.15.116:0/414726915 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5458031810 con 0x7f546415bab0 2023-11-28T17:09:42.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.292+0000 7f5461ffb640 1 -- 172.21.15.116:0/414726915 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5458031ba0 con 0x7f546415bab0 2023-11-28T17:09:42.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.293+0000 7f54697b0640 1 -- 172.21.15.116:0/414726915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546415bab0 msgr2=0x7f546406c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:42.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.293+0000 7f54697b0640 1 --2- 172.21.15.116:0/414726915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546415bab0 0x7f546406c230 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f54580097a0 tx=0x7f545802b8b0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.293+0000 7f54697b0640 1 -- 172.21.15.116:0/414726915 shutdown_connections 2023-11-28T17:09:42.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.293+0000 7f54697b0640 1 --2- 172.21.15.116:0/414726915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546415bab0 0x7f546406c230 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.293+0000 7f54697b0640 1 --2- 172.21.15.116:0/414726915 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54640cd0c0 0x7f546415b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.293+0000 7f54697b0640 1 -- 172.21.15.116:0/414726915 >> 172.21.15.116:0/414726915 conn(0x7f546406a4c0 msgr2=0x7f546406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:42.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.293+0000 7f54697b0640 1 -- 172.21.15.116:0/414726915 shutdown_connections 2023-11-28T17:09:42.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.293+0000 7f54697b0640 1 -- 172.21.15.116:0/414726915 wait complete. 2023-11-28T17:09:42.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.294+0000 7f54697b0640 1 Processor -- start 2023-11-28T17:09:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.295+0000 7f54697b0640 1 -- start start 2023-11-28T17:09:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f54697b0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546406c770 0x7f5464158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f54697b0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54640cd0c0 0x7f54641525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f54697b0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f546415bab0 0x7f5464152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f54697b0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464158670 con 0x7f546406c770 2023-11-28T17:09:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f54697b0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5464157090 con 0x7f546415bab0 2023-11-28T17:09:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f54697b0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5464157370 con 0x7f54640cd0c0 2023-11-28T17:09:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f5462ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546406c770 0x7f5464158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f5462ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546406c770 0x7f5464158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59912/0 (socket says 172.21.15.116:59912) 2023-11-28T17:09:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f5462ffd640 1 -- 172.21.15.116:0/877586322 learned_addr learned my addr 172.21.15.116:0/877586322 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f54637fe640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f546415bab0 0x7f5464152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:42.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f54627fc640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54640cd0c0 0x7f54641525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f5462ffd640 1 -- 172.21.15.116:0/877586322 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54640cd0c0 msgr2=0x7f54641525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f5462ffd640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54640cd0c0 0x7f54641525c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f5462ffd640 1 -- 172.21.15.116:0/877586322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f546415bab0 msgr2=0x7f5464152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f5462ffd640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f546415bab0 0x7f5464152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f5462ffd640 1 -- 172.21.15.116:0/877586322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5458009400 con 0x7f546406c770 2023-11-28T17:09:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.296+0000 7f54637fe640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f546415bab0 0x7f5464152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.297+0000 7f5462ffd640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546406c770 0x7f5464158130 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f545400c7f0 tx=0x7f545400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.297+0000 7f544b7fe640 1 -- 172.21.15.116:0/877586322 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454004220 con 0x7f546406c770 2023-11-28T17:09:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.297+0000 7f544b7fe640 1 -- 172.21.15.116:0/877586322 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54540043a0 con 0x7f546406c770 2023-11-28T17:09:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.297+0000 7f544b7fe640 1 -- 172.21.15.116:0/877586322 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5454007860 con 0x7f546406c770 2023-11-28T17:09:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.297+0000 7f54697b0640 1 -- 172.21.15.116:0/877586322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5464157650 con 0x7f546406c770 2023-11-28T17:09:42.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.297+0000 7f54697b0640 1 -- 172.21.15.116:0/877586322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5464157c10 con 0x7f546406c770 2023-11-28T17:09:42.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.298+0000 7f54697b0640 1 -- 172.21.15.116:0/877586322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54640693d0 con 0x7f546406c770 2023-11-28T17:09:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.299+0000 7f544b7fe640 1 -- 172.21.15.116:0/877586322 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f5454007bd0 con 0x7f546406c770 2023-11-28T17:09:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.300+0000 7f544b7fe640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f545006b2a0 0x7f545006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.300+0000 7f544b7fe640 1 -- 172.21.15.116:0/877586322 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5454089b70 con 0x7f546406c770 2023-11-28T17:09:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.300+0000 7f54627fc640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f545006b2a0 0x7f545006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:42.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.301+0000 7f54627fc640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f545006b2a0 0x7f545006d760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5458036000 tx=0x7f545803c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:42.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.304+0000 7f544b7fe640 1 -- 172.21.15.116:0/877586322 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f54540549c0 con 0x7f546406c770 2023-11-28T17:09:42.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.495+0000 7f54697b0640 1 -- 172.21.15.116:0/877586322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f546415ce90 con 0x7f546406c770 2023-11-28T17:09:42.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.503+0000 7f544b7fe640 1 -- 172.21.15.116:0/877586322 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f545401e020 con 0x7f546406c770 2023-11-28T17:09:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.506+0000 7f54697b0640 1 -- 172.21.15.116:0/877586322 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f545006b2a0 msgr2=0x7f545006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.506+0000 7f54697b0640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f545006b2a0 0x7f545006d760 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5458036000 tx=0x7f545803c040 comp rx=0 tx=0).stop 2023-11-28T17:09:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.506+0000 7f54697b0640 1 -- 172.21.15.116:0/877586322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546406c770 msgr2=0x7f5464158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.506+0000 7f54697b0640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546406c770 0x7f5464158130 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f545400c7f0 tx=0x7f545400ccb0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.506+0000 7f54697b0640 1 -- 172.21.15.116:0/877586322 shutdown_connections 2023-11-28T17:09:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.506+0000 7f54697b0640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f545006b2a0 0x7f545006d760 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.506+0000 7f54697b0640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f546415bab0 0x7f5464152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.506+0000 7f54697b0640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54640cd0c0 0x7f54641525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.506+0000 7f54697b0640 1 --2- 172.21.15.116:0/877586322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f546406c770 0x7f5464158130 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.506+0000 7f54697b0640 1 -- 172.21.15.116:0/877586322 >> 172.21.15.116:0/877586322 conn(0x7f546406a4c0 msgr2=0x7f546415c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.506+0000 7f54697b0640 1 -- 172.21.15.116:0/877586322 shutdown_connections 2023-11-28T17:09:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:42.506+0000 7f54697b0640 1 -- 172.21.15.116:0/877586322 wait complete. 2023-11-28T17:09:42.525 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T17:09:42.526 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T17:09:42.636 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T17:09:42.765 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T17:09:42.765 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:09:42.765 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:09:42.831 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:09:42.831 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:09:42.882 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:43.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.086+0000 7f7e560ec640 1 Processor -- start 2023-11-28T17:09:43.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.087+0000 7f7e560ec640 1 -- start start 2023-11-28T17:09:43.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.088+0000 7f7e560ec640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7e5015aba0 0x7f7e50164e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.088+0000 7f7e560ec640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e500aea20 con 0x7f7e5006c960 2023-11-28T17:09:43.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.088+0000 7f7e560ec640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5013d2c0 con 0x7f7e5013cee0 2023-11-28T17:09:43.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.088+0000 7f7e560ec640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5015b3a0 con 0x7f7e5015aba0 2023-11-28T17:09:43.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.089+0000 7f7e46ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7e5006c960 0x7f7e5015a490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41290/0 (socket says 172.21.15.116:41290) 2023-11-28T17:09:43.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.089+0000 7f7e46ffd640 1 -- 172.21.15.116:0/2322785075 learned_addr learned my addr 172.21.15.116:0/2322785075 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:43.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.089+0000 7f7e4ffff640 1 --2- 172.21.15.116:0/2322785075 >> v2:172.21.15.116:3300/0 conn(0x7f7e5015aba0 0x7f7e50164e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:43.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.089+0000 7f7e4effd640 1 -- 172.21.15.116:0/2322785075 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954067434 0 0) 0x7f7e500aea20 con 0x7f7e5006c960 2023-11-28T17:09:43.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.089+0000 7f7e4effd640 1 -- 172.21.15.116:0/2322785075 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e3c003520 con 0x7f7e5006c960 2023-11-28T17:09:43.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.089+0000 7f7e4ffff640 1 -- 172.21.15.116:0/2322785075 >> v1:172.21.15.156:6789/0 conn(0x7f7e5013cee0 legacy=0x7f7e5006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:43.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.089+0000 7f7e4ffff640 1 -- 172.21.15.116:0/2322785075 >> v1:172.21.15.116:6789/0 conn(0x7f7e5006c960 legacy=0x7f7e5015a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.089+0000 7f7e4ffff640 1 -- 172.21.15.116:0/2322785075 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5015b0c0 con 0x7f7e5015aba0 2023-11-28T17:09:43.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.090+0000 7f7e4ffff640 1 --2- 172.21.15.116:0/2322785075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5015aba0 0x7f7e50164e10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7e480097a0 tx=0x7f7e4802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee90f8d5256a802 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:43.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.090+0000 7f7e4effd640 1 -- 172.21.15.116:0/2322785075 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4802e070 con 0x7f7e5015aba0 2023-11-28T17:09:43.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.091+0000 7f7e4effd640 1 -- 172.21.15.116:0/2322785075 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e480315a0 con 0x7f7e5015aba0 2023-11-28T17:09:43.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.091+0000 7f7e4effd640 1 -- 172.21.15.116:0/2322785075 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48031930 con 0x7f7e5015aba0 2023-11-28T17:09:43.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.091+0000 7f7e560ec640 1 -- 172.21.15.116:0/2322785075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5015aba0 msgr2=0x7f7e50164e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.091+0000 7f7e560ec640 1 --2- 172.21.15.116:0/2322785075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5015aba0 0x7f7e50164e10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7e480097a0 tx=0x7f7e4802b840 comp rx=0 tx=0).stop 2023-11-28T17:09:43.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.091+0000 7f7e560ec640 1 -- 172.21.15.116:0/2322785075 shutdown_connections 2023-11-28T17:09:43.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.091+0000 7f7e560ec640 1 --2- 172.21.15.116:0/2322785075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5015aba0 0x7f7e50164e10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.091+0000 7f7e560ec640 1 -- 172.21.15.116:0/2322785075 >> 172.21.15.116:0/2322785075 conn(0x7f7e5006a4c0 msgr2=0x7f7e5006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:43.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.091+0000 7f7e560ec640 1 -- 172.21.15.116:0/2322785075 shutdown_connections 2023-11-28T17:09:43.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.091+0000 7f7e560ec640 1 -- 172.21.15.116:0/2322785075 wait complete. 2023-11-28T17:09:43.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.093+0000 7f7e560ec640 1 Processor -- start 2023-11-28T17:09:43.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.093+0000 7f7e560ec640 1 -- start start 2023-11-28T17:09:43.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.094+0000 7f7e560ec640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e5006c960 0x7f7e501516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.094+0000 7f7e560ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5013cee0 0x7f7e50135eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.094+0000 7f7e560ec640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e5015aba0 0x7f7e5013e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.094+0000 7f7e560ec640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500af9a0 con 0x7f7e5013cee0 2023-11-28T17:09:43.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.094+0000 7f7e560ec640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500aed90 con 0x7f7e5015aba0 2023-11-28T17:09:43.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.094+0000 7f7e560ec640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500af070 con 0x7f7e5006c960 2023-11-28T17:09:43.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.094+0000 7f7e46ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5013cee0 0x7f7e50135eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:43.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.094+0000 7f7e46ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5013cee0 0x7f7e50135eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59934/0 (socket says 172.21.15.116:59934) 2023-11-28T17:09:43.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.094+0000 7f7e46ffd640 1 -- 172.21.15.116:0/2538961154 learned_addr learned my addr 172.21.15.116:0/2538961154 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:43.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.095+0000 7f7e4f7fe640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e5006c960 0x7f7e501516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:43.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.095+0000 7f7e46ffd640 1 -- 172.21.15.116:0/2538961154 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e5006c960 msgr2=0x7f7e501516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.095+0000 7f7e4ffff640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e5015aba0 0x7f7e5013e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:43.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.095+0000 7f7e46ffd640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e5006c960 0x7f7e501516f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.095+0000 7f7e46ffd640 1 -- 172.21.15.116:0/2538961154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e5015aba0 msgr2=0x7f7e5013e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.095+0000 7f7e46ffd640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e5015aba0 0x7f7e5013e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.095+0000 7f7e46ffd640 1 -- 172.21.15.116:0/2538961154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e48009400 con 0x7f7e5013cee0 2023-11-28T17:09:43.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.095+0000 7f7e4f7fe640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e5006c960 0x7f7e501516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:43.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.095+0000 7f7e4ffff640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e5015aba0 0x7f7e5013e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:43.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.095+0000 7f7e46ffd640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5013cee0 0x7f7e50135eb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7e3800c500 tx=0x7f7e3800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:43.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.095+0000 7f7e4cff9640 1 -- 172.21.15.116:0/2538961154 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38005ce0 con 0x7f7e5013cee0 2023-11-28T17:09:43.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.095+0000 7f7e4cff9640 1 -- 172.21.15.116:0/2538961154 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e38005e40 con 0x7f7e5013cee0 2023-11-28T17:09:43.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.095+0000 7f7e560ec640 1 -- 172.21.15.116:0/2538961154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e500af350 con 0x7f7e5013cee0 2023-11-28T17:09:43.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.095+0000 7f7e4cff9640 1 -- 172.21.15.116:0/2538961154 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38007330 con 0x7f7e5013cee0 2023-11-28T17:09:43.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.096+0000 7f7e560ec640 1 -- 172.21.15.116:0/2538961154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e50151c30 con 0x7f7e5013cee0 2023-11-28T17:09:43.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.097+0000 7f7e560ec640 1 -- 172.21.15.116:0/2538961154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e500693d0 con 0x7f7e5013cee0 2023-11-28T17:09:43.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.098+0000 7f7e4cff9640 1 -- 172.21.15.116:0/2538961154 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f7e38007550 con 0x7f7e5013cee0 2023-11-28T17:09:43.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.098+0000 7f7e4cff9640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7e3006b2a0 0x7f7e3006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.098+0000 7f7e4cff9640 1 -- 172.21.15.116:0/2538961154 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7e38089260 con 0x7f7e5013cee0 2023-11-28T17:09:43.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.099+0000 7f7e4f7fe640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7e3006b2a0 0x7f7e3006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:43.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.099+0000 7f7e4f7fe640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7e3006b2a0 0x7f7e3006d760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7e40004520 tx=0x7f7e40009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:43.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.103+0000 7f7e4cff9640 1 -- 172.21.15.116:0/2538961154 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e38054160 con 0x7f7e5013cee0 2023-11-28T17:09:43.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.295+0000 7f7e560ec640 1 -- 172.21.15.116:0/2538961154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e50151e50 con 0x7f7e5013cee0 2023-11-28T17:09:43.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.298+0000 7f7e4cff9640 1 -- 172.21.15.116:0/2538961154 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f7e38057780 con 0x7f7e5013cee0 2023-11-28T17:09:43.300 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:43.300 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:43.049465+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:43.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.301+0000 7f7e560ec640 1 -- 172.21.15.116:0/2538961154 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7e3006b2a0 msgr2=0x7f7e3006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.301+0000 7f7e560ec640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7e3006b2a0 0x7f7e3006d760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7e40004520 tx=0x7f7e40009040 comp rx=0 tx=0).stop 2023-11-28T17:09:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.301+0000 7f7e560ec640 1 -- 172.21.15.116:0/2538961154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5013cee0 msgr2=0x7f7e50135eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.301+0000 7f7e560ec640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5013cee0 0x7f7e50135eb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7e3800c500 tx=0x7f7e3800c9c0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.301+0000 7f7e560ec640 1 -- 172.21.15.116:0/2538961154 shutdown_connections 2023-11-28T17:09:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.301+0000 7f7e560ec640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7e3006b2a0 0x7f7e3006d760 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.301+0000 7f7e560ec640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e5015aba0 0x7f7e5013e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.301+0000 7f7e560ec640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e5013cee0 0x7f7e50135eb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.301+0000 7f7e560ec640 1 --2- 172.21.15.116:0/2538961154 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e5006c960 0x7f7e501516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.301+0000 7f7e560ec640 1 -- 172.21.15.116:0/2538961154 >> 172.21.15.116:0/2538961154 conn(0x7f7e5006a4c0 msgr2=0x7f7e5015bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.301+0000 7f7e560ec640 1 -- 172.21.15.116:0/2538961154 shutdown_connections 2023-11-28T17:09:43.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.301+0000 7f7e560ec640 1 -- 172.21.15.116:0/2538961154 wait complete. 2023-11-28T17:09:43.320 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:09:43.321 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:09:43.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.531+0000 7f511327e640 1 Processor -- start 2023-11-28T17:09:43.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.532+0000 7f511327e640 1 -- start start 2023-11-28T17:09:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.533+0000 7f511327e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f510c152050 0x7f510c15bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.533+0000 7f511327e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f510c0b2fe0 0x7f510c0ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.533+0000 7f511327e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f510c0b2b50 con 0x7f510c0abd30 2023-11-28T17:09:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.533+0000 7f511327e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f510c15c520 con 0x7f510c152050 2023-11-28T17:09:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.533+0000 7f511327e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f510c0ac2d0 con 0x7f510c0b2fe0 2023-11-28T17:09:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.533+0000 7f511227c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f510c152050 0x7f510c15bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.533+0000 7f511227c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f510c152050 0x7f510c15bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59936/0 (socket says 172.21.15.116:59936) 2023-11-28T17:09:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.533+0000 7f511227c640 1 -- 172.21.15.116:0/2078484995 learned_addr learned my addr 172.21.15.116:0/2078484995 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.534+0000 7f511227c640 1 -- 172.21.15.116:0/2078484995 >> v2:172.21.15.156:3300/0 conn(0x7f510c0b2fe0 msgr2=0x7f510c0ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:43.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.534+0000 7f511227c640 1 --2- 172.21.15.116:0/2078484995 >> v2:172.21.15.156:3300/0 conn(0x7f510c0b2fe0 0x7f510c0ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.534+0000 7f511227c640 1 -- 172.21.15.116:0/2078484995 >> v1:172.21.15.116:6789/0 conn(0x7f510c0abd30 legacy=0x7f510c0cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.534+0000 7f511227c640 1 -- 172.21.15.116:0/2078484995 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510c0b2220 con 0x7f510c152050 2023-11-28T17:09:43.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.534+0000 7f511127a640 1 -- 172.21.15.116:0/2078484995 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045803322 0 0) 0x7f510c0b2b50 con 0x7f510c0abd30 2023-11-28T17:09:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.535+0000 7f511227c640 1 --2- 172.21.15.116:0/2078484995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f510c152050 0x7f510c15bfe0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f510c06e200 tx=0x7f510802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d728e6801fc98e30 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.535+0000 7f511127a640 1 -- 172.21.15.116:0/2078484995 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108030860 con 0x7f510c152050 2023-11-28T17:09:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.535+0000 7f511127a640 1 -- 172.21.15.116:0/2078484995 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51080309c0 con 0x7f510c152050 2023-11-28T17:09:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.535+0000 7f511127a640 1 -- 172.21.15.116:0/2078484995 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108030d50 con 0x7f510c152050 2023-11-28T17:09:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.535+0000 7f511327e640 1 -- 172.21.15.116:0/2078484995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f510c152050 msgr2=0x7f510c15bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.535+0000 7f511327e640 1 --2- 172.21.15.116:0/2078484995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f510c152050 0x7f510c15bfe0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f510c06e200 tx=0x7f510802b9a0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.536+0000 7f511327e640 1 -- 172.21.15.116:0/2078484995 shutdown_connections 2023-11-28T17:09:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.536+0000 7f511327e640 1 --2- 172.21.15.116:0/2078484995 >> v2:172.21.15.156:3300/0 conn(0x7f510c0b2fe0 0x7f510c0ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.536+0000 7f511327e640 1 --2- 172.21.15.116:0/2078484995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f510c152050 0x7f510c15bfe0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.536+0000 7f511327e640 1 -- 172.21.15.116:0/2078484995 >> 172.21.15.116:0/2078484995 conn(0x7f510c0b0950 msgr2=0x7f510c0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:43.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.536+0000 7f511327e640 1 -- 172.21.15.116:0/2078484995 shutdown_connections 2023-11-28T17:09:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.536+0000 7f511327e640 1 -- 172.21.15.116:0/2078484995 wait complete. 2023-11-28T17:09:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.537+0000 7f511327e640 1 Processor -- start 2023-11-28T17:09:43.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.538+0000 7f511327e640 1 -- start start 2023-11-28T17:09:43.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.538+0000 7f511327e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f510c0abd30 0x7f510c14c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.538+0000 7f511327e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f510c0b2fe0 0x7f510c1475c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.538+0000 7f511327e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f510c152050 0x7f510c1f61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.538+0000 7f511327e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510c147b00 con 0x7f510c0b2fe0 2023-11-28T17:09:43.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.538+0000 7f511327e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510c070520 con 0x7f510c0abd30 2023-11-28T17:09:43.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.538+0000 7f511327e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f510c0701f0 con 0x7f510c152050 2023-11-28T17:09:43.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.539+0000 7f5111a7b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f510c0b2fe0 0x7f510c1475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:43.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.539+0000 7f5111a7b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f510c0b2fe0 0x7f510c1475c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59938/0 (socket says 172.21.15.116:59938) 2023-11-28T17:09:43.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.539+0000 7f5111a7b640 1 -- 172.21.15.116:0/2126947683 learned_addr learned my addr 172.21.15.116:0/2126947683 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:43.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.539+0000 7f511227c640 1 --2- 172.21.15.116:0/2126947683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f510c0abd30 0x7f510c14c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:43.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.539+0000 7f5111a7b640 1 -- 172.21.15.116:0/2126947683 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f510c152050 msgr2=0x7f510c1f61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.539+0000 7f5111a7b640 1 --2- 172.21.15.116:0/2126947683 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f510c152050 0x7f510c1f61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.539+0000 7f5111a7b640 1 -- 172.21.15.116:0/2126947683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f510c0abd30 msgr2=0x7f510c14c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.539+0000 7f5111a7b640 1 --2- 172.21.15.116:0/2126947683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f510c0abd30 0x7f510c14c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.539+0000 7f5111a7b640 1 -- 172.21.15.116:0/2126947683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51080094d0 con 0x7f510c0b2fe0 2023-11-28T17:09:43.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.539+0000 7f5111a7b640 1 --2- 172.21.15.116:0/2126947683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f510c0b2fe0 0x7f510c1475c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f50fc00b4d0 tx=0x7f50fc00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:43.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.540+0000 7f5102ffd640 1 -- 172.21.15.116:0/2126947683 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50fc010870 con 0x7f510c0b2fe0 2023-11-28T17:09:43.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.540+0000 7f5102ffd640 1 -- 172.21.15.116:0/2126947683 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50fc0109d0 con 0x7f510c0b2fe0 2023-11-28T17:09:43.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.540+0000 7f5102ffd640 1 -- 172.21.15.116:0/2126947683 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50fc010d80 con 0x7f510c0b2fe0 2023-11-28T17:09:43.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.540+0000 7f511327e640 1 -- 172.21.15.116:0/2126947683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f510c1f6a30 con 0x7f510c0b2fe0 2023-11-28T17:09:43.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.540+0000 7f511327e640 1 -- 172.21.15.116:0/2126947683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f510c1f6ff0 con 0x7f510c0b2fe0 2023-11-28T17:09:43.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.541+0000 7f511327e640 1 -- 172.21.15.116:0/2126947683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510c09d0b0 con 0x7f510c0b2fe0 2023-11-28T17:09:43.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.542+0000 7f5102ffd640 1 -- 172.21.15.116:0/2126947683 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f50fc00f3c0 con 0x7f510c0b2fe0 2023-11-28T17:09:43.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.543+0000 7f5102ffd640 1 --2- 172.21.15.116:0/2126947683 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f50f006b2a0 0x7f50f006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.543+0000 7f5102ffd640 1 -- 172.21.15.116:0/2126947683 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f50fc089bf0 con 0x7f510c0b2fe0 2023-11-28T17:09:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.543+0000 7f511227c640 1 --2- 172.21.15.116:0/2126947683 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f50f006b2a0 0x7f50f006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:43.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.544+0000 7f511227c640 1 --2- 172.21.15.116:0/2126947683 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f50f006b2a0 0x7f50f006d760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5108036000 tx=0x7f510803b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:43.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.548+0000 7f5102ffd640 1 -- 172.21.15.116:0/2126947683 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50fc0549a0 con 0x7f510c0b2fe0 2023-11-28T17:09:43.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.739+0000 7f511327e640 1 -- 172.21.15.116:0/2126947683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f510c04e960 con 0x7f510c0b2fe0 2023-11-28T17:09:43.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.740+0000 7f5102ffd640 1 -- 172.21.15.116:0/2126947683 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f50fc01e020 con 0x7f510c0b2fe0 2023-11-28T17:09:43.741 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:09:43.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.742+0000 7f511327e640 1 -- 172.21.15.116:0/2126947683 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f50f006b2a0 msgr2=0x7f50f006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.742+0000 7f511327e640 1 --2- 172.21.15.116:0/2126947683 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f50f006b2a0 0x7f50f006d760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5108036000 tx=0x7f510803b040 comp rx=0 tx=0).stop 2023-11-28T17:09:43.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.742+0000 7f511327e640 1 -- 172.21.15.116:0/2126947683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f510c0b2fe0 msgr2=0x7f510c1475c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.742+0000 7f511327e640 1 --2- 172.21.15.116:0/2126947683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f510c0b2fe0 0x7f510c1475c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f50fc00b4d0 tx=0x7f50fc00b990 comp rx=0 tx=0).stop 2023-11-28T17:09:43.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.743+0000 7f511327e640 1 -- 172.21.15.116:0/2126947683 shutdown_connections 2023-11-28T17:09:43.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.743+0000 7f511327e640 1 --2- 172.21.15.116:0/2126947683 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f50f006b2a0 0x7f50f006d760 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.743+0000 7f511327e640 1 --2- 172.21.15.116:0/2126947683 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f510c152050 0x7f510c1f61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.743+0000 7f511327e640 1 --2- 172.21.15.116:0/2126947683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f510c0b2fe0 0x7f510c1475c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.743+0000 7f511327e640 1 --2- 172.21.15.116:0/2126947683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f510c0abd30 0x7f510c14c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.743+0000 7f511327e640 1 -- 172.21.15.116:0/2126947683 >> 172.21.15.116:0/2126947683 conn(0x7f510c0b0950 msgr2=0x7f510c150310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:43.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.743+0000 7f511327e640 1 -- 172.21.15.116:0/2126947683 shutdown_connections 2023-11-28T17:09:43.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.743+0000 7f511327e640 1 -- 172.21.15.116:0/2126947683 wait complete. 2023-11-28T17:09:43.763 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:09:43.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.974+0000 7fb01f91d640 1 Processor -- start 2023-11-28T17:09:43.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.974+0000 7fb01f91d640 1 -- start start 2023-11-28T17:09:43.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.975+0000 7fb01f91d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb01815a110 0x7fb01815e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.975+0000 7fb01f91d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb018071100 0x7fb0180b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.975+0000 7fb01f91d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0180b3660 con 0x7fb0180b0b10 2023-11-28T17:09:43.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.975+0000 7fb01f91d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb018071e60 con 0x7fb018071100 2023-11-28T17:09:43.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.975+0000 7fb01f91d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0180718a0 con 0x7fb01815a110 2023-11-28T17:09:43.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.976+0000 7fb01ceb8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb018071100 0x7fb0180b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:43.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.976+0000 7fb01ceb8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb018071100 0x7fb0180b3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59954/0 (socket says 172.21.15.116:59954) 2023-11-28T17:09:43.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.976+0000 7fb01ceb8640 1 -- 172.21.15.116:0/4079712445 learned_addr learned my addr 172.21.15.116:0/4079712445 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:43.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.976+0000 7fb01d6b9640 1 --2- 172.21.15.116:0/4079712445 >> v2:172.21.15.156:3300/0 conn(0x7fb01815a110 0x7fb01815e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:43.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.976+0000 7fb01ceb8640 1 -- 172.21.15.116:0/4079712445 >> v2:172.21.15.156:3300/0 conn(0x7fb01815a110 msgr2=0x7fb01815e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:43.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.976+0000 7fb01ceb8640 1 --2- 172.21.15.116:0/4079712445 >> v2:172.21.15.156:3300/0 conn(0x7fb01815a110 0x7fb01815e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.976+0000 7fb01ceb8640 1 -- 172.21.15.116:0/4079712445 >> v1:172.21.15.156:6789/0 conn(0x7fb0180b0b10 legacy=0x7fb0180b0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:43.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.976+0000 7fb01ceb8640 1 -- 172.21.15.116:0/4079712445 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb018071b80 con 0x7fb018071100 2023-11-28T17:09:43.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.977+0000 7fb01ceb8640 1 --2- 172.21.15.116:0/4079712445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb018071100 0x7fb0180b3af0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb01806e5b0 tx=0x7fb00802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfdfdabc83a6c3a4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:43.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.978+0000 7fb00ffff640 1 -- 172.21.15.116:0/4079712445 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008031870 con 0x7fb018071100 2023-11-28T17:09:43.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.978+0000 7fb00ffff640 1 -- 172.21.15.116:0/4079712445 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0080319d0 con 0x7fb018071100 2023-11-28T17:09:43.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.978+0000 7fb00ffff640 1 -- 172.21.15.116:0/4079712445 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008031d60 con 0x7fb018071100 2023-11-28T17:09:43.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.978+0000 7fb01f91d640 1 -- 172.21.15.116:0/4079712445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb018071100 msgr2=0x7fb0180b3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.978+0000 7fb01f91d640 1 --2- 172.21.15.116:0/4079712445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb018071100 0x7fb0180b3af0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb01806e5b0 tx=0x7fb00802b910 comp rx=0 tx=0).stop 2023-11-28T17:09:43.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.979+0000 7fb01f91d640 1 -- 172.21.15.116:0/4079712445 shutdown_connections 2023-11-28T17:09:43.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.979+0000 7fb01f91d640 1 --2- 172.21.15.116:0/4079712445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb018071100 0x7fb0180b3af0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.979+0000 7fb01f91d640 1 --2- 172.21.15.116:0/4079712445 >> v2:172.21.15.156:3300/0 conn(0x7fb01815a110 0x7fb01815e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.979+0000 7fb01f91d640 1 -- 172.21.15.116:0/4079712445 >> 172.21.15.116:0/4079712445 conn(0x7fb01806a4c0 msgr2=0x7fb01806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:43.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.979+0000 7fb01f91d640 1 -- 172.21.15.116:0/4079712445 shutdown_connections 2023-11-28T17:09:43.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.979+0000 7fb01f91d640 1 -- 172.21.15.116:0/4079712445 wait complete. 2023-11-28T17:09:43.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.980+0000 7fb01f91d640 1 Processor -- start 2023-11-28T17:09:43.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.981+0000 7fb01f91d640 1 -- start start 2023-11-28T17:09:43.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.982+0000 7fb01f91d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb018071100 0x7fb01814f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.982+0000 7fb01f91d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0180b0b10 0x7fb01814abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.986 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.982+0000 7fb01f91d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb01815a110 0x7fb018151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.982+0000 7fb01f91d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180ce550 con 0x7fb0180b0b10 2023-11-28T17:09:43.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.982+0000 7fb01f91d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180ce830 con 0x7fb01815a110 2023-11-28T17:09:43.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.982+0000 7fb01f91d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180ceb10 con 0x7fb018071100 2023-11-28T17:09:43.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.982+0000 7fb01ceb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0180b0b10 0x7fb01814abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:43.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.982+0000 7fb01ceb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0180b0b10 0x7fb01814abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59962/0 (socket says 172.21.15.116:59962) 2023-11-28T17:09:43.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.982+0000 7fb01ceb8640 1 -- 172.21.15.116:0/1956734543 learned_addr learned my addr 172.21.15.116:0/1956734543 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:43.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.982+0000 7fb01ceb8640 1 -- 172.21.15.116:0/1956734543 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb018071100 msgr2=0x7fb01814f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.982+0000 7fb01ceb8640 1 --2- 172.21.15.116:0/1956734543 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb018071100 0x7fb01814f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.982+0000 7fb01ceb8640 1 -- 172.21.15.116:0/1956734543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb01815a110 msgr2=0x7fb018151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:43.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.982+0000 7fb01ceb8640 1 --2- 172.21.15.116:0/1956734543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb01815a110 0x7fb018151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:43.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.982+0000 7fb01ceb8640 1 -- 172.21.15.116:0/1956734543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0080094d0 con 0x7fb0180b0b10 2023-11-28T17:09:43.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.983+0000 7fb01ceb8640 1 --2- 172.21.15.116:0/1956734543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0180b0b10 0x7fb01814abb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb00802b950 tx=0x7fb0080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:43.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.983+0000 7fb00dffb640 1 -- 172.21.15.116:0/1956734543 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00803ad30 con 0x7fb0180b0b10 2023-11-28T17:09:43.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.983+0000 7fb00dffb640 1 -- 172.21.15.116:0/1956734543 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0080443f0 con 0x7fb0180b0b10 2023-11-28T17:09:43.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.983+0000 7fb01f91d640 1 -- 172.21.15.116:0/1956734543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0180cedf0 con 0x7fb0180b0b10 2023-11-28T17:09:43.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.983+0000 7fb00dffb640 1 -- 172.21.15.116:0/1956734543 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0080447a0 con 0x7fb0180b0b10 2023-11-28T17:09:43.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.983+0000 7fb01f91d640 1 -- 172.21.15.116:0/1956734543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01814b0f0 con 0x7fb0180b0b10 2023-11-28T17:09:43.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.984+0000 7fb01f91d640 1 -- 172.21.15.116:0/1956734543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0180cf000 con 0x7fb0180b0b10 2023-11-28T17:09:43.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.985+0000 7fb00dffb640 1 -- 172.21.15.116:0/1956734543 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb00804d450 con 0x7fb0180b0b10 2023-11-28T17:09:43.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.986+0000 7fb00dffb640 1 --2- 172.21.15.116:0/1956734543 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7faffc06b2a0 0x7faffc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:43.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.986+0000 7fb00dffb640 1 -- 172.21.15.116:0/1956734543 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb00802d040 con 0x7fb0180b0b10 2023-11-28T17:09:43.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.986+0000 7fb01d6b9640 1 --2- 172.21.15.116:0/1956734543 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7faffc06b2a0 0x7faffc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:43.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.987+0000 7fb01d6b9640 1 --2- 172.21.15.116:0/1956734543 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7faffc06b2a0 0x7faffc06d760 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb014006fd0 tx=0x7fb014008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:43.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:43.990+0000 7fb00dffb640 1 -- 172.21.15.116:0/1956734543 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0080dee90 con 0x7fb0180b0b10 2023-11-28T17:09:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.184+0000 7fb01f91d640 1 -- 172.21.15.116:0/1956734543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb0180ce690 con 0x7fb0180b0b10 2023-11-28T17:09:44.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.185+0000 7fb00dffb640 1 -- 172.21.15.116:0/1956734543 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb008035020 con 0x7fb0180b0b10 2023-11-28T17:09:44.186 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:09:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.187+0000 7fb01f91d640 1 -- 172.21.15.116:0/1956734543 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7faffc06b2a0 msgr2=0x7faffc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.187+0000 7fb01f91d640 1 --2- 172.21.15.116:0/1956734543 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7faffc06b2a0 0x7faffc06d760 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb014006fd0 tx=0x7fb014008040 comp rx=0 tx=0).stop 2023-11-28T17:09:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.187+0000 7fb01f91d640 1 -- 172.21.15.116:0/1956734543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0180b0b10 msgr2=0x7fb01814abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.187+0000 7fb01f91d640 1 --2- 172.21.15.116:0/1956734543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0180b0b10 0x7fb01814abb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb00802b950 tx=0x7fb0080046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.187+0000 7fb01f91d640 1 -- 172.21.15.116:0/1956734543 shutdown_connections 2023-11-28T17:09:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.187+0000 7fb01f91d640 1 --2- 172.21.15.116:0/1956734543 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7faffc06b2a0 0x7faffc06d760 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.187+0000 7fb01f91d640 1 --2- 172.21.15.116:0/1956734543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb01815a110 0x7fb018151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.187+0000 7fb01f91d640 1 --2- 172.21.15.116:0/1956734543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0180b0b10 0x7fb01814abb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.187+0000 7fb01f91d640 1 --2- 172.21.15.116:0/1956734543 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb018071100 0x7fb01814f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.187+0000 7fb01f91d640 1 -- 172.21.15.116:0/1956734543 >> 172.21.15.116:0/1956734543 conn(0x7fb01806a4c0 msgr2=0x7fb0180aea70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.187+0000 7fb01f91d640 1 -- 172.21.15.116:0/1956734543 shutdown_connections 2023-11-28T17:09:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.187+0000 7fb01f91d640 1 -- 172.21.15.116:0/1956734543 wait complete. 2023-11-28T17:09:44.208 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:09:44.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.426+0000 7f0ca4c51640 1 Processor -- start 2023-11-28T17:09:44.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.426+0000 7f0ca4c51640 1 -- start start 2023-11-28T17:09:44.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.427+0000 7f0ca4c51640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ca00d1780 0x7f0ca00d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.427+0000 7f0ca4c51640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0ca00b20a0 0x7f0ca00d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0ca4c51640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0ca00b2ad0 0x7f0ca00ce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0ca4c51640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca00cdcb0 con 0x7f0ca00b20a0 2023-11-28T17:09:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0ca4c51640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca00d20a0 con 0x7f0ca00b2ad0 2023-11-28T17:09:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0ca4c51640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00d39e0 con 0x7f0ca00d1780 2023-11-28T17:09:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0c9dd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0ca00b20a0 0x7f0ca00d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0c9dd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0ca00b20a0 0x7f0ca00d1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59976/0 (socket says 172.21.15.116:59976) 2023-11-28T17:09:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0c9dd9b640 1 -- 172.21.15.116:0/2494743632 learned_addr learned my addr 172.21.15.116:0/2494743632 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0c9e59c640 1 --2- 172.21.15.116:0/2494743632 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ca00d1780 0x7f0ca00d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0c9ed9d640 1 --2- 172.21.15.116:0/2494743632 >> v2:172.21.15.156:3300/0 conn(0x7f0ca00b2ad0 0x7f0ca00ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:44.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0c9dd9b640 1 -- 172.21.15.116:0/2494743632 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ca00d1780 msgr2=0x7f0ca00d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0c9dd9b640 1 --2- 172.21.15.116:0/2494743632 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ca00d1780 0x7f0ca00d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0c9dd9b640 1 -- 172.21.15.116:0/2494743632 >> v2:172.21.15.156:3300/0 conn(0x7f0ca00b2ad0 msgr2=0x7f0ca00ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0c9dd9b640 1 --2- 172.21.15.116:0/2494743632 >> v2:172.21.15.156:3300/0 conn(0x7f0ca00b2ad0 0x7f0ca00ce750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0c9dd9b640 1 -- 172.21.15.116:0/2494743632 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca00d3cc0 con 0x7f0ca00b20a0 2023-11-28T17:09:44.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.428+0000 7f0c9e59c640 1 --2- 172.21.15.116:0/2494743632 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ca00d1780 0x7f0ca00d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.430+0000 7f0c9dd9b640 1 --2- 172.21.15.116:0/2494743632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ca00b20a0 0x7f0ca00d1b60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0162190 tx=0x7f0c8802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8bdc73483cf43f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.430+0000 7f0c9d59a640 1 -- 172.21.15.116:0/2494743632 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880317e0 con 0x7f0ca00b20a0 2023-11-28T17:09:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.430+0000 7f0c9d59a640 1 -- 172.21.15.116:0/2494743632 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c88031940 con 0x7f0ca00b20a0 2023-11-28T17:09:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.430+0000 7f0c9d59a640 1 -- 172.21.15.116:0/2494743632 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88031cd0 con 0x7f0ca00b20a0 2023-11-28T17:09:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.430+0000 7f0ca4c51640 1 -- 172.21.15.116:0/2494743632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ca00b20a0 msgr2=0x7f0ca00d1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.430+0000 7f0ca4c51640 1 --2- 172.21.15.116:0/2494743632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ca00b20a0 0x7f0ca00d1b60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0162190 tx=0x7f0c8802b880 comp rx=0 tx=0).stop 2023-11-28T17:09:44.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.431+0000 7f0ca4c51640 1 -- 172.21.15.116:0/2494743632 shutdown_connections 2023-11-28T17:09:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.431+0000 7f0ca4c51640 1 --2- 172.21.15.116:0/2494743632 >> v2:172.21.15.156:3300/0 conn(0x7f0ca00b2ad0 0x7f0ca00ce750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.431+0000 7f0ca4c51640 1 --2- 172.21.15.116:0/2494743632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ca00b20a0 0x7f0ca00d1b60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.431+0000 7f0ca4c51640 1 --2- 172.21.15.116:0/2494743632 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ca00d1780 0x7f0ca00d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.431+0000 7f0ca4c51640 1 -- 172.21.15.116:0/2494743632 >> 172.21.15.116:0/2494743632 conn(0x7f0ca009c4e0 msgr2=0x7f0ca00d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.431+0000 7f0ca4c51640 1 -- 172.21.15.116:0/2494743632 shutdown_connections 2023-11-28T17:09:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.431+0000 7f0ca4c51640 1 -- 172.21.15.116:0/2494743632 wait complete. 2023-11-28T17:09:44.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.432+0000 7f0ca4c51640 1 Processor -- start 2023-11-28T17:09:44.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.433+0000 7f0ca4c51640 1 -- start start 2023-11-28T17:09:44.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.433+0000 7f0ca4c51640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ca00b20a0 0x7f0ca01f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:44.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0ca4c51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ca00b2ad0 0x7f0ca01599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:44.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0ca4c51640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ca00d1780 0x7f0ca00c8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:44.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0ca4c51640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00b74d0 con 0x7f0ca00b2ad0 2023-11-28T17:09:44.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0ca4c51640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00b77b0 con 0x7f0ca00d1780 2023-11-28T17:09:44.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0ca4c51640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00b7a90 con 0x7f0ca00b20a0 2023-11-28T17:09:44.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0c9dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ca00b2ad0 0x7f0ca01599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:44.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0c9dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ca00b2ad0 0x7f0ca01599c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59984/0 (socket says 172.21.15.116:59984) 2023-11-28T17:09:44.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0c9dd9b640 1 -- 172.21.15.116:0/1159015968 learned_addr learned my addr 172.21.15.116:0/1159015968 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:44.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0c9ed9d640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ca00d1780 0x7f0ca00c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:44.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0c9e59c640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ca00b20a0 0x7f0ca01f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:44.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0c9dd9b640 1 -- 172.21.15.116:0/1159015968 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ca00b20a0 msgr2=0x7f0ca01f2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:44.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0c9dd9b640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ca00b20a0 0x7f0ca01f2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0c9dd9b640 1 -- 172.21.15.116:0/1159015968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ca00d1780 msgr2=0x7f0ca00c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:44.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0c9dd9b640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ca00d1780 0x7f0ca00c8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0c9dd9b640 1 -- 172.21.15.116:0/1159015968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c880094d0 con 0x7f0ca00b2ad0 2023-11-28T17:09:44.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0c9ed9d640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ca00d1780 0x7f0ca00c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:44.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.434+0000 7f0c9e59c640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ca00b20a0 0x7f0ca01f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:44.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.435+0000 7f0c9dd9b640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ca00b2ad0 0x7f0ca01599c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0c88000c00 tx=0x7f0c880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:44.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.435+0000 7f0c96ffd640 1 -- 172.21.15.116:0/1159015968 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88032880 con 0x7f0ca00b2ad0 2023-11-28T17:09:44.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.435+0000 7f0c96ffd640 1 -- 172.21.15.116:0/1159015968 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c88032a20 con 0x7f0ca00b2ad0 2023-11-28T17:09:44.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.435+0000 7f0c96ffd640 1 -- 172.21.15.116:0/1159015968 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88032dd0 con 0x7f0ca00b2ad0 2023-11-28T17:09:44.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.435+0000 7f0ca4c51640 1 -- 172.21.15.116:0/1159015968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca00b7d70 con 0x7f0ca00b2ad0 2023-11-28T17:09:44.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.435+0000 7f0ca4c51640 1 -- 172.21.15.116:0/1159015968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca0159f00 con 0x7f0ca00b2ad0 2023-11-28T17:09:44.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.436+0000 7f0ca4c51640 1 -- 172.21.15.116:0/1159015968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca00b7fb0 con 0x7f0ca00b2ad0 2023-11-28T17:09:44.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.437+0000 7f0c96ffd640 1 -- 172.21.15.116:0/1159015968 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0c8804e450 con 0x7f0ca00b2ad0 2023-11-28T17:09:44.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.438+0000 7f0c96ffd640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0c8006b2d0 0x7f0c8006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:44.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.438+0000 7f0c96ffd640 1 -- 172.21.15.116:0/1159015968 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0c880abd10 con 0x7f0ca00b2ad0 2023-11-28T17:09:44.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.438+0000 7f0c9e59c640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0c8006b2d0 0x7f0c8006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:44.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.439+0000 7f0c9e59c640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0c8006b2d0 0x7f0c8006d790 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00cf1c0 tx=0x7f0c90007550 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:44.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.442+0000 7f0c96ffd640 1 -- 172.21.15.116:0/1159015968 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ca00b7fb0 con 0x7f0ca00b2ad0 2023-11-28T17:09:44.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.632+0000 7f0ca4c51640 1 -- 172.21.15.116:0/1159015968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0ca00b7fb0 con 0x7f0ca00b2ad0 2023-11-28T17:09:44.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.633+0000 7f0c96ffd640 1 -- 172.21.15.116:0/1159015968 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0ca00b7fb0 con 0x7f0ca00b2ad0 2023-11-28T17:09:44.634 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:09:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.635+0000 7f0ca4c51640 1 -- 172.21.15.116:0/1159015968 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0c8006b2d0 msgr2=0x7f0c8006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.635+0000 7f0ca4c51640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0c8006b2d0 0x7f0c8006d790 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00cf1c0 tx=0x7f0c90007550 comp rx=0 tx=0).stop 2023-11-28T17:09:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.635+0000 7f0ca4c51640 1 -- 172.21.15.116:0/1159015968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ca00b2ad0 msgr2=0x7f0ca01599c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.635+0000 7f0ca4c51640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ca00b2ad0 0x7f0ca01599c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0c88000c00 tx=0x7f0c880046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.635+0000 7f0ca4c51640 1 -- 172.21.15.116:0/1159015968 shutdown_connections 2023-11-28T17:09:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.635+0000 7f0ca4c51640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0c8006b2d0 0x7f0c8006d790 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.635+0000 7f0ca4c51640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ca00d1780 0x7f0ca00c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.635+0000 7f0ca4c51640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ca00b2ad0 0x7f0ca01599c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.635+0000 7f0ca4c51640 1 --2- 172.21.15.116:0/1159015968 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ca00b20a0 0x7f0ca01f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.636+0000 7f0ca4c51640 1 -- 172.21.15.116:0/1159015968 >> 172.21.15.116:0/1159015968 conn(0x7f0ca009c4e0 msgr2=0x7f0ca006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:44.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.636+0000 7f0ca4c51640 1 -- 172.21.15.116:0/1159015968 shutdown_connections 2023-11-28T17:09:44.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.636+0000 7f0ca4c51640 1 -- 172.21.15.116:0/1159015968 wait complete. 2023-11-28T17:09:44.656 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:09:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.857+0000 7f6964675640 1 Processor -- start 2023-11-28T17:09:44.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.857+0000 7f6964675640 1 -- start start 2023-11-28T17:09:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.859+0000 7f6964675640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f695c15d2c0 0x7f695c0cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.859+0000 7f6964675640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f695c0d2f40 0x7f695c0b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.859+0000 7f6964675640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695c0b6640 con 0x7f695c0cdf70 2023-11-28T17:09:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.859+0000 7f6964675640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695c0ce420 con 0x7f695c15d2c0 2023-11-28T17:09:44.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.859+0000 7f6964675640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0d3480 con 0x7f695c0d2f40 2023-11-28T17:09:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.859+0000 7f6962411640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f695c15d2c0 0x7f695c0cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.859+0000 7f6962c12640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f695c0d2f40 0x7f695c0b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.859+0000 7f6962c12640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f695c0d2f40 0x7f695c0b6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:47346/0 (socket says 172.21.15.116:47346) 2023-11-28T17:09:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.859+0000 7f6962411640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f695c15d2c0 0x7f695c0cda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:53218/0 (socket says 172.21.15.116:53218) 2023-11-28T17:09:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.860+0000 7f6962c12640 1 -- 172.21.15.116:0/2438626244 learned_addr learned my addr 172.21.15.116:0/2438626244 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:44.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.860+0000 7f6962c12640 1 -- 172.21.15.116:0/2438626244 >> v2:172.21.15.156:3300/0 conn(0x7f695c15d2c0 msgr2=0x7f695c0cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.860+0000 7f6962c12640 1 --2- 172.21.15.116:0/2438626244 >> v2:172.21.15.156:3300/0 conn(0x7f695c15d2c0 0x7f695c0cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.860+0000 7f6962c12640 1 -- 172.21.15.116:0/2438626244 >> v1:172.21.15.156:6789/0 conn(0x7f695c0cdf70 legacy=0x7f695c0d2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:44.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.860+0000 7f6962c12640 1 -- 172.21.15.116:0/2438626244 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c0d3760 con 0x7f695c0d2f40 2023-11-28T17:09:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.862+0000 7f6962c12640 1 --2- 172.21.15.116:0/2438626244 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f695c0d2f40 0x7f695c0b6ad0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6954009870 tx=0x7f695402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd5da664493aa80c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.862+0000 7f696140f640 1 -- 172.21.15.116:0/2438626244 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954030840 con 0x7f695c0d2f40 2023-11-28T17:09:44.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.862+0000 7f696140f640 1 -- 172.21.15.116:0/2438626244 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69540309a0 con 0x7f695c0d2f40 2023-11-28T17:09:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.862+0000 7f696140f640 1 -- 172.21.15.116:0/2438626244 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954030d30 con 0x7f695c0d2f40 2023-11-28T17:09:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.863+0000 7f6964675640 1 -- 172.21.15.116:0/2438626244 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f695c0d2f40 msgr2=0x7f695c0b6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:44.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.863+0000 7f6964675640 1 --2- 172.21.15.116:0/2438626244 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f695c0d2f40 0x7f695c0b6ad0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6954009870 tx=0x7f695402b980 comp rx=0 tx=0).stop 2023-11-28T17:09:44.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.863+0000 7f6964675640 1 -- 172.21.15.116:0/2438626244 shutdown_connections 2023-11-28T17:09:44.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.863+0000 7f6964675640 1 --2- 172.21.15.116:0/2438626244 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f695c0d2f40 0x7f695c0b6ad0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.863+0000 7f6964675640 1 --2- 172.21.15.116:0/2438626244 >> v2:172.21.15.156:3300/0 conn(0x7f695c15d2c0 0x7f695c0cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.863+0000 7f6964675640 1 -- 172.21.15.116:0/2438626244 >> 172.21.15.116:0/2438626244 conn(0x7f695c072010 msgr2=0x7f695c072420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:44.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.863+0000 7f6964675640 1 -- 172.21.15.116:0/2438626244 shutdown_connections 2023-11-28T17:09:44.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.863+0000 7f6964675640 1 -- 172.21.15.116:0/2438626244 wait complete. 2023-11-28T17:09:44.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.865+0000 7f6964675640 1 Processor -- start 2023-11-28T17:09:44.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.865+0000 7f6964675640 1 -- start start 2023-11-28T17:09:44.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.866+0000 7f6964675640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c0cdf70 0x7f695c0cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:44.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.866+0000 7f6964675640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f695c0d2f40 0x7f695c0c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:44.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.866+0000 7f6964675640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f695c15d2c0 0x7f695c15f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:44.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.866+0000 7f6964675640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0ca9c0 con 0x7f695c0cdf70 2023-11-28T17:09:44.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.866+0000 7f6964675640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0cb320 con 0x7f695c15d2c0 2023-11-28T17:09:44.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.866+0000 7f6964675640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0ca340 con 0x7f695c0d2f40 2023-11-28T17:09:44.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.866+0000 7f6962411640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c0cdf70 0x7f695c0cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:44.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.866+0000 7f6962411640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c0cdf70 0x7f695c0cade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59988/0 (socket says 172.21.15.116:59988) 2023-11-28T17:09:44.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.866+0000 7f6962411640 1 -- 172.21.15.116:0/2936305612 learned_addr learned my addr 172.21.15.116:0/2936305612 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:44.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.867+0000 7f6962c12640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f695c15d2c0 0x7f695c15f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:44.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.867+0000 7f6961c10640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f695c0d2f40 0x7f695c0c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:44.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.867+0000 7f6962411640 1 -- 172.21.15.116:0/2936305612 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f695c0d2f40 msgr2=0x7f695c0c6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:44.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.867+0000 7f6962411640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f695c0d2f40 0x7f695c0c6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.867+0000 7f6962411640 1 -- 172.21.15.116:0/2936305612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f695c15d2c0 msgr2=0x7f695c15f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:44.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.867+0000 7f6962411640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f695c15d2c0 0x7f695c15f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:44.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.867+0000 7f6962411640 1 -- 172.21.15.116:0/2936305612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69540094d0 con 0x7f695c0cdf70 2023-11-28T17:09:44.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.867+0000 7f6962c12640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f695c15d2c0 0x7f695c15f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:44.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.867+0000 7f6961c10640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f695c0d2f40 0x7f695c0c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:44.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.867+0000 7f6962411640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c0cdf70 0x7f695c0cade0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f695000efc0 tx=0x7f695000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:44.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.867+0000 7f695affd640 1 -- 172.21.15.116:0/2936305612 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6950009b80 con 0x7f695c0cdf70 2023-11-28T17:09:44.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.867+0000 7f695affd640 1 -- 172.21.15.116:0/2936305612 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695000f040 con 0x7f695c0cdf70 2023-11-28T17:09:44.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.867+0000 7f695affd640 1 -- 172.21.15.116:0/2936305612 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69500146a0 con 0x7f695c0cdf70 2023-11-28T17:09:44.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.868+0000 7f6964675640 1 -- 172.21.15.116:0/2936305612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c0c7530 con 0x7f695c0cdf70 2023-11-28T17:09:44.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.868+0000 7f6964675640 1 -- 172.21.15.116:0/2936305612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c0c7240 con 0x7f695c0cdf70 2023-11-28T17:09:44.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.869+0000 7f6964675640 1 -- 172.21.15.116:0/2936305612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695c0cac30 con 0x7f695c0cdf70 2023-11-28T17:09:44.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.870+0000 7f695affd640 1 -- 172.21.15.116:0/2936305612 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6950014800 con 0x7f695c0cdf70 2023-11-28T17:09:44.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.871+0000 7f695affd640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f694006b2a0 0x7f694006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:44.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.871+0000 7f695affd640 1 -- 172.21.15.116:0/2936305612 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f695008a730 con 0x7f695c0cdf70 2023-11-28T17:09:44.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.871+0000 7f6961c10640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f694006b2a0 0x7f694006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:44.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.872+0000 7f6961c10640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f694006b2a0 0x7f694006d760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f695c158f70 tx=0x7f6948009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:44.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:44.875+0000 7f695affd640 1 -- 172.21.15.116:0/2936305612 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6950055570 con 0x7f695c0cdf70 2023-11-28T17:09:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.078+0000 7f6964675640 1 -- 172.21.15.116:0/2936305612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f695c0c6b70 con 0x7f695c0cdf70 2023-11-28T17:09:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.078+0000 7f695affd640 1 -- 172.21.15.116:0/2936305612 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6950058b90 con 0x7f695c0cdf70 2023-11-28T17:09:45.080 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-28T17:09:45.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.081+0000 7f6964675640 1 -- 172.21.15.116:0/2936305612 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f694006b2a0 msgr2=0x7f694006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.081+0000 7f6964675640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f694006b2a0 0x7f694006d760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f695c158f70 tx=0x7f6948009040 comp rx=0 tx=0).stop 2023-11-28T17:09:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.081+0000 7f6964675640 1 -- 172.21.15.116:0/2936305612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c0cdf70 msgr2=0x7f695c0cade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.081+0000 7f6964675640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c0cdf70 0x7f695c0cade0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f695000efc0 tx=0x7f695000c490 comp rx=0 tx=0).stop 2023-11-28T17:09:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.081+0000 7f6964675640 1 -- 172.21.15.116:0/2936305612 shutdown_connections 2023-11-28T17:09:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.081+0000 7f6964675640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f694006b2a0 0x7f694006d760 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.081+0000 7f6964675640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f695c15d2c0 0x7f695c15f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.081+0000 7f6964675640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f695c0d2f40 0x7f695c0c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.081+0000 7f6964675640 1 --2- 172.21.15.116:0/2936305612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f695c0cdf70 0x7f695c0cade0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.081+0000 7f6964675640 1 -- 172.21.15.116:0/2936305612 >> 172.21.15.116:0/2936305612 conn(0x7f695c072010 msgr2=0x7f695c0b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.081+0000 7f6964675640 1 -- 172.21.15.116:0/2936305612 shutdown_connections 2023-11-28T17:09:45.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.081+0000 7f6964675640 1 -- 172.21.15.116:0/2936305612 wait complete. 2023-11-28T17:09:45.101 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:09:45.101 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:09:45.152 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:45.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.357+0000 7f1305541640 1 Processor -- start 2023-11-28T17:09:45.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.357+0000 7f1305541640 1 -- start start 2023-11-28T17:09:45.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.359+0000 7f1305541640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f130006c960 0x7f130015a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:45.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.359+0000 7f1305541640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13000aea20 con 0x7f130013cee0 2023-11-28T17:09:45.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.359+0000 7f1305541640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130013d2c0 con 0x7f1300140200 2023-11-28T17:09:45.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.359+0000 7f1305541640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130015b3a0 con 0x7f130006c960 2023-11-28T17:09:45.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.359+0000 7f12feffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f130013cee0 0x7f130006c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41306/0 (socket says 172.21.15.116:41306) 2023-11-28T17:09:45.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.359+0000 7f12feffd640 1 -- 172.21.15.116:0/1927578900 learned_addr learned my addr 172.21.15.116:0/1927578900 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:45.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 -- 172.21.15.116:0/1927578900 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244762862 0 0) 0x7f13000aea20 con 0x7f130013cee0 2023-11-28T17:09:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 -- 172.21.15.116:0/1927578900 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12ec003520 con 0x7f130013cee0 2023-11-28T17:09:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12f7fff640 1 --2- 172.21.15.116:0/1927578900 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f130006c960 0x7f130015a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 -- 172.21.15.116:0/1927578900 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2518394561 0 0) 0x7f12ec003520 con 0x7f130013cee0 2023-11-28T17:09:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 -- 172.21.15.116:0/1927578900 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f130006c960 msgr2=0x7f130015a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 --2- 172.21.15.116:0/1927578900 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f130006c960 0x7f130015a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 -- 172.21.15.116:0/1927578900 >> v1:172.21.15.156:6789/0 conn(0x7f1300140200 legacy=0x7f13000cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:45.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 -- 172.21.15.116:0/1927578900 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130015b0c0 con 0x7f130013cee0 2023-11-28T17:09:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 -- 172.21.15.116:0/1927578900 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f12f0001670 con 0x7f130013cee0 2023-11-28T17:09:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 -- 172.21.15.116:0/1927578900 >> v1:172.21.15.116:6789/0 conn(0x7f130013cee0 legacy=0x7f130006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 --2- 172.21.15.116:0/1927578900 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f130006c960 0x7f12ec0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 --2- 172.21.15.116:0/1927578900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12ec004020 0x7f12ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 --2- 172.21.15.116:0/1927578900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12ec004d00 0x7f12ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:45.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 -- 172.21.15.116:0/1927578900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130015b3a0 con 0x7f12ec004020 2023-11-28T17:09:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 -- 172.21.15.116:0/1927578900 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ec00bc60 con 0x7f12ec004d00 2023-11-28T17:09:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.360+0000 7f12fe7fc640 1 -- 172.21.15.116:0/1927578900 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12ec00be20 con 0x7f130006c960 2023-11-28T17:09:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.361+0000 7f12f7fff640 1 --2- 172.21.15.116:0/1927578900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12ec004020 0x7f12ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.361+0000 7f12feffd640 1 --2- 172.21.15.116:0/1927578900 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f130006c960 0x7f12ec0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.361+0000 7f12f7fff640 1 -- 172.21.15.116:0/1927578900 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f130006c960 msgr2=0x7f12ec0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:45.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.361+0000 7f12f7fff640 1 --2- 172.21.15.116:0/1927578900 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f130006c960 0x7f12ec0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.361+0000 7f12f7fff640 1 -- 172.21.15.116:0/1927578900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12ec004d00 msgr2=0x7f12ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:45.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.361+0000 7f12f7fff640 1 --2- 172.21.15.116:0/1927578900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12ec004d00 0x7f12ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.361+0000 7f12f7fff640 1 -- 172.21.15.116:0/1927578900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ec003520 con 0x7f12ec004020 2023-11-28T17:09:45.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.362+0000 7f12f7fff640 1 --2- 172.21.15.116:0/1927578900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12ec004020 0x7f12ec0047c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f12e800b5c0 tx=0x7f12ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef65284276db726b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:45.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.362+0000 7f12fe7fc640 1 -- 172.21.15.116:0/1927578900 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e802e730 con 0x7f12ec004020 2023-11-28T17:09:45.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.362+0000 7f12fe7fc640 1 -- 172.21.15.116:0/1927578900 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e8032070 con 0x7f12ec004020 2023-11-28T17:09:45.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.362+0000 7f12fe7fc640 1 -- 172.21.15.116:0/1927578900 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e802eab0 con 0x7f12ec004020 2023-11-28T17:09:45.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.363+0000 7f1305541640 1 -- 172.21.15.116:0/1927578900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12ec004020 msgr2=0x7f12ec0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:45.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.363+0000 7f1305541640 1 --2- 172.21.15.116:0/1927578900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12ec004020 0x7f12ec0047c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f12e800b5c0 tx=0x7f12ec0057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.363+0000 7f12ff7fe640 1 -- 172.21.15.116:0/1927578900 reap_dead start 2023-11-28T17:09:45.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.363+0000 7f1305541640 1 -- 172.21.15.116:0/1927578900 shutdown_connections 2023-11-28T17:09:45.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.363+0000 7f1305541640 1 -- 172.21.15.116:0/1927578900 >> 172.21.15.116:0/1927578900 conn(0x7f130006a4c0 msgr2=0x7f130006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:45.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.363+0000 7f1305541640 1 -- 172.21.15.116:0/1927578900 shutdown_connections 2023-11-28T17:09:45.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.363+0000 7f1305541640 1 -- 172.21.15.116:0/1927578900 wait complete. 2023-11-28T17:09:45.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.365+0000 7f1305541640 1 Processor -- start 2023-11-28T17:09:45.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.365+0000 7f1305541640 1 -- start start 2023-11-28T17:09:45.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.366+0000 7f1305541640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12ec004020 0x7f130013de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:45.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.366+0000 7f1305541640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1300071d60 0x7f130013b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:45.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.366+0000 7f1305541640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f130013bb90 0x7f130015b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:45.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.366+0000 7f1305541640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130013c010 con 0x7f130013bb90 2023-11-28T17:09:45.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.366+0000 7f1305541640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13001e4010 con 0x7f12ec004020 2023-11-28T17:09:45.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.366+0000 7f1305541640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13001e42f0 con 0x7f1300071d60 2023-11-28T17:09:45.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.366+0000 7f12ff7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f130013bb90 0x7f130015b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:45.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.366+0000 7f12ff7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f130013bb90 0x7f130015b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60012/0 (socket says 172.21.15.116:60012) 2023-11-28T17:09:45.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.366+0000 7f12ff7fe640 1 -- 172.21.15.116:0/552366731 learned_addr learned my addr 172.21.15.116:0/552366731 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:45.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.367+0000 7f12feffd640 1 --2- 172.21.15.116:0/552366731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12ec004020 0x7f130013de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:45.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.367+0000 7f12ff7fe640 1 -- 172.21.15.116:0/552366731 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1300071d60 msgr2=0x7f130013b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:45.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.367+0000 7f12ff7fe640 1 --2- 172.21.15.116:0/552366731 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1300071d60 0x7f130013b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.367+0000 7f12ff7fe640 1 -- 172.21.15.116:0/552366731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12ec004020 msgr2=0x7f130013de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:45.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.367+0000 7f12ff7fe640 1 --2- 172.21.15.116:0/552366731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12ec004020 0x7f130013de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.367+0000 7f12ff7fe640 1 -- 172.21.15.116:0/552366731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13001e45d0 con 0x7f130013bb90 2023-11-28T17:09:45.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.367+0000 7f12ff7fe640 1 --2- 172.21.15.116:0/552366731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f130013bb90 0x7f130015b4e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1300067340 tx=0x7f12f800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:45.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.367+0000 7f12f77fe640 1 -- 172.21.15.116:0/552366731 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f8005d40 con 0x7f130013bb90 2023-11-28T17:09:45.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.367+0000 7f12f77fe640 1 -- 172.21.15.116:0/552366731 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f800ced0 con 0x7f130013bb90 2023-11-28T17:09:45.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.367+0000 7f12f77fe640 1 -- 172.21.15.116:0/552366731 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f800a850 con 0x7f130013bb90 2023-11-28T17:09:45.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.367+0000 7f1305541640 1 -- 172.21.15.116:0/552366731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13001e48b0 con 0x7f130013bb90 2023-11-28T17:09:45.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.367+0000 7f1305541640 1 -- 172.21.15.116:0/552366731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130013e360 con 0x7f130013bb90 2023-11-28T17:09:45.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.368+0000 7f1305541640 1 -- 172.21.15.116:0/552366731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13000cdf40 con 0x7f130013bb90 2023-11-28T17:09:45.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.369+0000 7f12f77fe640 1 -- 172.21.15.116:0/552366731 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f12f800a9b0 con 0x7f130013bb90 2023-11-28T17:09:45.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.370+0000 7f12f77fe640 1 --2- 172.21.15.116:0/552366731 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f12e006b2a0 0x7f12e006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:45.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.370+0000 7f12f77fe640 1 -- 172.21.15.116:0/552366731 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f12f80889b0 con 0x7f130013bb90 2023-11-28T17:09:45.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.370+0000 7f12feffd640 1 --2- 172.21.15.116:0/552366731 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f12e006b2a0 0x7f12e006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:45.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.371+0000 7f12feffd640 1 --2- 172.21.15.116:0/552366731 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f12e006b2a0 0x7f12e006d760 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f12f0001eb0 tx=0x7f12f0000f50 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:45.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.374+0000 7f12f77fe640 1 -- 172.21.15.116:0/552366731 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12f8053760 con 0x7f130013bb90 2023-11-28T17:09:45.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.567+0000 7f1305541640 1 -- 172.21.15.116:0/552366731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f130013d2c0 con 0x7f130013bb90 2023-11-28T17:09:45.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.569+0000 7f12f77fe640 1 -- 172.21.15.116:0/552366731 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f130013d2c0 con 0x7f130013bb90 2023-11-28T17:09:45.570 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:45.570 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:43.049465+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:45.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.571+0000 7f1305541640 1 -- 172.21.15.116:0/552366731 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f12e006b2a0 msgr2=0x7f12e006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:45.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.571+0000 7f1305541640 1 --2- 172.21.15.116:0/552366731 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f12e006b2a0 0x7f12e006d760 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f12f0001eb0 tx=0x7f12f0000f50 comp rx=0 tx=0).stop 2023-11-28T17:09:45.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.571+0000 7f1305541640 1 -- 172.21.15.116:0/552366731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f130013bb90 msgr2=0x7f130015b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.571+0000 7f1305541640 1 --2- 172.21.15.116:0/552366731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f130013bb90 0x7f130015b4e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1300067340 tx=0x7f12f800cb90 comp rx=0 tx=0).stop 2023-11-28T17:09:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.571+0000 7f1305541640 1 -- 172.21.15.116:0/552366731 shutdown_connections 2023-11-28T17:09:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.571+0000 7f1305541640 1 --2- 172.21.15.116:0/552366731 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f12e006b2a0 0x7f12e006d760 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.571+0000 7f1305541640 1 --2- 172.21.15.116:0/552366731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f130013bb90 0x7f130015b4e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.571+0000 7f1305541640 1 --2- 172.21.15.116:0/552366731 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1300071d60 0x7f130013b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.571+0000 7f1305541640 1 --2- 172.21.15.116:0/552366731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f12ec004020 0x7f130013de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.571+0000 7f1305541640 1 -- 172.21.15.116:0/552366731 >> 172.21.15.116:0/552366731 conn(0x7f130006a4c0 msgr2=0x7f13000aecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:45.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.572+0000 7f1305541640 1 -- 172.21.15.116:0/552366731 shutdown_connections 2023-11-28T17:09:45.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.572+0000 7f1305541640 1 -- 172.21.15.116:0/552366731 wait complete. 2023-11-28T17:09:45.592 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:09:45.592 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:09:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.803+0000 7fe70d167640 1 Processor -- start 2023-11-28T17:09:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.804+0000 7fe70d167640 1 -- start start 2023-11-28T17:09:45.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.805+0000 7fe70d167640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe7080d1ca0 0x7fe7080b0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:45.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.805+0000 7fe70d167640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe7080ce6a0 0x7fe7080b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:45.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.805+0000 7fe70d167640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70815d860 con 0x7fe70815d480 2023-11-28T17:09:45.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.805+0000 7fe70d167640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7080b00d0 con 0x7fe7080ce6a0 2023-11-28T17:09:45.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.805+0000 7fe70d167640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7080cebd0 con 0x7fe7080d1ca0 2023-11-28T17:09:45.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.805+0000 7fe70c966640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe7080ce6a0 0x7fe7080b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:45.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.805+0000 7fe70c966640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe7080ce6a0 0x7fe7080b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60022/0 (socket says 172.21.15.116:60022) 2023-11-28T17:09:45.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.805+0000 7fe70c966640 1 -- 172.21.15.116:0/2007781201 learned_addr learned my addr 172.21.15.116:0/2007781201 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:45.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.806+0000 7fe7077fe640 1 --2- 172.21.15.116:0/2007781201 >> v2:172.21.15.156:3300/0 conn(0x7fe7080d1ca0 0x7fe7080b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.806+0000 7fe70c966640 1 -- 172.21.15.116:0/2007781201 >> v2:172.21.15.156:3300/0 conn(0x7fe7080d1ca0 msgr2=0x7fe7080b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.806+0000 7fe70c966640 1 --2- 172.21.15.116:0/2007781201 >> v2:172.21.15.156:3300/0 conn(0x7fe7080d1ca0 0x7fe7080b0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.806+0000 7fe70c966640 1 -- 172.21.15.116:0/2007781201 >> v1:172.21.15.156:6789/0 conn(0x7fe70815d480 legacy=0x7fe7080d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.806+0000 7fe70c966640 1 -- 172.21.15.116:0/2007781201 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7080ceeb0 con 0x7fe7080ce6a0 2023-11-28T17:09:45.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.807+0000 7fe70c966640 1 --2- 172.21.15.116:0/2007781201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7080ce6a0 0x7fe7080b0aa0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe7080d0b30 tx=0x7fe70002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe3b4927d39eeae server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:45.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.807+0000 7fe706ffd640 1 -- 172.21.15.116:0/2007781201 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70002c9f0 con 0x7fe7080ce6a0 2023-11-28T17:09:45.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.808+0000 7fe706ffd640 1 -- 172.21.15.116:0/2007781201 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70002cb50 con 0x7fe7080ce6a0 2023-11-28T17:09:45.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.808+0000 7fe706ffd640 1 -- 172.21.15.116:0/2007781201 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7000343f0 con 0x7fe7080ce6a0 2023-11-28T17:09:45.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.808+0000 7fe70d167640 1 -- 172.21.15.116:0/2007781201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7080ce6a0 msgr2=0x7fe7080b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:45.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.808+0000 7fe70d167640 1 --2- 172.21.15.116:0/2007781201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7080ce6a0 0x7fe7080b0aa0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe7080d0b30 tx=0x7fe70002be50 comp rx=0 tx=0).stop 2023-11-28T17:09:45.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.808+0000 7fe70d167640 1 -- 172.21.15.116:0/2007781201 shutdown_connections 2023-11-28T17:09:45.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.808+0000 7fe70d167640 1 --2- 172.21.15.116:0/2007781201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7080ce6a0 0x7fe7080b0aa0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.808+0000 7fe70d167640 1 --2- 172.21.15.116:0/2007781201 >> v2:172.21.15.156:3300/0 conn(0x7fe7080d1ca0 0x7fe7080b0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.808+0000 7fe70d167640 1 -- 172.21.15.116:0/2007781201 >> 172.21.15.116:0/2007781201 conn(0x7fe70809c2f0 msgr2=0x7fe7080d2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:45.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.808+0000 7fe70d167640 1 -- 172.21.15.116:0/2007781201 shutdown_connections 2023-11-28T17:09:45.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.808+0000 7fe70d167640 1 -- 172.21.15.116:0/2007781201 wait complete. 2023-11-28T17:09:45.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.810+0000 7fe70d167640 1 Processor -- start 2023-11-28T17:09:45.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.810+0000 7fe70d167640 1 -- start start 2023-11-28T17:09:45.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.811+0000 7fe70d167640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7080ce6a0 0x7fe7081f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:45.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.811+0000 7fe70d167640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7080d1ca0 0x7fe7080b7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:45.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.811+0000 7fe70d167640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe70815d480 0x7fe7080b7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:45.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.811+0000 7fe70d167640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7080cbfd0 con 0x7fe7080d1ca0 2023-11-28T17:09:45.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.811+0000 7fe70d167640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7081f3de0 con 0x7fe7080ce6a0 2023-11-28T17:09:45.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.811+0000 7fe70d167640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7080ca540 con 0x7fe70815d480 2023-11-28T17:09:45.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.811+0000 7fe7077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7080d1ca0 0x7fe7080b7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:45.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.811+0000 7fe7077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7080d1ca0 0x7fe7080b7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60032/0 (socket says 172.21.15.116:60032) 2023-11-28T17:09:45.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.811+0000 7fe7077fe640 1 -- 172.21.15.116:0/1648247655 learned_addr learned my addr 172.21.15.116:0/1648247655 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:45.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.812+0000 7fe70c966640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe70815d480 0x7fe7080b7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:45.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.812+0000 7fe707fff640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7080ce6a0 0x7fe7081f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:45.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.812+0000 7fe7077fe640 1 -- 172.21.15.116:0/1648247655 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe70815d480 msgr2=0x7fe7080b7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:45.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.812+0000 7fe7077fe640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe70815d480 0x7fe7080b7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.812+0000 7fe7077fe640 1 -- 172.21.15.116:0/1648247655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7080ce6a0 msgr2=0x7fe7081f38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:45.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.812+0000 7fe7077fe640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7080ce6a0 0x7fe7081f38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:45.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.812+0000 7fe7077fe640 1 -- 172.21.15.116:0/1648247655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe700009b30 con 0x7fe7080d1ca0 2023-11-28T17:09:45.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.812+0000 7fe707fff640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7080ce6a0 0x7fe7081f38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:45.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.812+0000 7fe7077fe640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7080d1ca0 0x7fe7080b7190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00d7d0 tx=0x7fe6fc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:45.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.813+0000 7fe704ff9640 1 -- 172.21.15.116:0/1648247655 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc004030 con 0x7fe7080d1ca0 2023-11-28T17:09:45.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.813+0000 7fe704ff9640 1 -- 172.21.15.116:0/1648247655 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6fc004190 con 0x7fe7080d1ca0 2023-11-28T17:09:45.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.813+0000 7fe704ff9640 1 -- 172.21.15.116:0/1648247655 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc00b6c0 con 0x7fe7080d1ca0 2023-11-28T17:09:45.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.813+0000 7fe70d167640 1 -- 172.21.15.116:0/1648247655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7080ca820 con 0x7fe7080d1ca0 2023-11-28T17:09:45.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.813+0000 7fe70d167640 1 -- 172.21.15.116:0/1648247655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7080cade0 con 0x7fe7080d1ca0 2023-11-28T17:09:45.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.814+0000 7fe70d167640 1 -- 172.21.15.116:0/1648247655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7080a0f60 con 0x7fe7080d1ca0 2023-11-28T17:09:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.815+0000 7fe704ff9640 1 -- 172.21.15.116:0/1648247655 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe6fc00b820 con 0x7fe7080d1ca0 2023-11-28T17:09:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.816+0000 7fe704ff9640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe6f406b2d0 0x7fe6f406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.816+0000 7fe704ff9640 1 -- 172.21.15.116:0/1648247655 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe6fc089e00 con 0x7fe7080d1ca0 2023-11-28T17:09:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.816+0000 7fe707fff640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe6f406b2d0 0x7fe6f406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.817+0000 7fe707fff640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe6f406b2d0 0x7fe6f406d790 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe7080b7f50 tx=0x7fe6f8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:45.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:45.820+0000 7fe704ff9640 1 -- 172.21.15.116:0/1648247655 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6fc054c50 con 0x7fe7080d1ca0 2023-11-28T17:09:46.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.014+0000 7fe70d167640 1 -- 172.21.15.116:0/1648247655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe70808c770 con 0x7fe7080d1ca0 2023-11-28T17:09:46.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.014+0000 7fe704ff9640 1 -- 172.21.15.116:0/1648247655 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe6fc058270 con 0x7fe7080d1ca0 2023-11-28T17:09:46.016 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:09:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.017+0000 7fe70d167640 1 -- 172.21.15.116:0/1648247655 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe6f406b2d0 msgr2=0x7fe6f406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.017+0000 7fe70d167640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe6f406b2d0 0x7fe6f406d790 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe7080b7f50 tx=0x7fe6f8008040 comp rx=0 tx=0).stop 2023-11-28T17:09:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.017+0000 7fe70d167640 1 -- 172.21.15.116:0/1648247655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7080d1ca0 msgr2=0x7fe7080b7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.017+0000 7fe70d167640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7080d1ca0 0x7fe7080b7190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc00d7d0 tx=0x7fe6fc00dc90 comp rx=0 tx=0).stop 2023-11-28T17:09:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.017+0000 7fe70d167640 1 -- 172.21.15.116:0/1648247655 shutdown_connections 2023-11-28T17:09:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.017+0000 7fe70d167640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe6f406b2d0 0x7fe6f406d790 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.017+0000 7fe70d167640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe70815d480 0x7fe7080b7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.017+0000 7fe70d167640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7080d1ca0 0x7fe7080b7190 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.017+0000 7fe70d167640 1 --2- 172.21.15.116:0/1648247655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7080ce6a0 0x7fe7081f38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.017+0000 7fe70d167640 1 -- 172.21.15.116:0/1648247655 >> 172.21.15.116:0/1648247655 conn(0x7fe70809c2f0 msgr2=0x7fe7080b0560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:46.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.018+0000 7fe70d167640 1 -- 172.21.15.116:0/1648247655 shutdown_connections 2023-11-28T17:09:46.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.018+0000 7fe70d167640 1 -- 172.21.15.116:0/1648247655 wait complete. 2023-11-28T17:09:46.036 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:09:46.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.247+0000 7fbdface0640 1 Processor -- start 2023-11-28T17:09:46.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.247+0000 7fbdface0640 1 -- start start 2023-11-28T17:09:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.249+0000 7fbdface0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbdf415b0c0 0x7fbdf415ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.249+0000 7fbdface0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbdf40ce940 0x7fbdf40cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.249+0000 7fbdface0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf40cd320 con 0x7fbdf4071b40 2023-11-28T17:09:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.249+0000 7fbdface0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf40ced80 con 0x7fbdf415b0c0 2023-11-28T17:09:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.249+0000 7fbdface0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf40cbee0 con 0x7fbdf40ce940 2023-11-28T17:09:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.249+0000 7fbdf8a7c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbdf415b0c0 0x7fbdf415ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:46.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.249+0000 7fbdf8a7c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbdf415b0c0 0x7fbdf415ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60036/0 (socket says 172.21.15.116:60036) 2023-11-28T17:09:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.249+0000 7fbdf8a7c640 1 -- 172.21.15.116:0/523138649 learned_addr learned my addr 172.21.15.116:0/523138649 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.250+0000 7fbdf3fff640 1 --2- 172.21.15.116:0/523138649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbdf40ce940 0x7fbdf40cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.250+0000 7fbdf8a7c640 1 -- 172.21.15.116:0/523138649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbdf40ce940 msgr2=0x7fbdf40cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.250+0000 7fbdf8a7c640 1 --2- 172.21.15.116:0/523138649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbdf40ce940 0x7fbdf40cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.250+0000 7fbdf8a7c640 1 -- 172.21.15.116:0/523138649 >> v1:172.21.15.156:6789/0 conn(0x7fbdf4071b40 legacy=0x7fbdf415b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.250+0000 7fbdf8a7c640 1 -- 172.21.15.116:0/523138649 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf40cc710 con 0x7fbdf415b0c0 2023-11-28T17:09:46.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.250+0000 7fbdf3fff640 1 --2- 172.21.15.116:0/523138649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbdf40ce940 0x7fbdf40cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:46.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.251+0000 7fbdf8a7c640 1 --2- 172.21.15.116:0/523138649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf415b0c0 0x7fbdf415ed10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009870 tx=0x7fbde802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2608dc290739492a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.251+0000 7fbdf37fe640 1 -- 172.21.15.116:0/523138649 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80317c0 con 0x7fbdf415b0c0 2023-11-28T17:09:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.251+0000 7fbdf37fe640 1 -- 172.21.15.116:0/523138649 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde8031920 con 0x7fbdf415b0c0 2023-11-28T17:09:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.251+0000 7fbdf37fe640 1 -- 172.21.15.116:0/523138649 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8031cb0 con 0x7fbdf415b0c0 2023-11-28T17:09:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.252+0000 7fbdface0640 1 -- 172.21.15.116:0/523138649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf415b0c0 msgr2=0x7fbdf415ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.252+0000 7fbdface0640 1 --2- 172.21.15.116:0/523138649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf415b0c0 0x7fbdf415ed10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009870 tx=0x7fbde802b860 comp rx=0 tx=0).stop 2023-11-28T17:09:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.252+0000 7fbdface0640 1 -- 172.21.15.116:0/523138649 shutdown_connections 2023-11-28T17:09:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.252+0000 7fbdface0640 1 --2- 172.21.15.116:0/523138649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbdf40ce940 0x7fbdf40cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.252+0000 7fbdface0640 1 --2- 172.21.15.116:0/523138649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf415b0c0 0x7fbdf415ed10 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.252+0000 7fbdface0640 1 -- 172.21.15.116:0/523138649 >> 172.21.15.116:0/523138649 conn(0x7fbdf40b0cf0 msgr2=0x7fbdf40b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.252+0000 7fbdface0640 1 -- 172.21.15.116:0/523138649 shutdown_connections 2023-11-28T17:09:46.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.252+0000 7fbdface0640 1 -- 172.21.15.116:0/523138649 wait complete. 2023-11-28T17:09:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.254+0000 7fbdface0640 1 Processor -- start 2023-11-28T17:09:46.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.254+0000 7fbdface0640 1 -- start start 2023-11-28T17:09:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.255+0000 7fbdface0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf4071b40 0x7fbdf414cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:46.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.255+0000 7fbdface0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbdf40ce940 0x7fbdf415df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.255+0000 7fbdface0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbdf415b0c0 0x7fbdf40afd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.255+0000 7fbdface0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf414d4e0 con 0x7fbdf4071b40 2023-11-28T17:09:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.255+0000 7fbdface0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf414d780 con 0x7fbdf40ce940 2023-11-28T17:09:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.255+0000 7fbdface0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf414da60 con 0x7fbdf415b0c0 2023-11-28T17:09:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.255+0000 7fbdf8a7c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf4071b40 0x7fbdf414cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.255+0000 7fbdf8a7c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf4071b40 0x7fbdf414cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60042/0 (socket says 172.21.15.116:60042) 2023-11-28T17:09:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.255+0000 7fbdf8a7c640 1 -- 172.21.15.116:0/3451973046 learned_addr learned my addr 172.21.15.116:0/3451973046 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.256+0000 7fbdf8a7c640 1 -- 172.21.15.116:0/3451973046 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbdf415b0c0 msgr2=0x7fbdf40afd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.256+0000 7fbdf8a7c640 1 --2- 172.21.15.116:0/3451973046 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbdf415b0c0 0x7fbdf40afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.256+0000 7fbdf8a7c640 1 -- 172.21.15.116:0/3451973046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbdf40ce940 msgr2=0x7fbdf415df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.256+0000 7fbdf8a7c640 1 --2- 172.21.15.116:0/3451973046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbdf40ce940 0x7fbdf415df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.256+0000 7fbdf8a7c640 1 -- 172.21.15.116:0/3451973046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde80094d0 con 0x7fbdf4071b40 2023-11-28T17:09:46.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.256+0000 7fbdf8a7c640 1 --2- 172.21.15.116:0/3451973046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf4071b40 0x7fbdf414cb90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbde802b8a0 tx=0x7fbde80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:46.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.256+0000 7fbdf17fa640 1 -- 172.21.15.116:0/3451973046 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80329a0 con 0x7fbdf4071b40 2023-11-28T17:09:46.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.256+0000 7fbdf17fa640 1 -- 172.21.15.116:0/3451973046 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde8032b20 con 0x7fbdf4071b40 2023-11-28T17:09:46.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.256+0000 7fbdf17fa640 1 -- 172.21.15.116:0/3451973046 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80453f0 con 0x7fbdf4071b40 2023-11-28T17:09:46.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.256+0000 7fbdface0640 1 -- 172.21.15.116:0/3451973046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf414dd40 con 0x7fbdf4071b40 2023-11-28T17:09:46.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.256+0000 7fbdface0640 1 -- 172.21.15.116:0/3451973046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf40c4120 con 0x7fbdf4071b40 2023-11-28T17:09:46.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.257+0000 7fbdface0640 1 -- 172.21.15.116:0/3451973046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf41581c0 con 0x7fbdf4071b40 2023-11-28T17:09:46.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.258+0000 7fbdf17fa640 1 -- 172.21.15.116:0/3451973046 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fbde804d3f0 con 0x7fbdf4071b40 2023-11-28T17:09:46.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.259+0000 7fbdf17fa640 1 --2- 172.21.15.116:0/3451973046 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbdd806b2a0 0x7fbdd806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:46.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.259+0000 7fbdf17fa640 1 -- 172.21.15.116:0/3451973046 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbde80abc30 con 0x7fbdf4071b40 2023-11-28T17:09:46.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.260+0000 7fbdf3fff640 1 --2- 172.21.15.116:0/3451973046 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbdd806b2a0 0x7fbdd806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:46.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.260+0000 7fbdf3fff640 1 --2- 172.21.15.116:0/3451973046 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbdd806b2a0 0x7fbdd806d760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbde0006fd0 tx=0x7fbde0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:46.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.264+0000 7fbdf17fa640 1 -- 172.21.15.116:0/3451973046 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbdf41581c0 con 0x7fbdf4071b40 2023-11-28T17:09:46.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.452+0000 7fbdface0640 1 -- 172.21.15.116:0/3451973046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbdf41581c0 con 0x7fbdf4071b40 2023-11-28T17:09:46.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.453+0000 7fbdf17fa640 1 -- 172.21.15.116:0/3451973046 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbdf41581c0 con 0x7fbdf4071b40 2023-11-28T17:09:46.454 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:09:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.455+0000 7fbdface0640 1 -- 172.21.15.116:0/3451973046 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbdd806b2a0 msgr2=0x7fbdd806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.455+0000 7fbdface0640 1 --2- 172.21.15.116:0/3451973046 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbdd806b2a0 0x7fbdd806d760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbde0006fd0 tx=0x7fbde0008040 comp rx=0 tx=0).stop 2023-11-28T17:09:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.455+0000 7fbdface0640 1 -- 172.21.15.116:0/3451973046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf4071b40 msgr2=0x7fbdf414cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:46.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.455+0000 7fbdface0640 1 --2- 172.21.15.116:0/3451973046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf4071b40 0x7fbdf414cb90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbde802b8a0 tx=0x7fbde80046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.455+0000 7fbdface0640 1 -- 172.21.15.116:0/3451973046 shutdown_connections 2023-11-28T17:09:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.455+0000 7fbdface0640 1 --2- 172.21.15.116:0/3451973046 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbdd806b2a0 0x7fbdd806d760 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.455+0000 7fbdface0640 1 --2- 172.21.15.116:0/3451973046 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbdf415b0c0 0x7fbdf40afd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.455+0000 7fbdface0640 1 --2- 172.21.15.116:0/3451973046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbdf40ce940 0x7fbdf415df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.455+0000 7fbdface0640 1 --2- 172.21.15.116:0/3451973046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdf4071b40 0x7fbdf414cb90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.455+0000 7fbdface0640 1 -- 172.21.15.116:0/3451973046 >> 172.21.15.116:0/3451973046 conn(0x7fbdf40b0cf0 msgr2=0x7fbdf40b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.456+0000 7fbdface0640 1 -- 172.21.15.116:0/3451973046 shutdown_connections 2023-11-28T17:09:46.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.456+0000 7fbdface0640 1 -- 172.21.15.116:0/3451973046 wait complete. 2023-11-28T17:09:46.474 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:09:46.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.690+0000 7fa254967640 1 Processor -- start 2023-11-28T17:09:46.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.690+0000 7fa254967640 1 -- start start 2023-11-28T17:09:46.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.692+0000 7fa254967640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa25015a270 0x7fa2500ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:46.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.692+0000 7fa254967640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25015a650 con 0x7fa2500b2f90 2023-11-28T17:09:46.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.692+0000 7fa254967640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25006de10 con 0x7fa25006d910 2023-11-28T17:09:46.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.692+0000 7fa254967640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2500b34e0 con 0x7fa25015a270 2023-11-28T17:09:46.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.693+0000 7fa24f59e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa2500b2f90 0x7fa2500709f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41308/0 (socket says 172.21.15.116:41308) 2023-11-28T17:09:46.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.693+0000 7fa24f59e640 1 -- 172.21.15.116:0/4141423158 learned_addr learned my addr 172.21.15.116:0/4141423158 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:46.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.693+0000 7fa24ed9d640 1 --2- 172.21.15.116:0/4141423158 >> v2:172.21.15.156:3300/0 conn(0x7fa25015a270 0x7fa2500ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:46.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.693+0000 7fa24e59c640 1 -- 172.21.15.116:0/4141423158 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668308516 0 0) 0x7fa25015a650 con 0x7fa2500b2f90 2023-11-28T17:09:46.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.693+0000 7fa24e59c640 1 -- 172.21.15.116:0/4141423158 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa23c003520 con 0x7fa2500b2f90 2023-11-28T17:09:46.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.693+0000 7fa24e59c640 1 -- 172.21.15.116:0/4141423158 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2904415403 0 0) 0x7fa23c003520 con 0x7fa2500b2f90 2023-11-28T17:09:46.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.693+0000 7fa24e59c640 1 -- 172.21.15.116:0/4141423158 >> v2:172.21.15.156:3300/0 conn(0x7fa25015a270 msgr2=0x7fa2500ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.693+0000 7fa24e59c640 1 --2- 172.21.15.116:0/4141423158 >> v2:172.21.15.156:3300/0 conn(0x7fa25015a270 0x7fa2500ae2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.693+0000 7fa24e59c640 1 -- 172.21.15.116:0/4141423158 >> v1:172.21.15.156:6789/0 conn(0x7fa25006d910 legacy=0x7fa2500b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.693+0000 7fa24e59c640 1 -- 172.21.15.116:0/4141423158 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2500b37c0 con 0x7fa2500b2f90 2023-11-28T17:09:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.693+0000 7fa24e59c640 1 -- 172.21.15.116:0/4141423158 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fa2480019f0 con 0x7fa2500b2f90 2023-11-28T17:09:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.693+0000 7fa24e59c640 1 -- 172.21.15.116:0/4141423158 >> v1:172.21.15.116:6789/0 conn(0x7fa2500b2f90 legacy=0x7fa2500709f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.694+0000 7fa24e59c640 1 --2- 172.21.15.116:0/4141423158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23c003970 0x7fa23c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.694+0000 7fa24e59c640 1 --2- 172.21.15.116:0/4141423158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa23c0047c0 0x7fa23c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.694+0000 7fa24e59c640 1 --2- 172.21.15.116:0/4141423158 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa23c005180 0x7fa23c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:46.697 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.694+0000 7fa24e59c640 1 -- 172.21.15.116:0/4141423158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2500b34e0 con 0x7fa23c0047c0 2023-11-28T17:09:46.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.694+0000 7fa24e59c640 1 -- 172.21.15.116:0/4141423158 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c00c020 con 0x7fa23c003970 2023-11-28T17:09:46.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.694+0000 7fa24e59c640 1 -- 172.21.15.116:0/4141423158 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23c00e3f0 con 0x7fa23c005180 2023-11-28T17:09:46.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.694+0000 7fa247fff640 1 --2- 172.21.15.116:0/4141423158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa23c0047c0 0x7fa23c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:46.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.694+0000 7fa247fff640 1 -- 172.21.15.116:0/4141423158 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa23c005180 msgr2=0x7fa23c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:46.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.694+0000 7fa24ed9d640 1 --2- 172.21.15.116:0/4141423158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23c003970 0x7fa23c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:46.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.694+0000 7fa247fff640 1 --2- 172.21.15.116:0/4141423158 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa23c005180 0x7fa23c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.694+0000 7fa247fff640 1 -- 172.21.15.116:0/4141423158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23c003970 msgr2=0x7fa23c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:46.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.694+0000 7fa247fff640 1 --2- 172.21.15.116:0/4141423158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23c003970 0x7fa23c003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.694+0000 7fa247fff640 1 -- 172.21.15.116:0/4141423158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23c003520 con 0x7fa23c0047c0 2023-11-28T17:09:46.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.694+0000 7fa24f59e640 1 -- 172.21.15.116:0/4141423158 reap_dead start 2023-11-28T17:09:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.695+0000 7fa247fff640 1 --2- 172.21.15.116:0/4141423158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa23c0047c0 0x7fa23c004c40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa238001750 tx=0x7fa23c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d6baa2a5933b216 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.696+0000 7fa24e59c640 1 -- 172.21.15.116:0/4141423158 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2380052d0 con 0x7fa23c0047c0 2023-11-28T17:09:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.696+0000 7fa24e59c640 1 -- 172.21.15.116:0/4141423158 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23802e050 con 0x7fa23c0047c0 2023-11-28T17:09:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.696+0000 7fa24e59c640 1 -- 172.21.15.116:0/4141423158 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa238003830 con 0x7fa23c0047c0 2023-11-28T17:09:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.696+0000 7fa254967640 1 -- 172.21.15.116:0/4141423158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa23c0047c0 msgr2=0x7fa23c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.696+0000 7fa254967640 1 --2- 172.21.15.116:0/4141423158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa23c0047c0 0x7fa23c004c40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa238001750 tx=0x7fa23c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.696+0000 7fa254967640 1 -- 172.21.15.116:0/4141423158 shutdown_connections 2023-11-28T17:09:46.700 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.696+0000 7fa254967640 1 --2- 172.21.15.116:0/4141423158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa23c0047c0 0x7fa23c004c40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.696+0000 7fa254967640 1 -- 172.21.15.116:0/4141423158 >> 172.21.15.116:0/4141423158 conn(0x7fa2500b0950 msgr2=0x7fa2500b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:46.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.696+0000 7fa254967640 1 -- 172.21.15.116:0/4141423158 shutdown_connections 2023-11-28T17:09:46.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.696+0000 7fa254967640 1 -- 172.21.15.116:0/4141423158 wait complete. 2023-11-28T17:09:46.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.698+0000 7fa254967640 1 Processor -- start 2023-11-28T17:09:46.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.698+0000 7fa254967640 1 -- start start 2023-11-28T17:09:46.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.699+0000 7fa254967640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23c0047c0 0x7fa2501491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:46.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.699+0000 7fa254967640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2500c45d0 0x7fa250148750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.699+0000 7fa254967640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa2500c3c30 0x7fa2501f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.699+0000 7fa254967640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2500c4e00 con 0x7fa2500c45d0 2023-11-28T17:09:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.699+0000 7fa254967640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25009b630 con 0x7fa23c0047c0 2023-11-28T17:09:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.699+0000 7fa254967640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa250148c90 con 0x7fa2500c3c30 2023-11-28T17:09:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.699+0000 7fa247fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2500c45d0 0x7fa250148750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.700+0000 7fa247fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2500c45d0 0x7fa250148750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60060/0 (socket says 172.21.15.116:60060) 2023-11-28T17:09:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.700+0000 7fa247fff640 1 -- 172.21.15.116:0/1244033270 learned_addr learned my addr 172.21.15.116:0/1244033270 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.700+0000 7fa24ed9d640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23c0047c0 0x7fa2501491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.700+0000 7fa247fff640 1 -- 172.21.15.116:0/1244033270 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa2500c3c30 msgr2=0x7fa2501f2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.700+0000 7fa24f59e640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa2500c3c30 0x7fa2501f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.700+0000 7fa247fff640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa2500c3c30 0x7fa2501f2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.700+0000 7fa247fff640 1 -- 172.21.15.116:0/1244033270 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23c0047c0 msgr2=0x7fa2501491a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.700+0000 7fa247fff640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23c0047c0 0x7fa2501491a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.700+0000 7fa247fff640 1 -- 172.21.15.116:0/1244033270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2500c4b20 con 0x7fa2500c45d0 2023-11-28T17:09:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.700+0000 7fa24ed9d640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23c0047c0 0x7fa2501491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.700+0000 7fa247fff640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2500c45d0 0x7fa250148750 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa238005040 tx=0x7fa238001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.701+0000 7fa2477fe640 1 -- 172.21.15.116:0/1244033270 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa238003e80 con 0x7fa2500c45d0 2023-11-28T17:09:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.701+0000 7fa2477fe640 1 -- 172.21.15.116:0/1244033270 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa238007a30 con 0x7fa2500c45d0 2023-11-28T17:09:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.701+0000 7fa2477fe640 1 -- 172.21.15.116:0/1244033270 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23803d6b0 con 0x7fa2500c45d0 2023-11-28T17:09:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.701+0000 7fa254967640 1 -- 172.21.15.116:0/1244033270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25009afe0 con 0x7fa2500c45d0 2023-11-28T17:09:46.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.701+0000 7fa254967640 1 -- 172.21.15.116:0/1244033270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2500c40b0 con 0x7fa2500c45d0 2023-11-28T17:09:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.702+0000 7fa254967640 1 -- 172.21.15.116:0/1244033270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25009ae40 con 0x7fa2500c45d0 2023-11-28T17:09:46.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.703+0000 7fa2477fe640 1 -- 172.21.15.116:0/1244033270 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa23803d810 con 0x7fa2500c45d0 2023-11-28T17:09:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.704+0000 7fa2477fe640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa23006b2a0 0x7fa23006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.704+0000 7fa2477fe640 1 -- 172.21.15.116:0/1244033270 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa2380ab770 con 0x7fa2500c45d0 2023-11-28T17:09:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.704+0000 7fa24ed9d640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa23006b2a0 0x7fa23006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.705+0000 7fa24ed9d640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa23006b2a0 0x7fa23006d760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa25004faf0 tx=0x7fa240008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:46.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.708+0000 7fa2477fe640 1 -- 172.21.15.116:0/1244033270 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa25009ae40 con 0x7fa2500c45d0 2023-11-28T17:09:46.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.892+0000 7fa254967640 1 -- 172.21.15.116:0/1244033270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa25009ae40 con 0x7fa2500c45d0 2023-11-28T17:09:46.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.893+0000 7fa2477fe640 1 -- 172.21.15.116:0/1244033270 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa238041020 con 0x7fa2500c45d0 2023-11-28T17:09:46.894 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:09:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.895+0000 7fa254967640 1 -- 172.21.15.116:0/1244033270 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa23006b2a0 msgr2=0x7fa23006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.895+0000 7fa254967640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa23006b2a0 0x7fa23006d760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa25004faf0 tx=0x7fa240008040 comp rx=0 tx=0).stop 2023-11-28T17:09:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.895+0000 7fa254967640 1 -- 172.21.15.116:0/1244033270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2500c45d0 msgr2=0x7fa250148750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.895+0000 7fa254967640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2500c45d0 0x7fa250148750 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa238005040 tx=0x7fa238001120 comp rx=0 tx=0).stop 2023-11-28T17:09:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.895+0000 7fa254967640 1 -- 172.21.15.116:0/1244033270 shutdown_connections 2023-11-28T17:09:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.895+0000 7fa254967640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa23006b2a0 0x7fa23006d760 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.895+0000 7fa254967640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa2500c3c30 0x7fa2501f2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.895+0000 7fa254967640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2500c45d0 0x7fa250148750 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.895+0000 7fa254967640 1 --2- 172.21.15.116:0/1244033270 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa23c0047c0 0x7fa2501491a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:46.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.896+0000 7fa254967640 1 -- 172.21.15.116:0/1244033270 >> 172.21.15.116:0/1244033270 conn(0x7fa2500b0950 msgr2=0x7fa25006d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:46.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.896+0000 7fa254967640 1 -- 172.21.15.116:0/1244033270 shutdown_connections 2023-11-28T17:09:46.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:46.896+0000 7fa254967640 1 -- 172.21.15.116:0/1244033270 wait complete. 2023-11-28T17:09:46.916 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:09:47.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.124+0000 7fe6049ec640 1 Processor -- start 2023-11-28T17:09:47.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.124+0000 7fe6049ec640 1 -- start start 2023-11-28T17:09:47.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.126+0000 7fe6049ec640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe60013e070 0x7fe6001412f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:47.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.126+0000 7fe6049ec640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6000cf7e0 0x7fe60006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:47.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.126+0000 7fe6049ec640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe600141830 con 0x7fe60006ed00 2023-11-28T17:09:47.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.126+0000 7fe6049ec640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe60006f170 con 0x7fe60013e070 2023-11-28T17:09:47.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.126+0000 7fe6049ec640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60015efb0 con 0x7fe6000cf7e0 2023-11-28T17:09:47.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.127+0000 7fe5fe59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe60013e070 0x7fe6001412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:47.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.127+0000 7fe5fdd9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6000cf7e0 0x7fe60006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:47.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.127+0000 7fe5fe59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe60013e070 0x7fe6001412f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:53300/0 (socket says 172.21.15.116:53300) 2023-11-28T17:09:47.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.127+0000 7fe5fe59c640 1 -- 172.21.15.116:0/1690278556 learned_addr learned my addr 172.21.15.116:0/1690278556 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:47.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.127+0000 7fe5fdd9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6000cf7e0 0x7fe60006e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:47408/0 (socket says 172.21.15.116:47408) 2023-11-28T17:09:47.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.127+0000 7fe5fdd9b640 1 -- 172.21.15.116:0/1690278556 >> v2:172.21.15.156:3300/0 conn(0x7fe60013e070 msgr2=0x7fe6001412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:47.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.127+0000 7fe5fdd9b640 1 --2- 172.21.15.116:0/1690278556 >> v2:172.21.15.156:3300/0 conn(0x7fe60013e070 0x7fe6001412f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.128+0000 7fe5fdd9b640 1 -- 172.21.15.116:0/1690278556 >> v1:172.21.15.156:6789/0 conn(0x7fe60006ed00 legacy=0x7fe60015b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:47.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.128+0000 7fe5fdd9b640 1 -- 172.21.15.116:0/1690278556 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60015f290 con 0x7fe6000cf7e0 2023-11-28T17:09:47.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.129+0000 7fe5fdd9b640 1 --2- 172.21.15.116:0/1690278556 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6000cf7e0 0x7fe60006e7c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80097a0 tx=0x7fe5e802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=994308219a8d70e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:47.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.129+0000 7fe5fd59a640 1 -- 172.21.15.116:0/1690278556 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8031780 con 0x7fe6000cf7e0 2023-11-28T17:09:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.130+0000 7fe5fd59a640 1 -- 172.21.15.116:0/1690278556 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e80318e0 con 0x7fe6000cf7e0 2023-11-28T17:09:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.130+0000 7fe5fd59a640 1 -- 172.21.15.116:0/1690278556 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8031c70 con 0x7fe6000cf7e0 2023-11-28T17:09:47.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.130+0000 7fe6049ec640 1 -- 172.21.15.116:0/1690278556 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6000cf7e0 msgr2=0x7fe60006e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.130+0000 7fe6049ec640 1 --2- 172.21.15.116:0/1690278556 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6000cf7e0 0x7fe60006e7c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe5e80097a0 tx=0x7fe5e802b820 comp rx=0 tx=0).stop 2023-11-28T17:09:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.130+0000 7fe6049ec640 1 -- 172.21.15.116:0/1690278556 shutdown_connections 2023-11-28T17:09:47.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.130+0000 7fe6049ec640 1 --2- 172.21.15.116:0/1690278556 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6000cf7e0 0x7fe60006e7c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.130+0000 7fe6049ec640 1 --2- 172.21.15.116:0/1690278556 >> v2:172.21.15.156:3300/0 conn(0x7fe60013e070 0x7fe6001412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.130+0000 7fe6049ec640 1 -- 172.21.15.116:0/1690278556 >> 172.21.15.116:0/1690278556 conn(0x7fe60006a8b0 msgr2=0x7fe60006acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.130+0000 7fe6049ec640 1 -- 172.21.15.116:0/1690278556 shutdown_connections 2023-11-28T17:09:47.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.130+0000 7fe6049ec640 1 -- 172.21.15.116:0/1690278556 wait complete. 2023-11-28T17:09:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.132+0000 7fe6049ec640 1 Processor -- start 2023-11-28T17:09:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.132+0000 7fe6049ec640 1 -- start start 2023-11-28T17:09:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.133+0000 7fe6049ec640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe60006ed00 0x7fe600135860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:47.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.133+0000 7fe6049ec640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6000cf7e0 0x7fe60015b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.133+0000 7fe6049ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60013e070 0x7fe6000b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.133+0000 7fe6049ec640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe600135da0 con 0x7fe60013e070 2023-11-28T17:09:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.133+0000 7fe6049ec640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe600156020 con 0x7fe60006ed00 2023-11-28T17:09:47.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.133+0000 7fe6049ec640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe600156300 con 0x7fe6000cf7e0 2023-11-28T17:09:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.134+0000 7fe5fed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60013e070 0x7fe6000b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.134+0000 7fe5fed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60013e070 0x7fe6000b0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60070/0 (socket says 172.21.15.116:60070) 2023-11-28T17:09:47.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.134+0000 7fe5fed9d640 1 -- 172.21.15.116:0/2898420656 learned_addr learned my addr 172.21.15.116:0/2898420656 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:47.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.134+0000 7fe5fe59c640 1 --2- 172.21.15.116:0/2898420656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe60006ed00 0x7fe600135860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:47.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.134+0000 7fe5fdd9b640 1 --2- 172.21.15.116:0/2898420656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6000cf7e0 0x7fe60015b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:47.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.134+0000 7fe5fed9d640 1 -- 172.21.15.116:0/2898420656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6000cf7e0 msgr2=0x7fe60015b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:47.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.134+0000 7fe5fed9d640 1 --2- 172.21.15.116:0/2898420656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6000cf7e0 0x7fe60015b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.134+0000 7fe5fed9d640 1 -- 172.21.15.116:0/2898420656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe60006ed00 msgr2=0x7fe600135860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:47.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.134+0000 7fe5fed9d640 1 --2- 172.21.15.116:0/2898420656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe60006ed00 0x7fe600135860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.134+0000 7fe5fed9d640 1 -- 172.21.15.116:0/2898420656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e8009400 con 0x7fe60013e070 2023-11-28T17:09:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.134+0000 7fe5fed9d640 1 --2- 172.21.15.116:0/2898420656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60013e070 0x7fe6000b0c90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe5f800c390 tx=0x7fe5f800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.134+0000 7fe5f6ffd640 1 -- 172.21.15.116:0/2898420656 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f80052d0 con 0x7fe60013e070 2023-11-28T17:09:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.135+0000 7fe5f6ffd640 1 -- 172.21.15.116:0/2898420656 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f8005430 con 0x7fe60013e070 2023-11-28T17:09:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.135+0000 7fe5f6ffd640 1 -- 172.21.15.116:0/2898420656 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8003810 con 0x7fe60013e070 2023-11-28T17:09:47.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.135+0000 7fe6049ec640 1 -- 172.21.15.116:0/2898420656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6001565e0 con 0x7fe60013e070 2023-11-28T17:09:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.135+0000 7fe6049ec640 1 -- 172.21.15.116:0/2898420656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe600156ba0 con 0x7fe60013e070 2023-11-28T17:09:47.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.136+0000 7fe6049ec640 1 -- 172.21.15.116:0/2898420656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe600156d90 con 0x7fe60013e070 2023-11-28T17:09:47.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.137+0000 7fe5f6ffd640 1 -- 172.21.15.116:0/2898420656 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe5f8003970 con 0x7fe60013e070 2023-11-28T17:09:47.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.138+0000 7fe5f6ffd640 1 --2- 172.21.15.116:0/2898420656 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe5e006b2d0 0x7fe5e006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:47.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.138+0000 7fe5f6ffd640 1 -- 172.21.15.116:0/2898420656 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe5f8088f30 con 0x7fe60013e070 2023-11-28T17:09:47.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.138+0000 7fe5fe59c640 1 --2- 172.21.15.116:0/2898420656 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe5e006b2d0 0x7fe5e006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:47.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.139+0000 7fe5fe59c640 1 --2- 172.21.15.116:0/2898420656 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe5e006b2d0 0x7fe5e006d790 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009d80 tx=0x7fe5f000b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:47.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.143+0000 7fe5f6ffd640 1 -- 172.21.15.116:0/2898420656 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5f8053d80 con 0x7fe60013e070 2023-11-28T17:09:47.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.347+0000 7fe6049ec640 1 -- 172.21.15.116:0/2898420656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe600067460 con 0x7fe60013e070 2023-11-28T17:09:47.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.348+0000 7fe5f6ffd640 1 -- 172.21.15.116:0/2898420656 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe5f8017080 con 0x7fe60013e070 2023-11-28T17:09:47.349 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-28T17:09:47.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.350+0000 7fe6049ec640 1 -- 172.21.15.116:0/2898420656 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe5e006b2d0 msgr2=0x7fe5e006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:47.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.350+0000 7fe6049ec640 1 --2- 172.21.15.116:0/2898420656 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe5e006b2d0 0x7fe5e006d790 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009d80 tx=0x7fe5f000b040 comp rx=0 tx=0).stop 2023-11-28T17:09:47.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.350+0000 7fe6049ec640 1 -- 172.21.15.116:0/2898420656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60013e070 msgr2=0x7fe6000b0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:47.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.350+0000 7fe6049ec640 1 --2- 172.21.15.116:0/2898420656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60013e070 0x7fe6000b0c90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe5f800c390 tx=0x7fe5f800c850 comp rx=0 tx=0).stop 2023-11-28T17:09:47.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.351+0000 7fe6049ec640 1 -- 172.21.15.116:0/2898420656 shutdown_connections 2023-11-28T17:09:47.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.351+0000 7fe6049ec640 1 --2- 172.21.15.116:0/2898420656 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe5e006b2d0 0x7fe5e006d790 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.351+0000 7fe6049ec640 1 --2- 172.21.15.116:0/2898420656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60013e070 0x7fe6000b0c90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.351+0000 7fe6049ec640 1 --2- 172.21.15.116:0/2898420656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6000cf7e0 0x7fe60015b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.351+0000 7fe6049ec640 1 --2- 172.21.15.116:0/2898420656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe60006ed00 0x7fe600135860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.351+0000 7fe6049ec640 1 -- 172.21.15.116:0/2898420656 >> 172.21.15.116:0/2898420656 conn(0x7fe60006a8b0 msgr2=0x7fe60006cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:47.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.351+0000 7fe6049ec640 1 -- 172.21.15.116:0/2898420656 shutdown_connections 2023-11-28T17:09:47.353 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.351+0000 7fe6049ec640 1 -- 172.21.15.116:0/2898420656 wait complete. 2023-11-28T17:09:47.370 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2023-11-28T17:09:47.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:47.371 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-11-28T17:09:47.467 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:09:47.468 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:09:47.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:47.468 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T17:09:47.540 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:09:47.541 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:09:47.599 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:09:47.600 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:09:47.653 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:47.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.872+0000 7f1392905640 1 Processor -- start 2023-11-28T17:09:47.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.872+0000 7f1392905640 1 -- start start 2023-11-28T17:09:47.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.873+0000 7f1392905640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f138c06c960 0x7f138c15a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:47.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.873+0000 7f1392905640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138c0aea20 con 0x7f138c140200 2023-11-28T17:09:47.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.873+0000 7f1392905640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138c13d2c0 con 0x7f138c13cee0 2023-11-28T17:09:47.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.873+0000 7f1392905640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138c15b3a0 con 0x7f138c06c960 2023-11-28T17:09:47.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.874+0000 7f1390ea2640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f138c140200 0x7f138c0cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41312/0 (socket says 172.21.15.116:41312) 2023-11-28T17:09:47.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.874+0000 7f1390ea2640 1 -- 172.21.15.116:0/1838437611 learned_addr learned my addr 172.21.15.116:0/1838437611 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.874+0000 7f138b7fe640 1 --2- 172.21.15.116:0/1838437611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f138c06c960 0x7f138c15a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.874+0000 7f138affd640 1 -- 172.21.15.116:0/1838437611 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983729182 0 0) 0x7f138c0aea20 con 0x7f138c140200 2023-11-28T17:09:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.874+0000 7f138affd640 1 -- 172.21.15.116:0/1838437611 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1374003520 con 0x7f138c140200 2023-11-28T17:09:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.875+0000 7f138affd640 1 -- 172.21.15.116:0/1838437611 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4085176228 0 0) 0x7f1374003520 con 0x7f138c140200 2023-11-28T17:09:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.875+0000 7f138affd640 1 -- 172.21.15.116:0/1838437611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f138c06c960 msgr2=0x7f138c15a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.875+0000 7f138affd640 1 --2- 172.21.15.116:0/1838437611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f138c06c960 0x7f138c15a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.875+0000 7f138affd640 1 -- 172.21.15.116:0/1838437611 >> v1:172.21.15.156:6789/0 conn(0x7f138c13cee0 legacy=0x7f138c06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.875+0000 7f138affd640 1 -- 172.21.15.116:0/1838437611 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c15b0c0 con 0x7f138c140200 2023-11-28T17:09:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.875+0000 7f138affd640 1 -- 172.21.15.116:0/1838437611 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f13840019f0 con 0x7f138c140200 2023-11-28T17:09:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.875+0000 7f138affd640 1 -- 172.21.15.116:0/1838437611 >> v1:172.21.15.116:6789/0 conn(0x7f138c140200 legacy=0x7f138c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:47.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.875+0000 7f138affd640 1 --2- 172.21.15.116:0/1838437611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13740039a0 0x7f1374003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.875+0000 7f138affd640 1 --2- 172.21.15.116:0/1838437611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13740047c0 0x7f1374004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.875+0000 7f138affd640 1 --2- 172.21.15.116:0/1838437611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f138c06c960 0x7f13740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.875+0000 7f138affd640 1 -- 172.21.15.116:0/1838437611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c15b3a0 con 0x7f13740047c0 2023-11-28T17:09:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.875+0000 7f138affd640 1 -- 172.21.15.116:0/1838437611 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137400dcc0 con 0x7f13740039a0 2023-11-28T17:09:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.875+0000 7f138affd640 1 -- 172.21.15.116:0/1838437611 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137400de80 con 0x7f138c06c960 2023-11-28T17:09:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.875+0000 7f138b7fe640 1 --2- 172.21.15.116:0/1838437611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13740047c0 0x7f1374004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.876+0000 7f138b7fe640 1 -- 172.21.15.116:0/1838437611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f138c06c960 msgr2=0x7f13740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.876+0000 7f138bfff640 1 --2- 172.21.15.116:0/1838437611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13740039a0 0x7f1374003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.876+0000 7f1390ea2640 1 --2- 172.21.15.116:0/1838437611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f138c06c960 0x7f13740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.876+0000 7f138b7fe640 1 --2- 172.21.15.116:0/1838437611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f138c06c960 0x7f13740051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.876+0000 7f138b7fe640 1 -- 172.21.15.116:0/1838437611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13740039a0 msgr2=0x7f1374003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.876+0000 7f138b7fe640 1 --2- 172.21.15.116:0/1838437611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13740039a0 0x7f1374003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.876+0000 7f138b7fe640 1 -- 172.21.15.116:0/1838437611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1374003520 con 0x7f13740047c0 2023-11-28T17:09:47.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.876+0000 7f138bfff640 1 --2- 172.21.15.116:0/1838437611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13740039a0 0x7f1374003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:47.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.877+0000 7f138b7fe640 1 --2- 172.21.15.116:0/1838437611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13740047c0 0x7f1374004c40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1380009ac0 tx=0x7f13740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a314adad38dc6f7a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:47.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.877+0000 7f138affd640 1 -- 172.21.15.116:0/1838437611 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138002c6b0 con 0x7f13740047c0 2023-11-28T17:09:47.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.877+0000 7f138affd640 1 -- 172.21.15.116:0/1838437611 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138002c810 con 0x7f13740047c0 2023-11-28T17:09:47.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.877+0000 7f138affd640 1 -- 172.21.15.116:0/1838437611 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138002cba0 con 0x7f13740047c0 2023-11-28T17:09:47.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.878+0000 7f1392905640 1 -- 172.21.15.116:0/1838437611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13740047c0 msgr2=0x7f1374004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:47.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.878+0000 7f1392905640 1 --2- 172.21.15.116:0/1838437611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13740047c0 0x7f1374004c40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1380009ac0 tx=0x7f13740057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.878+0000 7f1390ea2640 1 -- 172.21.15.116:0/1838437611 reap_dead start 2023-11-28T17:09:47.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.878+0000 7f1392905640 1 -- 172.21.15.116:0/1838437611 shutdown_connections 2023-11-28T17:09:47.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.878+0000 7f1392905640 1 -- 172.21.15.116:0/1838437611 >> 172.21.15.116:0/1838437611 conn(0x7f138c06a4c0 msgr2=0x7f138c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:47.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.878+0000 7f1392905640 1 -- 172.21.15.116:0/1838437611 shutdown_connections 2023-11-28T17:09:47.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.878+0000 7f1392905640 1 -- 172.21.15.116:0/1838437611 wait complete. 2023-11-28T17:09:47.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.880+0000 7f1392905640 1 Processor -- start 2023-11-28T17:09:47.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.880+0000 7f1392905640 1 -- start start 2023-11-28T17:09:47.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.881+0000 7f1392905640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f13740047c0 0x7f138c1f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:47.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.881+0000 7f1392905640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f138c13d460 0x7f138c1392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:47.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.881+0000 7f1392905640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c15c530 0x7f138c15c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:47.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.881+0000 7f1392905640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c15cef0 con 0x7f138c15c530 2023-11-28T17:09:47.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.881+0000 7f1392905640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c13ae20 con 0x7f138c13d460 2023-11-28T17:09:47.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.881+0000 7f1392905640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138c13d9d0 con 0x7f13740047c0 2023-11-28T17:09:47.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.881+0000 7f1390ea2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c15c530 0x7f138c15c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:47.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.881+0000 7f1390ea2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c15c530 0x7f138c15c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60096/0 (socket says 172.21.15.116:60096) 2023-11-28T17:09:47.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.881+0000 7f1390ea2640 1 -- 172.21.15.116:0/179236769 learned_addr learned my addr 172.21.15.116:0/179236769 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:47.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.881+0000 7f1390ea2640 1 -- 172.21.15.116:0/179236769 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f13740047c0 msgr2=0x7f138c1f2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:47.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.881+0000 7f1390ea2640 1 --2- 172.21.15.116:0/179236769 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f13740047c0 0x7f138c1f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.881+0000 7f1390ea2640 1 -- 172.21.15.116:0/179236769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f138c13d460 msgr2=0x7f138c1392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:47.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.881+0000 7f1390ea2640 1 --2- 172.21.15.116:0/179236769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f138c13d460 0x7f138c1392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:47.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.881+0000 7f1390ea2640 1 -- 172.21.15.116:0/179236769 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c13dcb0 con 0x7f138c15c530 2023-11-28T17:09:47.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.882+0000 7f1390ea2640 1 --2- 172.21.15.116:0/179236769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c15c530 0x7f138c15c9b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1374004740 tx=0x7f138400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:47.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.882+0000 7f1388ff9640 1 -- 172.21.15.116:0/179236769 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1384004d60 con 0x7f138c15c530 2023-11-28T17:09:47.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.882+0000 7f1388ff9640 1 -- 172.21.15.116:0/179236769 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138400bb80 con 0x7f138c15c530 2023-11-28T17:09:47.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.882+0000 7f1392905640 1 -- 172.21.15.116:0/179236769 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138c13df90 con 0x7f138c15c530 2023-11-28T17:09:47.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.882+0000 7f1388ff9640 1 -- 172.21.15.116:0/179236769 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1384007070 con 0x7f138c15c530 2023-11-28T17:09:47.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.882+0000 7f1392905640 1 -- 172.21.15.116:0/179236769 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138c1f2d70 con 0x7f138c15c530 2023-11-28T17:09:47.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.883+0000 7f1392905640 1 -- 172.21.15.116:0/179236769 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f138c0693d0 con 0x7f138c15c530 2023-11-28T17:09:47.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.884+0000 7f1388ff9640 1 -- 172.21.15.116:0/179236769 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f13840071d0 con 0x7f138c15c530 2023-11-28T17:09:47.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.885+0000 7f1388ff9640 1 --2- 172.21.15.116:0/179236769 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f137806b2a0 0x7f137806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:47.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.885+0000 7f1388ff9640 1 -- 172.21.15.116:0/179236769 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f138408a070 con 0x7f138c15c530 2023-11-28T17:09:47.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.885+0000 7f138bfff640 1 --2- 172.21.15.116:0/179236769 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f137806b2a0 0x7f137806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:47.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.886+0000 7f138bfff640 1 --2- 172.21.15.116:0/179236769 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f137806b2a0 0x7f137806d760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f137c0016d0 tx=0x7f137c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:47.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:47.889+0000 7f1388ff9640 1 -- 172.21.15.116:0/179236769 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1384054e20 con 0x7f138c15c530 2023-11-28T17:09:48.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.082+0000 7f1392905640 1 -- 172.21.15.116:0/179236769 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f138c06d170 con 0x7f138c15c530 2023-11-28T17:09:48.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.084+0000 7f1388ff9640 1 -- 172.21.15.116:0/179236769 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f1384058440 con 0x7f138c15c530 2023-11-28T17:09:48.086 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:48.086 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:43.049465+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.088+0000 7f1392905640 1 -- 172.21.15.116:0/179236769 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f137806b2a0 msgr2=0x7f137806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.088+0000 7f1392905640 1 --2- 172.21.15.116:0/179236769 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f137806b2a0 0x7f137806d760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f137c0016d0 tx=0x7f137c009040 comp rx=0 tx=0).stop 2023-11-28T17:09:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.088+0000 7f1392905640 1 -- 172.21.15.116:0/179236769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c15c530 msgr2=0x7f138c15c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.088+0000 7f1392905640 1 --2- 172.21.15.116:0/179236769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c15c530 0x7f138c15c9b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1374004740 tx=0x7f138400b4a0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.088+0000 7f1392905640 1 -- 172.21.15.116:0/179236769 shutdown_connections 2023-11-28T17:09:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.088+0000 7f1392905640 1 --2- 172.21.15.116:0/179236769 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f137806b2a0 0x7f137806d760 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.088+0000 7f1392905640 1 --2- 172.21.15.116:0/179236769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f138c15c530 0x7f138c15c9b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.088+0000 7f1392905640 1 --2- 172.21.15.116:0/179236769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f138c13d460 0x7f138c1392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.088+0000 7f1392905640 1 --2- 172.21.15.116:0/179236769 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f13740047c0 0x7f138c1f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.088+0000 7f1392905640 1 -- 172.21.15.116:0/179236769 >> 172.21.15.116:0/179236769 conn(0x7f138c06a4c0 msgr2=0x7f138c0afdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.089+0000 7f1392905640 1 -- 172.21.15.116:0/179236769 shutdown_connections 2023-11-28T17:09:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.089+0000 7f1392905640 1 -- 172.21.15.116:0/179236769 wait complete. 2023-11-28T17:09:48.110 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:09:48.110 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:09:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.326+0000 7f3b4ef1d640 1 Processor -- start 2023-11-28T17:09:48.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.326+0000 7f3b4ef1d640 1 -- start start 2023-11-28T17:09:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.327+0000 7f3b4ef1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3b4813f640 0x7f3b4813fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.328+0000 7f3b4ef1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4813dd10 con 0x7f3b48071d20 2023-11-28T17:09:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.328+0000 7f3b4ef1d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4813d2c0 con 0x7f3b4813cee0 2023-11-28T17:09:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.328+0000 7f3b4ef1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b481400e0 con 0x7f3b4813f640 2023-11-28T17:09:48.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.328+0000 7f3b47fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3b4813f640 0x7f3b4813fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.328+0000 7f3b4d4ba640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3b48071d20 0x7f3b48072130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51398/0 (socket says 172.21.15.116:51398) 2023-11-28T17:09:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.328+0000 7f3b4d4ba640 1 -- 172.21.15.116:0/727738862 learned_addr learned my addr 172.21.15.116:0/727738862 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.328+0000 7f3b47fff640 1 -- 172.21.15.116:0/727738862 >> v1:172.21.15.156:6789/0 conn(0x7f3b4813cee0 legacy=0x7f3b4813d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.328+0000 7f3b47fff640 1 -- 172.21.15.116:0/727738862 >> v1:172.21.15.116:6789/0 conn(0x7f3b48071d20 legacy=0x7f3b48072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:48.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.328+0000 7f3b47fff640 1 -- 172.21.15.116:0/727738862 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b481403c0 con 0x7f3b4813f640 2023-11-28T17:09:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.330+0000 7f3b47fff640 1 --2- 172.21.15.116:0/727738862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b4813f640 0x7f3b4813fab0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3b340097a0 tx=0x7f3b3402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecaecf30de91bc14 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.330+0000 7f3b477fe640 1 -- 172.21.15.116:0/727738862 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3402f050 con 0x7f3b4813f640 2023-11-28T17:09:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.330+0000 7f3b477fe640 1 -- 172.21.15.116:0/727738862 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b34031700 con 0x7f3b4813f640 2023-11-28T17:09:48.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.330+0000 7f3b477fe640 1 -- 172.21.15.116:0/727738862 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b34031a90 con 0x7f3b4813f640 2023-11-28T17:09:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.330+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/727738862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b4813f640 msgr2=0x7f3b4813fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.330+0000 7f3b4ef1d640 1 --2- 172.21.15.116:0/727738862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b4813f640 0x7f3b4813fab0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3b340097a0 tx=0x7f3b3402b7a0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.331+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/727738862 shutdown_connections 2023-11-28T17:09:48.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.331+0000 7f3b4ef1d640 1 --2- 172.21.15.116:0/727738862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b4813f640 0x7f3b4813fab0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.331+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/727738862 >> 172.21.15.116:0/727738862 conn(0x7f3b480b0cf0 msgr2=0x7f3b480b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.331+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/727738862 shutdown_connections 2023-11-28T17:09:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.331+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/727738862 wait complete. 2023-11-28T17:09:48.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.332+0000 7f3b4ef1d640 1 Processor -- start 2023-11-28T17:09:48.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.333+0000 7f3b4ef1d640 1 -- start start 2023-11-28T17:09:48.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.333+0000 7f3b4ef1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b48071d20 0x7f3b481f0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:48.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.334+0000 7f3b4ef1d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b4813cee0 0x7f3b480b2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:48.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.334+0000 7f3b4ef1d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b4813f640 0x7f3b481385e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:48.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.334+0000 7f3b4ef1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b480b41f0 con 0x7f3b48071d20 2023-11-28T17:09:48.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.334+0000 7f3b4ef1d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b48142600 con 0x7f3b4813f640 2023-11-28T17:09:48.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.334+0000 7f3b4ef1d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b481428e0 con 0x7f3b4813cee0 2023-11-28T17:09:48.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.334+0000 7f3b4ccb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b48071d20 0x7f3b481f0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:48.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.334+0000 7f3b4ccb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b48071d20 0x7f3b481f0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56300/0 (socket says 172.21.15.116:56300) 2023-11-28T17:09:48.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.334+0000 7f3b4ccb9640 1 -- 172.21.15.116:0/3735126047 learned_addr learned my addr 172.21.15.116:0/3735126047 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:48.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.334+0000 7f3b4ccb9640 1 -- 172.21.15.116:0/3735126047 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b4813cee0 msgr2=0x7f3b480b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:48.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.334+0000 7f3b4ccb9640 1 --2- 172.21.15.116:0/3735126047 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b4813cee0 0x7f3b480b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.334+0000 7f3b4ccb9640 1 -- 172.21.15.116:0/3735126047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b4813f640 msgr2=0x7f3b481385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:48.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.334+0000 7f3b4ccb9640 1 --2- 172.21.15.116:0/3735126047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b4813f640 0x7f3b481385e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.334+0000 7f3b4ccb9640 1 -- 172.21.15.116:0/3735126047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b34009400 con 0x7f3b48071d20 2023-11-28T17:09:48.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.334+0000 7f3b4ccb9640 1 --2- 172.21.15.116:0/3735126047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b48071d20 0x7f3b481f0cd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c00c430 tx=0x7f3b3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:48.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.335+0000 7f3b457fa640 1 -- 172.21.15.116:0/3735126047 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c011790 con 0x7f3b48071d20 2023-11-28T17:09:48.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.335+0000 7f3b457fa640 1 -- 172.21.15.116:0/3735126047 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3c0118f0 con 0x7f3b48071d20 2023-11-28T17:09:48.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.335+0000 7f3b457fa640 1 -- 172.21.15.116:0/3735126047 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c011ce0 con 0x7f3b48071d20 2023-11-28T17:09:48.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.335+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/3735126047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b48152ee0 con 0x7f3b48071d20 2023-11-28T17:09:48.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.335+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/3735126047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b48152b00 con 0x7f3b48071d20 2023-11-28T17:09:48.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.336+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/3735126047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b48152cf0 con 0x7f3b48071d20 2023-11-28T17:09:48.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.337+0000 7f3b457fa640 1 -- 172.21.15.116:0/3735126047 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3b3c01c460 con 0x7f3b48071d20 2023-11-28T17:09:48.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.338+0000 7f3b457fa640 1 --2- 172.21.15.116:0/3735126047 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3b2c06b2a0 0x7f3b2c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:48.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.338+0000 7f3b457fa640 1 -- 172.21.15.116:0/3735126047 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3b3c089b60 con 0x7f3b48071d20 2023-11-28T17:09:48.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.338+0000 7f3b47fff640 1 --2- 172.21.15.116:0/3735126047 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3b2c06b2a0 0x7f3b2c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:48.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.339+0000 7f3b47fff640 1 --2- 172.21.15.116:0/3735126047 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3b2c06b2a0 0x7f3b2c06d760 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3b34035000 tx=0x7f3b34030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:48.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.342+0000 7f3b457fa640 1 -- 172.21.15.116:0/3735126047 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b3c0549b0 con 0x7f3b48071d20 2023-11-28T17:09:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.531+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/3735126047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3b48069360 con 0x7f3b48071d20 2023-11-28T17:09:48.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.532+0000 7f3b457fa640 1 -- 172.21.15.116:0/3735126047 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3b3c057fd0 con 0x7f3b48071d20 2023-11-28T17:09:48.533 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:09:48.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.534+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/3735126047 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3b2c06b2a0 msgr2=0x7f3b2c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:48.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.534+0000 7f3b4ef1d640 1 --2- 172.21.15.116:0/3735126047 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3b2c06b2a0 0x7f3b2c06d760 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3b34035000 tx=0x7f3b34030000 comp rx=0 tx=0).stop 2023-11-28T17:09:48.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.534+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/3735126047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b48071d20 msgr2=0x7f3b481f0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:48.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.534+0000 7f3b4ef1d640 1 --2- 172.21.15.116:0/3735126047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b48071d20 0x7f3b481f0cd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c00c430 tx=0x7f3b3c00c8f0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.534+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/3735126047 shutdown_connections 2023-11-28T17:09:48.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.534+0000 7f3b4ef1d640 1 --2- 172.21.15.116:0/3735126047 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3b2c06b2a0 0x7f3b2c06d760 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.534+0000 7f3b4ef1d640 1 --2- 172.21.15.116:0/3735126047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b4813f640 0x7f3b481385e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.534+0000 7f3b4ef1d640 1 --2- 172.21.15.116:0/3735126047 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3b4813cee0 0x7f3b480b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.534+0000 7f3b4ef1d640 1 --2- 172.21.15.116:0/3735126047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b48071d20 0x7f3b481f0cd0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.534+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/3735126047 >> 172.21.15.116:0/3735126047 conn(0x7f3b480b0cf0 msgr2=0x7f3b480ce350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:48.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.534+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/3735126047 shutdown_connections 2023-11-28T17:09:48.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.534+0000 7f3b4ef1d640 1 -- 172.21.15.116:0/3735126047 wait complete. 2023-11-28T17:09:48.555 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:09:48.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.758+0000 7f522a716640 1 Processor -- start 2023-11-28T17:09:48.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.758+0000 7f522a716640 1 -- start start 2023-11-28T17:09:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.760+0000 7f522a716640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f52240b3370 0x7f522406e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.760+0000 7f522a716640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f522406e820 0x7f522406cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.760+0000 7f522a716640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522406f050 con 0x7f5224157780 2023-11-28T17:09:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.760+0000 7f522a716640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f522415d780 con 0x7f52240b3370 2023-11-28T17:09:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.760+0000 7f522a716640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5224160e10 con 0x7f522406e820 2023-11-28T17:09:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.760+0000 7f5228f13640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f52240b3370 0x7f522406e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.761+0000 7f5228f13640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f52240b3370 0x7f522406e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56306/0 (socket says 172.21.15.116:56306) 2023-11-28T17:09:48.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.761+0000 7f5228f13640 1 -- 172.21.15.116:0/3761502602 learned_addr learned my addr 172.21.15.116:0/3761502602 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:48.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.761+0000 7f5228f13640 1 -- 172.21.15.116:0/3761502602 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f522406e820 msgr2=0x7f522406cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:48.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.761+0000 7f5228f13640 1 --2- 172.21.15.116:0/3761502602 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f522406e820 0x7f522406cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.761+0000 7f5228f13640 1 -- 172.21.15.116:0/3761502602 >> v1:172.21.15.156:6789/0 conn(0x7f5224157780 legacy=0x7f52240b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.761+0000 7f5228f13640 1 -- 172.21.15.116:0/3761502602 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52241610f0 con 0x7f52240b3370 2023-11-28T17:09:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.762+0000 7f5228f13640 1 --2- 172.21.15.116:0/3761502602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52240b3370 0x7f522406e2e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f522406f240 tx=0x7f521402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef98ee268450ca6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.762+0000 7f521bfff640 1 -- 172.21.15.116:0/3761502602 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5214031640 con 0x7f52240b3370 2023-11-28T17:09:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.762+0000 7f521bfff640 1 -- 172.21.15.116:0/3761502602 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52140317a0 con 0x7f52240b3370 2023-11-28T17:09:48.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.762+0000 7f521bfff640 1 -- 172.21.15.116:0/3761502602 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5214031b30 con 0x7f52240b3370 2023-11-28T17:09:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.763+0000 7f522a716640 1 -- 172.21.15.116:0/3761502602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52240b3370 msgr2=0x7f522406e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.763+0000 7f522a716640 1 --2- 172.21.15.116:0/3761502602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52240b3370 0x7f522406e2e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f522406f240 tx=0x7f521402b8f0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.763+0000 7f522a716640 1 -- 172.21.15.116:0/3761502602 shutdown_connections 2023-11-28T17:09:48.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.763+0000 7f522a716640 1 --2- 172.21.15.116:0/3761502602 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f522406e820 0x7f522406cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.763+0000 7f522a716640 1 --2- 172.21.15.116:0/3761502602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52240b3370 0x7f522406e2e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.763+0000 7f522a716640 1 -- 172.21.15.116:0/3761502602 >> 172.21.15.116:0/3761502602 conn(0x7f52240b1960 msgr2=0x7f52240b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:48.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.763+0000 7f522a716640 1 -- 172.21.15.116:0/3761502602 shutdown_connections 2023-11-28T17:09:48.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.763+0000 7f522a716640 1 -- 172.21.15.116:0/3761502602 wait complete. 2023-11-28T17:09:48.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.765+0000 7f522a716640 1 Processor -- start 2023-11-28T17:09:48.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.765+0000 7f522a716640 1 -- start start 2023-11-28T17:09:48.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.766+0000 7f522a716640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522406e820 0x7f52241f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.766+0000 7f522a716640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52240b3370 0x7f52240aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.766+0000 7f522a716640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5224157780 0x7f5224156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.766+0000 7f522a716640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5224160540 con 0x7f522406e820 2023-11-28T17:09:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.766+0000 7f522a716640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522415ee60 con 0x7f52240b3370 2023-11-28T17:09:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.766+0000 7f522a716640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522415f140 con 0x7f5224157780 2023-11-28T17:09:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.766+0000 7f5229714640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522406e820 0x7f52241f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.767+0000 7f5229714640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522406e820 0x7f52241f30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56316/0 (socket says 172.21.15.116:56316) 2023-11-28T17:09:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.767+0000 7f5229714640 1 -- 172.21.15.116:0/2694769580 learned_addr learned my addr 172.21.15.116:0/2694769580 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.767+0000 7f5229f15640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5224157780 0x7f5224156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.767+0000 7f5228f13640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52240b3370 0x7f52240aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:48.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.767+0000 7f5229714640 1 -- 172.21.15.116:0/2694769580 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5224157780 msgr2=0x7f5224156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:48.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.767+0000 7f5229714640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5224157780 0x7f5224156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.767+0000 7f5229714640 1 -- 172.21.15.116:0/2694769580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52240b3370 msgr2=0x7f52240aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:48.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.767+0000 7f5229714640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52240b3370 0x7f52240aea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.767+0000 7f5229714640 1 -- 172.21.15.116:0/2694769580 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52140094d0 con 0x7f522406e820 2023-11-28T17:09:48.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.767+0000 7f5228f13640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52240b3370 0x7f52240aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:48.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.767+0000 7f5229f15640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5224157780 0x7f5224156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:48.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.767+0000 7f5229714640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522406e820 0x7f52241f30c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f522000ba50 tx=0x7f522000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:48.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.768+0000 7f5219ffb640 1 -- 172.21.15.116:0/2694769580 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522000cc10 con 0x7f522406e820 2023-11-28T17:09:48.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.768+0000 7f5219ffb640 1 -- 172.21.15.116:0/2694769580 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f522000cd70 con 0x7f522406e820 2023-11-28T17:09:48.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.768+0000 7f5219ffb640 1 -- 172.21.15.116:0/2694769580 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5220014680 con 0x7f522406e820 2023-11-28T17:09:48.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.768+0000 7f522a716640 1 -- 172.21.15.116:0/2694769580 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522415f420 con 0x7f522406e820 2023-11-28T17:09:48.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.768+0000 7f522a716640 1 -- 172.21.15.116:0/2694769580 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522415f9e0 con 0x7f522406e820 2023-11-28T17:09:48.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.769+0000 7f522a716640 1 -- 172.21.15.116:0/2694769580 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5224067960 con 0x7f522406e820 2023-11-28T17:09:48.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.770+0000 7f5219ffb640 1 -- 172.21.15.116:0/2694769580 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f5220007370 con 0x7f522406e820 2023-11-28T17:09:48.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.771+0000 7f5219ffb640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f520806b2a0 0x7f520806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:48.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.771+0000 7f5219ffb640 1 -- 172.21.15.116:0/2694769580 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5220089cd0 con 0x7f522406e820 2023-11-28T17:09:48.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.771+0000 7f5228f13640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f520806b2a0 0x7f520806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:48.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.772+0000 7f5228f13640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f520806b2a0 0x7f520806d760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5214036000 tx=0x7f521403c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:48.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.775+0000 7f5219ffb640 1 -- 172.21.15.116:0/2694769580 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5220054a80 con 0x7f522406e820 2023-11-28T17:09:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.963+0000 7f522a716640 1 -- 172.21.15.116:0/2694769580 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5224143ad0 con 0x7f522406e820 2023-11-28T17:09:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.964+0000 7f5219ffb640 1 -- 172.21.15.116:0/2694769580 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f52200190a0 con 0x7f522406e820 2023-11-28T17:09:48.966 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:09:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.967+0000 7f522a716640 1 -- 172.21.15.116:0/2694769580 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f520806b2a0 msgr2=0x7f520806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.967+0000 7f522a716640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f520806b2a0 0x7f520806d760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5214036000 tx=0x7f521403c040 comp rx=0 tx=0).stop 2023-11-28T17:09:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.967+0000 7f522a716640 1 -- 172.21.15.116:0/2694769580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522406e820 msgr2=0x7f52241f30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.967+0000 7f522a716640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522406e820 0x7f52241f30c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f522000ba50 tx=0x7f522000bf10 comp rx=0 tx=0).stop 2023-11-28T17:09:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.967+0000 7f522a716640 1 -- 172.21.15.116:0/2694769580 shutdown_connections 2023-11-28T17:09:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.967+0000 7f522a716640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f520806b2a0 0x7f520806d760 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.967+0000 7f522a716640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5224157780 0x7f5224156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.967+0000 7f522a716640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f52240b3370 0x7f52240aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.967+0000 7f522a716640 1 --2- 172.21.15.116:0/2694769580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f522406e820 0x7f52241f30c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.967+0000 7f522a716640 1 -- 172.21.15.116:0/2694769580 >> 172.21.15.116:0/2694769580 conn(0x7f52240b1960 msgr2=0x7f52241433c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.968+0000 7f522a716640 1 -- 172.21.15.116:0/2694769580 shutdown_connections 2023-11-28T17:09:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:48.968+0000 7f522a716640 1 -- 172.21.15.116:0/2694769580 wait complete. 2023-11-28T17:09:48.988 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:09:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.193+0000 7fa74f3ef640 1 Processor -- start 2023-11-28T17:09:49.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.193+0000 7fa74f3ef640 1 -- start start 2023-11-28T17:09:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.194+0000 7fa74f3ef640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa74815a270 0x7fa7480ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.194+0000 7fa74f3ef640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa74806d910 0x7fa7480b2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.194+0000 7fa74f3ef640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74815a650 con 0x7fa7480b2dc0 2023-11-28T17:09:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.194+0000 7fa74f3ef640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7480b3250 con 0x7fa74815a270 2023-11-28T17:09:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.194+0000 7fa74f3ef640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74806de10 con 0x7fa74806d910 2023-11-28T17:09:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.195+0000 7fa74ebee640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa7480b2dc0 0x7fa7480ac7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51414/0 (socket says 172.21.15.116:51414) 2023-11-28T17:09:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.195+0000 7fa74ebee640 1 -- 172.21.15.116:0/2669119221 learned_addr learned my addr 172.21.15.116:0/2669119221 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:49.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.195+0000 7fa74e3ed640 1 --2- 172.21.15.116:0/2669119221 >> v2:172.21.15.156:3300/0 conn(0x7fa74815a270 0x7fa7480ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:49.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.195+0000 7fa74dbec640 1 --2- 172.21.15.116:0/2669119221 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa74806d910 0x7fa7480b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:49.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.195+0000 7fa74d3eb640 1 -- 172.21.15.116:0/2669119221 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001472363 0 0) 0x7fa74815a650 con 0x7fa7480b2dc0 2023-11-28T17:09:49.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.195+0000 7fa74d3eb640 1 -- 172.21.15.116:0/2669119221 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa734003520 con 0x7fa7480b2dc0 2023-11-28T17:09:49.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.195+0000 7fa74d3eb640 1 -- 172.21.15.116:0/2669119221 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 198176193 0 0) 0x7fa734003520 con 0x7fa7480b2dc0 2023-11-28T17:09:49.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.195+0000 7fa74d3eb640 1 -- 172.21.15.116:0/2669119221 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa74806d910 msgr2=0x7fa7480b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:49.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.195+0000 7fa74d3eb640 1 --2- 172.21.15.116:0/2669119221 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa74806d910 0x7fa7480b2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.195+0000 7fa74d3eb640 1 -- 172.21.15.116:0/2669119221 >> v2:172.21.15.156:3300/0 conn(0x7fa74815a270 msgr2=0x7fa7480ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.195+0000 7fa74d3eb640 1 --2- 172.21.15.116:0/2669119221 >> v2:172.21.15.156:3300/0 conn(0x7fa74815a270 0x7fa7480ae2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.195+0000 7fa74d3eb640 1 -- 172.21.15.116:0/2669119221 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7480b34e0 con 0x7fa7480b2dc0 2023-11-28T17:09:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.196+0000 7fa74d3eb640 1 -- 172.21.15.116:0/2669119221 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fa7400019f0 con 0x7fa7480b2dc0 2023-11-28T17:09:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.196+0000 7fa74d3eb640 1 -- 172.21.15.116:0/2669119221 >> v1:172.21.15.116:6789/0 conn(0x7fa7480b2dc0 legacy=0x7fa7480ac7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.196+0000 7fa74d3eb640 1 --2- 172.21.15.116:0/2669119221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa734003970 0x7fa734003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.196+0000 7fa74d3eb640 1 --2- 172.21.15.116:0/2669119221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7340047c0 0x7fa734004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.196+0000 7fa74d3eb640 1 --2- 172.21.15.116:0/2669119221 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa74806d910 0x7fa734005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.196+0000 7fa74d3eb640 1 -- 172.21.15.116:0/2669119221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7480b3250 con 0x7fa7340047c0 2023-11-28T17:09:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.196+0000 7fa74d3eb640 1 -- 172.21.15.116:0/2669119221 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74806de10 con 0x7fa734003970 2023-11-28T17:09:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.196+0000 7fa74d3eb640 1 -- 172.21.15.116:0/2669119221 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa73400db90 con 0x7fa74806d910 2023-11-28T17:09:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.196+0000 7fa74dbec640 1 --2- 172.21.15.116:0/2669119221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7340047c0 0x7fa734004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.196+0000 7fa74dbec640 1 -- 172.21.15.116:0/2669119221 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa74806d910 msgr2=0x7fa734005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.196+0000 7fa74dbec640 1 --2- 172.21.15.116:0/2669119221 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa74806d910 0x7fa734005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.196+0000 7fa74dbec640 1 -- 172.21.15.116:0/2669119221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa734003970 msgr2=0x7fa734003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:49.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.196+0000 7fa74dbec640 1 --2- 172.21.15.116:0/2669119221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa734003970 0x7fa734003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.196+0000 7fa74dbec640 1 -- 172.21.15.116:0/2669119221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa734003520 con 0x7fa7340047c0 2023-11-28T17:09:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.197+0000 7fa74dbec640 1 --2- 172.21.15.116:0/2669119221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7340047c0 0x7fa734004c40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa73800ba30 tx=0x7fa7340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548fb8fe7d371267 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.197+0000 7fa74d3eb640 1 -- 172.21.15.116:0/2669119221 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa738032680 con 0x7fa7340047c0 2023-11-28T17:09:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.197+0000 7fa74d3eb640 1 -- 172.21.15.116:0/2669119221 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7380327e0 con 0x7fa7340047c0 2023-11-28T17:09:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.197+0000 7fa74d3eb640 1 -- 172.21.15.116:0/2669119221 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa738032b70 con 0x7fa7340047c0 2023-11-28T17:09:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.198+0000 7fa74f3ef640 1 -- 172.21.15.116:0/2669119221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7340047c0 msgr2=0x7fa734004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.198+0000 7fa74f3ef640 1 --2- 172.21.15.116:0/2669119221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7340047c0 0x7fa734004c40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa73800ba30 tx=0x7fa7340057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.198+0000 7fa74ebee640 1 -- 172.21.15.116:0/2669119221 reap_dead start 2023-11-28T17:09:49.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.198+0000 7fa74f3ef640 1 -- 172.21.15.116:0/2669119221 shutdown_connections 2023-11-28T17:09:49.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.198+0000 7fa74f3ef640 1 -- 172.21.15.116:0/2669119221 >> 172.21.15.116:0/2669119221 conn(0x7fa7480b0950 msgr2=0x7fa7480b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:49.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.198+0000 7fa74f3ef640 1 -- 172.21.15.116:0/2669119221 shutdown_connections 2023-11-28T17:09:49.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.198+0000 7fa74f3ef640 1 -- 172.21.15.116:0/2669119221 wait complete. 2023-11-28T17:09:49.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.200+0000 7fa74f3ef640 1 Processor -- start 2023-11-28T17:09:49.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.200+0000 7fa74f3ef640 1 -- start start 2023-11-28T17:09:49.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.201+0000 7fa74f3ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7340047c0 0x7fa7480cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:49.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.201+0000 7fa74f3ef640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7481f4050 0x7fa748154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:49.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.201+0000 7fa74f3ef640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa74815cd00 0x7fa74815d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:49.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.201+0000 7fa74f3ef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74809ad10 con 0x7fa7340047c0 2023-11-28T17:09:49.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.201+0000 7fa74f3ef640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74809a7c0 con 0x7fa74815cd00 2023-11-28T17:09:49.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.201+0000 7fa74f3ef640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74815b310 con 0x7fa7481f4050 2023-11-28T17:09:49.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.201+0000 7fa74e3ed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7340047c0 0x7fa7480cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:49.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.201+0000 7fa74e3ed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7340047c0 0x7fa7480cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56338/0 (socket says 172.21.15.116:56338) 2023-11-28T17:09:49.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.201+0000 7fa74e3ed640 1 -- 172.21.15.116:0/1441871727 learned_addr learned my addr 172.21.15.116:0/1441871727 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:49.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.201+0000 7fa74e3ed640 1 -- 172.21.15.116:0/1441871727 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7481f4050 msgr2=0x7fa748154620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:49.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.201+0000 7fa74e3ed640 1 --2- 172.21.15.116:0/1441871727 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7481f4050 0x7fa748154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.201+0000 7fa74e3ed640 1 -- 172.21.15.116:0/1441871727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa74815cd00 msgr2=0x7fa74815d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:49.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.201+0000 7fa74e3ed640 1 --2- 172.21.15.116:0/1441871727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa74815cd00 0x7fa74815d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.201+0000 7fa74e3ed640 1 -- 172.21.15.116:0/1441871727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74809aa30 con 0x7fa7340047c0 2023-11-28T17:09:49.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.202+0000 7fa74e3ed640 1 --2- 172.21.15.116:0/1441871727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7340047c0 0x7fa7480cd680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa74400e890 tx=0x7fa74400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:49.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.202+0000 7fa73effd640 1 -- 172.21.15.116:0/1441871727 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74400c940 con 0x7fa7340047c0 2023-11-28T17:09:49.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.202+0000 7fa73effd640 1 -- 172.21.15.116:0/1441871727 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74400cac0 con 0x7fa7340047c0 2023-11-28T17:09:49.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.202+0000 7fa73effd640 1 -- 172.21.15.116:0/1441871727 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74400ce70 con 0x7fa7340047c0 2023-11-28T17:09:49.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.202+0000 7fa74f3ef640 1 -- 172.21.15.116:0/1441871727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74815b5b0 con 0x7fa7340047c0 2023-11-28T17:09:49.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.202+0000 7fa74f3ef640 1 -- 172.21.15.116:0/1441871727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7481f44d0 con 0x7fa7340047c0 2023-11-28T17:09:49.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.203+0000 7fa74f3ef640 1 -- 172.21.15.116:0/1441871727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa74815b170 con 0x7fa7340047c0 2023-11-28T17:09:49.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.204+0000 7fa73effd640 1 -- 172.21.15.116:0/1441871727 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa744013470 con 0x7fa7340047c0 2023-11-28T17:09:49.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.206+0000 7fa73effd640 1 --2- 172.21.15.116:0/1441871727 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa72c06b2a0 0x7fa72c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:49.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.206+0000 7fa73effd640 1 -- 172.21.15.116:0/1441871727 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa744089d00 con 0x7fa7340047c0 2023-11-28T17:09:49.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.206+0000 7fa74dbec640 1 --2- 172.21.15.116:0/1441871727 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa72c06b2a0 0x7fa72c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:49.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.206+0000 7fa74dbec640 1 --2- 172.21.15.116:0/1441871727 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa72c06b2a0 0x7fa72c06d760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa73800ba00 tx=0x7fa738032000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:49.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.210+0000 7fa73effd640 1 -- 172.21.15.116:0/1441871727 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa74815b170 con 0x7fa7340047c0 2023-11-28T17:09:49.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.393+0000 7fa74f3ef640 1 -- 172.21.15.116:0/1441871727 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa74815b170 con 0x7fa7340047c0 2023-11-28T17:09:49.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.394+0000 7fa73effd640 1 -- 172.21.15.116:0/1441871727 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa74815b170 con 0x7fa7340047c0 2023-11-28T17:09:49.395 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:09:49.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.396+0000 7fa74f3ef640 1 -- 172.21.15.116:0/1441871727 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa72c06b2a0 msgr2=0x7fa72c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:49.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.396+0000 7fa74f3ef640 1 --2- 172.21.15.116:0/1441871727 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa72c06b2a0 0x7fa72c06d760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa73800ba00 tx=0x7fa738032000 comp rx=0 tx=0).stop 2023-11-28T17:09:49.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.396+0000 7fa74f3ef640 1 -- 172.21.15.116:0/1441871727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7340047c0 msgr2=0x7fa7480cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:49.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.396+0000 7fa74f3ef640 1 --2- 172.21.15.116:0/1441871727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7340047c0 0x7fa7480cd680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa74400e890 tx=0x7fa74400ed50 comp rx=0 tx=0).stop 2023-11-28T17:09:49.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.397+0000 7fa74f3ef640 1 -- 172.21.15.116:0/1441871727 shutdown_connections 2023-11-28T17:09:49.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.397+0000 7fa74f3ef640 1 --2- 172.21.15.116:0/1441871727 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa72c06b2a0 0x7fa72c06d760 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.397+0000 7fa74f3ef640 1 --2- 172.21.15.116:0/1441871727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa74815cd00 0x7fa74815d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.397+0000 7fa74f3ef640 1 --2- 172.21.15.116:0/1441871727 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7481f4050 0x7fa748154620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.397+0000 7fa74f3ef640 1 --2- 172.21.15.116:0/1441871727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7340047c0 0x7fa7480cd680 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.397+0000 7fa74f3ef640 1 -- 172.21.15.116:0/1441871727 >> 172.21.15.116:0/1441871727 conn(0x7fa7480b0950 msgr2=0x7fa7480ad980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:49.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.397+0000 7fa74f3ef640 1 -- 172.21.15.116:0/1441871727 shutdown_connections 2023-11-28T17:09:49.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.397+0000 7fa74f3ef640 1 -- 172.21.15.116:0/1441871727 wait complete. 2023-11-28T17:09:49.416 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:09:49.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.616+0000 7f4245ae0640 1 Processor -- start 2023-11-28T17:09:49.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.617+0000 7f4245ae0640 1 -- start start 2023-11-28T17:09:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.618+0000 7f4245ae0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f424013cee0 0x7f424006c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.618+0000 7f4245ae0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42400aea20 con 0x7f424015aba0 2023-11-28T17:09:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.618+0000 7f4245ae0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424013d2c0 con 0x7f4240140200 2023-11-28T17:09:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.618+0000 7f4245ae0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424015b3a0 con 0x7f424013cee0 2023-11-28T17:09:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.619+0000 7f423f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f424013cee0 0x7f424006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.619+0000 7f423ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f424015aba0 0x7f42400cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51428/0 (socket says 172.21.15.116:51428) 2023-11-28T17:09:49.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.619+0000 7f423ffff640 1 -- 172.21.15.116:0/4216513010 learned_addr learned my addr 172.21.15.116:0/4216513010 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.619+0000 7f423f7fe640 1 -- 172.21.15.116:0/4216513010 >> v1:172.21.15.156:6789/0 conn(0x7f4240140200 legacy=0x7f424015a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.619+0000 7f423f7fe640 1 -- 172.21.15.116:0/4216513010 >> v1:172.21.15.116:6789/0 conn(0x7f424015aba0 legacy=0x7f42400cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.619+0000 7f423f7fe640 1 -- 172.21.15.116:0/4216513010 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424015b0c0 con 0x7f424013cee0 2023-11-28T17:09:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.619+0000 7f423effd640 1 -- 172.21.15.116:0/4216513010 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896697987 0 0) 0x7f42400aea20 con 0x7f424015aba0 2023-11-28T17:09:49.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.620+0000 7f423f7fe640 1 --2- 172.21.15.116:0/4216513010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424013cee0 0x7f424006c250 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f42300097a0 tx=0x7f423002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21bb266a04c1fe8a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.620+0000 7f423effd640 1 -- 172.21.15.116:0/4216513010 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423002e070 con 0x7f424013cee0 2023-11-28T17:09:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.620+0000 7f423effd640 1 -- 172.21.15.116:0/4216513010 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42300315a0 con 0x7f424013cee0 2023-11-28T17:09:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.620+0000 7f423effd640 1 -- 172.21.15.116:0/4216513010 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230031930 con 0x7f424013cee0 2023-11-28T17:09:49.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.621+0000 7f4245ae0640 1 -- 172.21.15.116:0/4216513010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424013cee0 msgr2=0x7f424006c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:49.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.621+0000 7f4245ae0640 1 --2- 172.21.15.116:0/4216513010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424013cee0 0x7f424006c250 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f42300097a0 tx=0x7f423002b840 comp rx=0 tx=0).stop 2023-11-28T17:09:49.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.621+0000 7f4245ae0640 1 -- 172.21.15.116:0/4216513010 shutdown_connections 2023-11-28T17:09:49.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.621+0000 7f4245ae0640 1 --2- 172.21.15.116:0/4216513010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424013cee0 0x7f424006c250 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.621+0000 7f4245ae0640 1 -- 172.21.15.116:0/4216513010 >> 172.21.15.116:0/4216513010 conn(0x7f424006a4c0 msgr2=0x7f424006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:49.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.621+0000 7f4245ae0640 1 -- 172.21.15.116:0/4216513010 shutdown_connections 2023-11-28T17:09:49.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.621+0000 7f4245ae0640 1 -- 172.21.15.116:0/4216513010 wait complete. 2023-11-28T17:09:49.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.623+0000 7f4245ae0640 1 Processor -- start 2023-11-28T17:09:49.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.623+0000 7f4245ae0640 1 -- start start 2023-11-28T17:09:49.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.624+0000 7f4245ae0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424013cee0 0x7f42400b3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:49.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.624+0000 7f4245ae0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4240140200 0x7f424013de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:49.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.624+0000 7f4245ae0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f424015aba0 0x7f424013e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:49.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.624+0000 7f4245ae0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424013f320 con 0x7f424013cee0 2023-11-28T17:09:49.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.624+0000 7f4245ae0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4240153c60 con 0x7f4240140200 2023-11-28T17:09:49.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.624+0000 7f4245ae0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4240070620 con 0x7f424015aba0 2023-11-28T17:09:49.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.624+0000 7f423f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424013cee0 0x7f42400b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:49.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.624+0000 7f423f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424013cee0 0x7f42400b3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56364/0 (socket says 172.21.15.116:56364) 2023-11-28T17:09:49.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.624+0000 7f423f7fe640 1 -- 172.21.15.116:0/3040314550 learned_addr learned my addr 172.21.15.116:0/3040314550 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:49.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.625+0000 7f423f7fe640 1 -- 172.21.15.116:0/3040314550 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f424015aba0 msgr2=0x7f424013e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:49.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.625+0000 7f423f7fe640 1 --2- 172.21.15.116:0/3040314550 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f424015aba0 0x7f424013e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.625+0000 7f423f7fe640 1 -- 172.21.15.116:0/3040314550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4240140200 msgr2=0x7f424013de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:49.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.625+0000 7f423f7fe640 1 --2- 172.21.15.116:0/3040314550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4240140200 0x7f424013de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.625+0000 7f423f7fe640 1 -- 172.21.15.116:0/3040314550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4230009400 con 0x7f424013cee0 2023-11-28T17:09:49.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.625+0000 7f423f7fe640 1 --2- 172.21.15.116:0/3040314550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424013cee0 0x7f42400b3670 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4230034040 tx=0x7f42300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:49.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.625+0000 7f423cff9640 1 -- 172.21.15.116:0/3040314550 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230032900 con 0x7f424013cee0 2023-11-28T17:09:49.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.625+0000 7f423cff9640 1 -- 172.21.15.116:0/3040314550 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4230032a80 con 0x7f424013cee0 2023-11-28T17:09:49.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.625+0000 7f423cff9640 1 -- 172.21.15.116:0/3040314550 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230032e30 con 0x7f424013cee0 2023-11-28T17:09:49.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.625+0000 7f4245ae0640 1 -- 172.21.15.116:0/3040314550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424013ea80 con 0x7f424013cee0 2023-11-28T17:09:49.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.625+0000 7f4245ae0640 1 -- 172.21.15.116:0/3040314550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424013f040 con 0x7f424013cee0 2023-11-28T17:09:49.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.626+0000 7f4245ae0640 1 -- 172.21.15.116:0/3040314550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42400703d0 con 0x7f424013cee0 2023-11-28T17:09:49.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.627+0000 7f423cff9640 1 -- 172.21.15.116:0/3040314550 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f423003b3f0 con 0x7f424013cee0 2023-11-28T17:09:49.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.628+0000 7f423cff9640 1 --2- 172.21.15.116:0/3040314550 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f422006b2a0 0x7f422006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:49.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.628+0000 7f423cff9640 1 -- 172.21.15.116:0/3040314550 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f42300abbc0 con 0x7f424013cee0 2023-11-28T17:09:49.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.628+0000 7f4237fff640 1 --2- 172.21.15.116:0/3040314550 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f422006b2a0 0x7f422006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:49.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.629+0000 7f4237fff640 1 --2- 172.21.15.116:0/3040314550 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f422006b2a0 0x7f422006d760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4240159960 tx=0x7f42280079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:49.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.633+0000 7f423cff9640 1 -- 172.21.15.116:0/3040314550 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4230076970 con 0x7f424013cee0 2023-11-28T17:09:49.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.834+0000 7f4245ae0640 1 -- 172.21.15.116:0/3040314550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f424013d2c0 con 0x7f424013cee0 2023-11-28T17:09:49.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.835+0000 7f423cff9640 1 -- 172.21.15.116:0/3040314550 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f424013d2c0 con 0x7f424013cee0 2023-11-28T17:09:49.837 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-28T17:09:49.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.838+0000 7f4245ae0640 1 -- 172.21.15.116:0/3040314550 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f422006b2a0 msgr2=0x7f422006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.838+0000 7f4245ae0640 1 --2- 172.21.15.116:0/3040314550 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f422006b2a0 0x7f422006d760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4240159960 tx=0x7f42280079b0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.838+0000 7f4245ae0640 1 -- 172.21.15.116:0/3040314550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424013cee0 msgr2=0x7f42400b3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.838+0000 7f4245ae0640 1 --2- 172.21.15.116:0/3040314550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424013cee0 0x7f42400b3670 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4230034040 tx=0x7f42300046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.838+0000 7f4245ae0640 1 -- 172.21.15.116:0/3040314550 shutdown_connections 2023-11-28T17:09:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.838+0000 7f4245ae0640 1 --2- 172.21.15.116:0/3040314550 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f422006b2a0 0x7f422006d760 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.838+0000 7f4245ae0640 1 --2- 172.21.15.116:0/3040314550 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f424015aba0 0x7f424013e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.838+0000 7f4245ae0640 1 --2- 172.21.15.116:0/3040314550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4240140200 0x7f424013de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.838+0000 7f4245ae0640 1 --2- 172.21.15.116:0/3040314550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f424013cee0 0x7f42400b3670 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.838+0000 7f4245ae0640 1 -- 172.21.15.116:0/3040314550 >> 172.21.15.116:0/3040314550 conn(0x7f424006a4c0 msgr2=0x7f424015b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.838+0000 7f4245ae0640 1 -- 172.21.15.116:0/3040314550 shutdown_connections 2023-11-28T17:09:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:49.838+0000 7f4245ae0640 1 -- 172.21.15.116:0/3040314550 wait complete. 2023-11-28T17:09:49.858 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:09:49.859 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:09:49.917 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:50.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.134+0000 7f5ebd6a6640 1 Processor -- start 2023-11-28T17:09:50.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.134+0000 7f5ebd6a6640 1 -- start start 2023-11-28T17:09:50.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.136+0000 7f5ebd6a6640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5eb80b3d20 0x7f5eb8161d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:50.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.136+0000 7f5ebd6a6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb80b29c0 con 0x7f5eb806c960 2023-11-28T17:09:50.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.136+0000 7f5ebd6a6640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb8162af0 con 0x7f5eb80b3940 2023-11-28T17:09:50.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.136+0000 7f5ebd6a6640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb80b2090 con 0x7f5eb80b3d20 2023-11-28T17:09:50.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.136+0000 7f5eb77fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5eb806c960 0x7f5eb81622c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51444/0 (socket says 172.21.15.116:51444) 2023-11-28T17:09:50.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.136+0000 7f5eb77fe640 1 -- 172.21.15.116:0/3004726267 learned_addr learned my addr 172.21.15.116:0/3004726267 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eae7fc640 1 --2- 172.21.15.116:0/3004726267 >> v2:172.21.15.156:3300/0 conn(0x7f5eb80b3d20 0x7f5eb8161d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 -- 172.21.15.116:0/3004726267 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071532538 0 0) 0x7f5eb80b29c0 con 0x7f5eb806c960 2023-11-28T17:09:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 -- 172.21.15.116:0/3004726267 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea4003520 con 0x7f5eb806c960 2023-11-28T17:09:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 -- 172.21.15.116:0/3004726267 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1179463474 0 0) 0x7f5ea4003520 con 0x7f5eb806c960 2023-11-28T17:09:50.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 -- 172.21.15.116:0/3004726267 >> v2:172.21.15.156:3300/0 conn(0x7f5eb80b3d20 msgr2=0x7f5eb8161d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 --2- 172.21.15.116:0/3004726267 >> v2:172.21.15.156:3300/0 conn(0x7f5eb80b3d20 0x7f5eb8161d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 -- 172.21.15.116:0/3004726267 >> v1:172.21.15.156:6789/0 conn(0x7f5eb80b3940 legacy=0x7f5eb806c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 -- 172.21.15.116:0/3004726267 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb80b2370 con 0x7f5eb806c960 2023-11-28T17:09:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 -- 172.21.15.116:0/3004726267 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f5eb00019f0 con 0x7f5eb806c960 2023-11-28T17:09:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 -- 172.21.15.116:0/3004726267 >> v1:172.21.15.116:6789/0 conn(0x7f5eb806c960 legacy=0x7f5eb81622c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 --2- 172.21.15.116:0/3004726267 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ea40039c0 0x7f5ea4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 --2- 172.21.15.116:0/3004726267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ea40047c0 0x7f5ea4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 --2- 172.21.15.116:0/3004726267 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ea4005180 0x7f5ea400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 -- 172.21.15.116:0/3004726267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb80b2090 con 0x7f5ea40047c0 2023-11-28T17:09:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 -- 172.21.15.116:0/3004726267 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea400c050 con 0x7f5ea4005180 2023-11-28T17:09:50.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.137+0000 7f5eb67fc640 1 -- 172.21.15.116:0/3004726267 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea400c210 con 0x7f5ea40039c0 2023-11-28T17:09:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.138+0000 7f5eae7fc640 1 --2- 172.21.15.116:0/3004726267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ea40047c0 0x7f5ea4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.138+0000 7f5eb77fe640 1 --2- 172.21.15.116:0/3004726267 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ea4005180 0x7f5ea400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.138+0000 7f5eb6ffd640 1 --2- 172.21.15.116:0/3004726267 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ea40039c0 0x7f5ea4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.138+0000 7f5eae7fc640 1 -- 172.21.15.116:0/3004726267 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ea40039c0 msgr2=0x7f5ea4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.138+0000 7f5eae7fc640 1 --2- 172.21.15.116:0/3004726267 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ea40039c0 0x7f5ea4003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.138+0000 7f5eae7fc640 1 -- 172.21.15.116:0/3004726267 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ea4005180 msgr2=0x7f5ea400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.138+0000 7f5eae7fc640 1 --2- 172.21.15.116:0/3004726267 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ea4005180 0x7f5ea400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.138+0000 7f5eae7fc640 1 -- 172.21.15.116:0/3004726267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea4003520 con 0x7f5ea40047c0 2023-11-28T17:09:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.138+0000 7f5eb77fe640 1 -- 172.21.15.116:0/3004726267 reap_dead start 2023-11-28T17:09:50.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.139+0000 7f5eae7fc640 1 --2- 172.21.15.116:0/3004726267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ea40047c0 0x7f5ea4004c40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0009a90 tx=0x7f5ea40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e630302b35332f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:50.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.139+0000 7f5eb67fc640 1 -- 172.21.15.116:0/3004726267 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea002c680 con 0x7f5ea40047c0 2023-11-28T17:09:50.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.139+0000 7f5eb67fc640 1 -- 172.21.15.116:0/3004726267 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea002c7e0 con 0x7f5ea40047c0 2023-11-28T17:09:50.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.139+0000 7f5eb67fc640 1 -- 172.21.15.116:0/3004726267 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea002cb70 con 0x7f5ea40047c0 2023-11-28T17:09:50.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.140+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/3004726267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ea40047c0 msgr2=0x7f5ea4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:50.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.140+0000 7f5ebd6a6640 1 --2- 172.21.15.116:0/3004726267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ea40047c0 0x7f5ea4004c40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0009a90 tx=0x7f5ea40057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.140+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/3004726267 shutdown_connections 2023-11-28T17:09:50.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.140+0000 7f5ebd6a6640 1 --2- 172.21.15.116:0/3004726267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ea40047c0 0x7f5ea4004c40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.140+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/3004726267 >> 172.21.15.116:0/3004726267 conn(0x7f5eb806a4c0 msgr2=0x7f5eb806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:50.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.140+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/3004726267 shutdown_connections 2023-11-28T17:09:50.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.140+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/3004726267 wait complete. 2023-11-28T17:09:50.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.142+0000 7f5ebd6a6640 1 Processor -- start 2023-11-28T17:09:50.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.142+0000 7f5ebd6a6640 1 -- start start 2023-11-28T17:09:50.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.143+0000 7f5ebd6a6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ea40047c0 0x7f5eb80ce400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:50.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.143+0000 7f5ebd6a6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb80cc6a0 0x7f5eb814dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:50.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.143+0000 7f5ebd6a6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eb813ed10 0x7f5eb813f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:50.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.143+0000 7f5ebd6a6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb813f6d0 con 0x7f5eb813ed10 2023-11-28T17:09:50.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.143+0000 7f5ebd6a6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb80ce940 con 0x7f5eb80cc6a0 2023-11-28T17:09:50.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.143+0000 7f5ebd6a6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb813d7c0 con 0x7f5ea40047c0 2023-11-28T17:09:50.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.143+0000 7f5eb77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eb813ed10 0x7f5eb813f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:50.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.143+0000 7f5eb77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eb813ed10 0x7f5eb813f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56384/0 (socket says 172.21.15.116:56384) 2023-11-28T17:09:50.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.143+0000 7f5eb77fe640 1 -- 172.21.15.116:0/1611331297 learned_addr learned my addr 172.21.15.116:0/1611331297 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:50.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.144+0000 7f5eb77fe640 1 -- 172.21.15.116:0/1611331297 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ea40047c0 msgr2=0x7f5eb80ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:50.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.144+0000 7f5eb6ffd640 1 --2- 172.21.15.116:0/1611331297 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ea40047c0 0x7f5eb80ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:50.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.144+0000 7f5eae7fc640 1 --2- 172.21.15.116:0/1611331297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb80cc6a0 0x7f5eb814dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:50.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.144+0000 7f5eb77fe640 1 --2- 172.21.15.116:0/1611331297 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ea40047c0 0x7f5eb80ce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.144+0000 7f5eb77fe640 1 -- 172.21.15.116:0/1611331297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb80cc6a0 msgr2=0x7f5eb814dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:50.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.144+0000 7f5eb77fe640 1 --2- 172.21.15.116:0/1611331297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb80cc6a0 0x7f5eb814dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.144+0000 7f5eb77fe640 1 -- 172.21.15.116:0/1611331297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb80aec90 con 0x7f5eb813ed10 2023-11-28T17:09:50.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.144+0000 7f5eb77fe640 1 --2- 172.21.15.116:0/1611331297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eb813ed10 0x7f5eb813f190 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8068020 tx=0x7f5eb0007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:50.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.144+0000 7f5eaffff640 1 -- 172.21.15.116:0/1611331297 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb000eb00 con 0x7f5eb813ed10 2023-11-28T17:09:50.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.144+0000 7f5eaffff640 1 -- 172.21.15.116:0/1611331297 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb000ec60 con 0x7f5eb813ed10 2023-11-28T17:09:50.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.144+0000 7f5eaffff640 1 -- 172.21.15.116:0/1611331297 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0013420 con 0x7f5eb813ed10 2023-11-28T17:09:50.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.144+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/1611331297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb80aef70 con 0x7f5eb813ed10 2023-11-28T17:09:50.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.144+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/1611331297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb80af250 con 0x7f5eb813ed10 2023-11-28T17:09:50.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.146+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/1611331297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb81628c0 con 0x7f5eb813ed10 2023-11-28T17:09:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.146+0000 7f5eaffff640 1 -- 172.21.15.116:0/1611331297 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f5eb0013580 con 0x7f5eb813ed10 2023-11-28T17:09:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.147+0000 7f5eaffff640 1 --2- 172.21.15.116:0/1611331297 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5e9806b2a0 0x7f5e9806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.147+0000 7f5eaffff640 1 -- 172.21.15.116:0/1611331297 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5eb008a020 con 0x7f5eb813ed10 2023-11-28T17:09:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.148+0000 7f5eb6ffd640 1 --2- 172.21.15.116:0/1611331297 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5e9806b2a0 0x7f5e9806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:50.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.148+0000 7f5eb6ffd640 1 --2- 172.21.15.116:0/1611331297 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5e9806b2a0 0x7f5e9806d760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8050060 tx=0x7f5ea8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:50.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.152+0000 7f5eaffff640 1 -- 172.21.15.116:0/1611331297 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5eb0054f00 con 0x7f5eb813ed10 2023-11-28T17:09:50.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.344+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/1611331297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5eb80693d0 con 0x7f5eb813ed10 2023-11-28T17:09:50.348 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.346+0000 7f5eaffff640 1 -- 172.21.15.116:0/1611331297 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f5eb0058520 con 0x7f5eb813ed10 2023-11-28T17:09:50.348 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:50.348 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:43.049465+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:50.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.349+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/1611331297 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5e9806b2a0 msgr2=0x7f5e9806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:50.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.349+0000 7f5ebd6a6640 1 --2- 172.21.15.116:0/1611331297 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5e9806b2a0 0x7f5e9806d760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8050060 tx=0x7f5ea8009040 comp rx=0 tx=0).stop 2023-11-28T17:09:50.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.349+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/1611331297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eb813ed10 msgr2=0x7f5eb813f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:50.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.349+0000 7f5ebd6a6640 1 --2- 172.21.15.116:0/1611331297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eb813ed10 0x7f5eb813f190 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8068020 tx=0x7f5eb0007bc0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.349+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/1611331297 shutdown_connections 2023-11-28T17:09:50.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.349+0000 7f5ebd6a6640 1 --2- 172.21.15.116:0/1611331297 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5e9806b2a0 0x7f5e9806d760 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.349+0000 7f5ebd6a6640 1 --2- 172.21.15.116:0/1611331297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5eb813ed10 0x7f5eb813f190 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.349+0000 7f5ebd6a6640 1 --2- 172.21.15.116:0/1611331297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb80cc6a0 0x7f5eb814dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.349+0000 7f5ebd6a6640 1 --2- 172.21.15.116:0/1611331297 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5ea40047c0 0x7f5eb80ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.349+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/1611331297 >> 172.21.15.116:0/1611331297 conn(0x7f5eb806a4c0 msgr2=0x7f5eb8164420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:50.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.349+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/1611331297 shutdown_connections 2023-11-28T17:09:50.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.350+0000 7f5ebd6a6640 1 -- 172.21.15.116:0/1611331297 wait complete. 2023-11-28T17:09:50.370 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:09:50.371 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:09:50.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.583+0000 7fa199c51640 1 Processor -- start 2023-11-28T17:09:50.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.583+0000 7fa199c51640 1 -- start start 2023-11-28T17:09:50.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.585+0000 7fa199c51640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa194072250 0x7fa19415ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:50.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.585+0000 7fa199c51640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1940ce940 0x7fa1940cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:50.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.585+0000 7fa199c51640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1940cd320 con 0x7fa19415b0c0 2023-11-28T17:09:50.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.585+0000 7fa199c51640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1940cedd0 con 0x7fa194072250 2023-11-28T17:09:50.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.585+0000 7fa199c51640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1940cbee0 con 0x7fa1940ce940 2023-11-28T17:09:50.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.585+0000 7fa1937fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa19415b0c0 0x7fa194071b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51454/0 (socket says 172.21.15.116:51454) 2023-11-28T17:09:50.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.585+0000 7fa1937fe640 1 -- 172.21.15.116:0/1820999490 learned_addr learned my addr 172.21.15.116:0/1820999490 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:50.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.586+0000 7fa192ffd640 1 --2- 172.21.15.116:0/1820999490 >> v2:172.21.15.116:3300/0 conn(0x7fa194072250 0x7fa19415ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:50.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.586+0000 7fa193fff640 1 --2- 172.21.15.116:0/1820999490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1940ce940 0x7fa1940cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:50.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.586+0000 7fa1927fc640 1 -- 172.21.15.116:0/1820999490 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016083452 0 0) 0x7fa1940cd320 con 0x7fa19415b0c0 2023-11-28T17:09:50.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.586+0000 7fa1927fc640 1 -- 172.21.15.116:0/1820999490 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa17c003520 con 0x7fa19415b0c0 2023-11-28T17:09:50.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.586+0000 7fa192ffd640 1 -- 172.21.15.116:0/1820999490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1940ce940 msgr2=0x7fa1940cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:50.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.586+0000 7fa192ffd640 1 --2- 172.21.15.116:0/1820999490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1940ce940 0x7fa1940cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.586+0000 7fa192ffd640 1 -- 172.21.15.116:0/1820999490 >> v1:172.21.15.116:6789/0 conn(0x7fa19415b0c0 legacy=0x7fa194071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:50.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.586+0000 7fa192ffd640 1 -- 172.21.15.116:0/1820999490 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1940cc710 con 0x7fa194072250 2023-11-28T17:09:50.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.588+0000 7fa192ffd640 1 --2- 172.21.15.116:0/1820999490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa194072250 0x7fa19415ed10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa1880097a0 tx=0x7fa18802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dd0ce1e97d6e934 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:50.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.588+0000 7fa1927fc640 1 -- 172.21.15.116:0/1820999490 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188031780 con 0x7fa194072250 2023-11-28T17:09:50.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.588+0000 7fa1927fc640 1 -- 172.21.15.116:0/1820999490 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1880318e0 con 0x7fa194072250 2023-11-28T17:09:50.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.588+0000 7fa1927fc640 1 -- 172.21.15.116:0/1820999490 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188031c70 con 0x7fa194072250 2023-11-28T17:09:50.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.588+0000 7fa199c51640 1 -- 172.21.15.116:0/1820999490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa194072250 msgr2=0x7fa19415ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:50.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.588+0000 7fa199c51640 1 --2- 172.21.15.116:0/1820999490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa194072250 0x7fa19415ed10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa1880097a0 tx=0x7fa18802b820 comp rx=0 tx=0).stop 2023-11-28T17:09:50.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.589+0000 7fa199c51640 1 -- 172.21.15.116:0/1820999490 shutdown_connections 2023-11-28T17:09:50.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.589+0000 7fa199c51640 1 --2- 172.21.15.116:0/1820999490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1940ce940 0x7fa1940cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.589+0000 7fa199c51640 1 --2- 172.21.15.116:0/1820999490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa194072250 0x7fa19415ed10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.589+0000 7fa199c51640 1 -- 172.21.15.116:0/1820999490 >> 172.21.15.116:0/1820999490 conn(0x7fa1940b0cf0 msgr2=0x7fa1940b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:50.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.589+0000 7fa199c51640 1 -- 172.21.15.116:0/1820999490 shutdown_connections 2023-11-28T17:09:50.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.589+0000 7fa199c51640 1 -- 172.21.15.116:0/1820999490 wait complete. 2023-11-28T17:09:50.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.590+0000 7fa199c51640 1 Processor -- start 2023-11-28T17:09:50.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.591+0000 7fa199c51640 1 -- start start 2023-11-28T17:09:50.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.591+0000 7fa199c51640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa194072250 0x7fa194159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:50.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.591+0000 7fa199c51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1940ce940 0x7fa194146220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:50.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.592+0000 7fa199c51640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa19415b0c0 0x7fa1940c3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:50.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.592+0000 7fa199c51640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1940cb810 con 0x7fa1940ce940 2023-11-28T17:09:50.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.592+0000 7fa199c51640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1941499e0 con 0x7fa19415b0c0 2023-11-28T17:09:50.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.592+0000 7fa199c51640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa19415a050 con 0x7fa194072250 2023-11-28T17:09:50.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.592+0000 7fa192ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1940ce940 0x7fa194146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:50.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.592+0000 7fa192ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1940ce940 0x7fa194146220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56404/0 (socket says 172.21.15.116:56404) 2023-11-28T17:09:50.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.592+0000 7fa192ffd640 1 -- 172.21.15.116:0/1723115037 learned_addr learned my addr 172.21.15.116:0/1723115037 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:50.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.592+0000 7fa1937fe640 1 --2- 172.21.15.116:0/1723115037 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa194072250 0x7fa194159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:50.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.592+0000 7fa192ffd640 1 -- 172.21.15.116:0/1723115037 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa194072250 msgr2=0x7fa194159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:50.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.592+0000 7fa193fff640 1 --2- 172.21.15.116:0/1723115037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa19415b0c0 0x7fa1940c3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:50.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.592+0000 7fa192ffd640 1 --2- 172.21.15.116:0/1723115037 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa194072250 0x7fa194159b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.592+0000 7fa192ffd640 1 -- 172.21.15.116:0/1723115037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa19415b0c0 msgr2=0x7fa1940c3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:50.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.592+0000 7fa192ffd640 1 --2- 172.21.15.116:0/1723115037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa19415b0c0 0x7fa1940c3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.592+0000 7fa192ffd640 1 -- 172.21.15.116:0/1723115037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa188009400 con 0x7fa1940ce940 2023-11-28T17:09:50.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.593+0000 7fa192ffd640 1 --2- 172.21.15.116:0/1723115037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1940ce940 0x7fa194146220 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa188034040 tx=0x7fa1880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:50.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.593+0000 7fa17bfff640 1 -- 172.21.15.116:0/1723115037 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188032820 con 0x7fa1940ce940 2023-11-28T17:09:50.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.593+0000 7fa17bfff640 1 -- 172.21.15.116:0/1723115037 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa188032980 con 0x7fa1940ce940 2023-11-28T17:09:50.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.593+0000 7fa17bfff640 1 -- 172.21.15.116:0/1723115037 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa188032d30 con 0x7fa1940ce940 2023-11-28T17:09:50.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.593+0000 7fa199c51640 1 -- 172.21.15.116:0/1723115037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa194159090 con 0x7fa1940ce940 2023-11-28T17:09:50.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.593+0000 7fa199c51640 1 -- 172.21.15.116:0/1723115037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1941467c0 con 0x7fa1940ce940 2023-11-28T17:09:50.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.594+0000 7fa199c51640 1 -- 172.21.15.116:0/1723115037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa194069360 con 0x7fa1940ce940 2023-11-28T17:09:50.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.595+0000 7fa17bfff640 1 -- 172.21.15.116:0/1723115037 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa18804e440 con 0x7fa1940ce940 2023-11-28T17:09:50.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.596+0000 7fa17bfff640 1 --2- 172.21.15.116:0/1723115037 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa18006b2a0 0x7fa18006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:50.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.596+0000 7fa17bfff640 1 -- 172.21.15.116:0/1723115037 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa1880abd40 con 0x7fa1940ce940 2023-11-28T17:09:50.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.596+0000 7fa1937fe640 1 --2- 172.21.15.116:0/1723115037 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa18006b2a0 0x7fa18006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:50.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.597+0000 7fa1937fe640 1 --2- 172.21.15.116:0/1723115037 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa18006b2a0 0x7fa18006d760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa1840011e0 tx=0x7fa184009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:50.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.600+0000 7fa17bfff640 1 -- 172.21.15.116:0/1723115037 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa188076c30 con 0x7fa1940ce940 2023-11-28T17:09:50.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.789+0000 7fa199c51640 1 -- 172.21.15.116:0/1723115037 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa19406cdc0 con 0x7fa1940ce940 2023-11-28T17:09:50.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.790+0000 7fa17bfff640 1 -- 172.21.15.116:0/1723115037 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa18807a250 con 0x7fa1940ce940 2023-11-28T17:09:50.792 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:09:50.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.792+0000 7fa199c51640 1 -- 172.21.15.116:0/1723115037 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa18006b2a0 msgr2=0x7fa18006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:50.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.792+0000 7fa199c51640 1 --2- 172.21.15.116:0/1723115037 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa18006b2a0 0x7fa18006d760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa1840011e0 tx=0x7fa184009040 comp rx=0 tx=0).stop 2023-11-28T17:09:50.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.792+0000 7fa199c51640 1 -- 172.21.15.116:0/1723115037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1940ce940 msgr2=0x7fa194146220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:50.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.793+0000 7fa199c51640 1 --2- 172.21.15.116:0/1723115037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1940ce940 0x7fa194146220 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa188034040 tx=0x7fa1880046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.793+0000 7fa199c51640 1 -- 172.21.15.116:0/1723115037 shutdown_connections 2023-11-28T17:09:50.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.793+0000 7fa199c51640 1 --2- 172.21.15.116:0/1723115037 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa18006b2a0 0x7fa18006d760 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.793+0000 7fa199c51640 1 --2- 172.21.15.116:0/1723115037 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa19415b0c0 0x7fa1940c3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.793+0000 7fa199c51640 1 --2- 172.21.15.116:0/1723115037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1940ce940 0x7fa194146220 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.793+0000 7fa199c51640 1 --2- 172.21.15.116:0/1723115037 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa194072250 0x7fa194159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.793+0000 7fa199c51640 1 -- 172.21.15.116:0/1723115037 >> 172.21.15.116:0/1723115037 conn(0x7fa1940b0cf0 msgr2=0x7fa19415cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.793+0000 7fa199c51640 1 -- 172.21.15.116:0/1723115037 shutdown_connections 2023-11-28T17:09:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:50.793+0000 7fa199c51640 1 -- 172.21.15.116:0/1723115037 wait complete. 2023-11-28T17:09:50.814 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:09:51.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.030+0000 7f1dee615640 1 Processor -- start 2023-11-28T17:09:51.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.030+0000 7f1dee615640 1 -- start start 2023-11-28T17:09:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.032+0000 7f1dee615640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1de8145ee0 0x7f1de80ac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.032+0000 7f1dee615640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1de8146540 0x7f1de8160e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.032+0000 7f1dee615640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de80b5110 con 0x7f1de8146a90 2023-11-28T17:09:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.032+0000 7f1dee615640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de8161dc0 con 0x7f1de8145ee0 2023-11-28T17:09:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.032+0000 7f1dee615640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de80b2190 con 0x7f1de8146540 2023-11-28T17:09:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.032+0000 7f1de7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1de8145ee0 0x7f1de80ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.032+0000 7f1de7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1de8145ee0 0x7f1de80ac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56408/0 (socket says 172.21.15.116:56408) 2023-11-28T17:09:51.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.032+0000 7f1de7fff640 1 -- 172.21.15.116:0/2503465982 learned_addr learned my addr 172.21.15.116:0/2503465982 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:51.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.033+0000 7f1de77fe640 1 --2- 172.21.15.116:0/2503465982 >> v2:172.21.15.156:3300/0 conn(0x7f1de8146540 0x7f1de8160e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.033+0000 7f1de7fff640 1 -- 172.21.15.116:0/2503465982 >> v2:172.21.15.156:3300/0 conn(0x7f1de8146540 msgr2=0x7f1de8160e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:51.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.033+0000 7f1de7fff640 1 --2- 172.21.15.116:0/2503465982 >> v2:172.21.15.156:3300/0 conn(0x7f1de8146540 0x7f1de8160e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.033+0000 7f1de7fff640 1 -- 172.21.15.116:0/2503465982 >> v1:172.21.15.156:6789/0 conn(0x7f1de8146a90 legacy=0x7f1de8071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:51.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.033+0000 7f1de7fff640 1 -- 172.21.15.116:0/2503465982 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de8161ae0 con 0x7f1de8145ee0 2023-11-28T17:09:51.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.033+0000 7f1de77fe640 1 --2- 172.21.15.116:0/2503465982 >> v2:172.21.15.156:3300/0 conn(0x7f1de8146540 0x7f1de8160e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:51.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.034+0000 7f1de7fff640 1 --2- 172.21.15.116:0/2503465982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8145ee0 0x7f1de80ac870 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1de8073e90 tx=0x7f1dd802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b5677532617d1b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:51.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.034+0000 7f1de6ffd640 1 -- 172.21.15.116:0/2503465982 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd802ca80 con 0x7f1de8145ee0 2023-11-28T17:09:51.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.034+0000 7f1de6ffd640 1 -- 172.21.15.116:0/2503465982 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd802cbe0 con 0x7f1de8145ee0 2023-11-28T17:09:51.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.034+0000 7f1de6ffd640 1 -- 172.21.15.116:0/2503465982 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd80343f0 con 0x7f1de8145ee0 2023-11-28T17:09:51.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.035+0000 7f1dee615640 1 -- 172.21.15.116:0/2503465982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8145ee0 msgr2=0x7f1de80ac870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.035+0000 7f1dee615640 1 --2- 172.21.15.116:0/2503465982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8145ee0 0x7f1de80ac870 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1de8073e90 tx=0x7f1dd802bee0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.035+0000 7f1dee615640 1 -- 172.21.15.116:0/2503465982 shutdown_connections 2023-11-28T17:09:51.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.035+0000 7f1dee615640 1 --2- 172.21.15.116:0/2503465982 >> v2:172.21.15.156:3300/0 conn(0x7f1de8146540 0x7f1de8160e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.035+0000 7f1dee615640 1 --2- 172.21.15.116:0/2503465982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8145ee0 0x7f1de80ac870 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.035+0000 7f1dee615640 1 -- 172.21.15.116:0/2503465982 >> 172.21.15.116:0/2503465982 conn(0x7f1de80b0cf0 msgr2=0x7f1de80b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:51.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.035+0000 7f1dee615640 1 -- 172.21.15.116:0/2503465982 shutdown_connections 2023-11-28T17:09:51.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.035+0000 7f1dee615640 1 -- 172.21.15.116:0/2503465982 wait complete. 2023-11-28T17:09:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.037+0000 7f1dee615640 1 Processor -- start 2023-11-28T17:09:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.037+0000 7f1dee615640 1 -- start start 2023-11-28T17:09:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.038+0000 7f1dee615640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1de8145ee0 0x7f1de8131100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.038+0000 7f1dee615640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8146540 0x7f1de80cd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.038+0000 7f1dee615640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1de8146a90 0x7f1de80af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.038+0000 7f1dee615640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80c42b0 con 0x7f1de8146540 2023-11-28T17:09:51.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.038+0000 7f1dee615640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de812e660 con 0x7f1de8145ee0 2023-11-28T17:09:51.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.038+0000 7f1dee615640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8131640 con 0x7f1de8146a90 2023-11-28T17:09:51.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.038+0000 7f1de77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8146540 0x7f1de80cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.038+0000 7f1de77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8146540 0x7f1de80cd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56410/0 (socket says 172.21.15.116:56410) 2023-11-28T17:09:51.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.038+0000 7f1de77fe640 1 -- 172.21.15.116:0/3434546669 learned_addr learned my addr 172.21.15.116:0/3434546669 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:51.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.039+0000 7f1de7fff640 1 --2- 172.21.15.116:0/3434546669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1de8145ee0 0x7f1de8131100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.039+0000 7f1decbb2640 1 --2- 172.21.15.116:0/3434546669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1de8146a90 0x7f1de80af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.039+0000 7f1de77fe640 1 -- 172.21.15.116:0/3434546669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1de8146a90 msgr2=0x7f1de80af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.039+0000 7f1de77fe640 1 --2- 172.21.15.116:0/3434546669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1de8146a90 0x7f1de80af0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.039+0000 7f1de77fe640 1 -- 172.21.15.116:0/3434546669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1de8145ee0 msgr2=0x7f1de8131100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.039+0000 7f1de77fe640 1 --2- 172.21.15.116:0/3434546669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1de8145ee0 0x7f1de8131100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.039+0000 7f1de77fe640 1 -- 172.21.15.116:0/3434546669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd8009b30 con 0x7f1de8146540 2023-11-28T17:09:51.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.039+0000 7f1de77fe640 1 --2- 172.21.15.116:0/3434546669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8146540 0x7f1de80cd860 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000c7f0 tx=0x7f1dd000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:51.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.040+0000 7f1de4ff9640 1 -- 172.21.15.116:0/3434546669 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0007820 con 0x7f1de8146540 2023-11-28T17:09:51.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.040+0000 7f1de4ff9640 1 -- 172.21.15.116:0/3434546669 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0007980 con 0x7f1de8146540 2023-11-28T17:09:51.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.040+0000 7f1dee615640 1 -- 172.21.15.116:0/3434546669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de81f63a0 con 0x7f1de8146540 2023-11-28T17:09:51.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.040+0000 7f1de4ff9640 1 -- 172.21.15.116:0/3434546669 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0007cf0 con 0x7f1de8146540 2023-11-28T17:09:51.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.040+0000 7f1dee615640 1 -- 172.21.15.116:0/3434546669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de81f6960 con 0x7f1de8146540 2023-11-28T17:09:51.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.041+0000 7f1dee615640 1 -- 172.21.15.116:0/3434546669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de8069360 con 0x7f1de8146540 2023-11-28T17:09:51.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.042+0000 7f1de4ff9640 1 -- 172.21.15.116:0/3434546669 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f1dd00113f0 con 0x7f1de8146540 2023-11-28T17:09:51.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.042+0000 7f1de4ff9640 1 --2- 172.21.15.116:0/3434546669 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1dc806b2d0 0x7f1dc806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.043+0000 7f1de4ff9640 1 -- 172.21.15.116:0/3434546669 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1dd0089fb0 con 0x7f1de8146540 2023-11-28T17:09:51.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.043+0000 7f1de7fff640 1 --2- 172.21.15.116:0/3434546669 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1dc806b2d0 0x7f1dc806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.043+0000 7f1de7fff640 1 --2- 172.21.15.116:0/3434546669 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1dc806b2d0 0x7f1dc806d790 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8035000 tx=0x7f1dd8030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:51.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.047+0000 7f1de4ff9640 1 -- 172.21.15.116:0/3434546669 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1dd0054d60 con 0x7f1de8146540 2023-11-28T17:09:51.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.238+0000 7f1dee615640 1 -- 172.21.15.116:0/3434546669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1de8140490 con 0x7f1de8146540 2023-11-28T17:09:51.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.239+0000 7f1de4ff9640 1 -- 172.21.15.116:0/3434546669 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1dd001f020 con 0x7f1de8146540 2023-11-28T17:09:51.240 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:09:51.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.241+0000 7f1dee615640 1 -- 172.21.15.116:0/3434546669 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1dc806b2d0 msgr2=0x7f1dc806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.241+0000 7f1dee615640 1 --2- 172.21.15.116:0/3434546669 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1dc806b2d0 0x7f1dc806d790 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8035000 tx=0x7f1dd8030000 comp rx=0 tx=0).stop 2023-11-28T17:09:51.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.241+0000 7f1dee615640 1 -- 172.21.15.116:0/3434546669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8146540 msgr2=0x7f1de80cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.241+0000 7f1dee615640 1 --2- 172.21.15.116:0/3434546669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8146540 0x7f1de80cd860 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000c7f0 tx=0x7f1dd000ccb0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.242+0000 7f1dee615640 1 -- 172.21.15.116:0/3434546669 shutdown_connections 2023-11-28T17:09:51.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.242+0000 7f1dee615640 1 --2- 172.21.15.116:0/3434546669 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1dc806b2d0 0x7f1dc806d790 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.242+0000 7f1dee615640 1 --2- 172.21.15.116:0/3434546669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1de8146a90 0x7f1de80af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.242+0000 7f1dee615640 1 --2- 172.21.15.116:0/3434546669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8146540 0x7f1de80cd860 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.242+0000 7f1dee615640 1 --2- 172.21.15.116:0/3434546669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1de8145ee0 0x7f1de8131100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.242+0000 7f1dee615640 1 -- 172.21.15.116:0/3434546669 >> 172.21.15.116:0/3434546669 conn(0x7f1de80b0cf0 msgr2=0x7f1de80ad460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:51.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.242+0000 7f1dee615640 1 -- 172.21.15.116:0/3434546669 shutdown_connections 2023-11-28T17:09:51.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.242+0000 7f1dee615640 1 -- 172.21.15.116:0/3434546669 wait complete. 2023-11-28T17:09:51.262 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:09:51.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.467+0000 7fce4851d640 1 Processor -- start 2023-11-28T17:09:51.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.467+0000 7fce4851d640 1 -- start start 2023-11-28T17:09:51.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.469+0000 7fce4851d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce4013cee0 0x7fce4013d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.469+0000 7fce4851d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fce4013f640 0x7fce4013fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.469+0000 7fce4851d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4013d2c0 con 0x7fce40071d20 2023-11-28T17:09:51.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.469+0000 7fce4851d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce401400e0 con 0x7fce4013cee0 2023-11-28T17:09:51.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.469+0000 7fce4851d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce401403c0 con 0x7fce4013f640 2023-11-28T17:09:51.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.469+0000 7fce462b9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce4013cee0 0x7fce4013d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.469+0000 7fce46aba640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fce40071d20 0x7fce40072130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51466/0 (socket says 172.21.15.116:51466) 2023-11-28T17:09:51.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.469+0000 7fce46aba640 1 -- 172.21.15.116:0/3962157339 learned_addr learned my addr 172.21.15.116:0/3962157339 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:51.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.469+0000 7fce45ab8640 1 --2- 172.21.15.116:0/3962157339 >> v2:172.21.15.156:3300/0 conn(0x7fce4013f640 0x7fce4013fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.470+0000 7fce462b9640 1 -- 172.21.15.116:0/3962157339 >> v2:172.21.15.156:3300/0 conn(0x7fce4013f640 msgr2=0x7fce4013fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:51.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.470+0000 7fce462b9640 1 --2- 172.21.15.116:0/3962157339 >> v2:172.21.15.156:3300/0 conn(0x7fce4013f640 0x7fce4013fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.470+0000 7fce462b9640 1 -- 172.21.15.116:0/3962157339 >> v1:172.21.15.116:6789/0 conn(0x7fce40071d20 legacy=0x7fce40072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.470+0000 7fce462b9640 1 -- 172.21.15.116:0/3962157339 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce400cdb50 con 0x7fce4013cee0 2023-11-28T17:09:51.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.470+0000 7fce45ab8640 1 --2- 172.21.15.116:0/3962157339 >> v2:172.21.15.156:3300/0 conn(0x7fce4013f640 0x7fce4013fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:51.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.471+0000 7fce462b9640 1 --2- 172.21.15.116:0/3962157339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4013cee0 0x7fce4013d600 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fce40143c70 tx=0x7fce3c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7848a676f230c706 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:51.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.471+0000 7fce452b7640 1 -- 172.21.15.116:0/3962157339 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3c031710 con 0x7fce4013cee0 2023-11-28T17:09:51.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.472+0000 7fce452b7640 1 -- 172.21.15.116:0/3962157339 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3c02f050 con 0x7fce4013cee0 2023-11-28T17:09:51.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.472+0000 7fce452b7640 1 -- 172.21.15.116:0/3962157339 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3c031a90 con 0x7fce4013cee0 2023-11-28T17:09:51.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.472+0000 7fce4851d640 1 -- 172.21.15.116:0/3962157339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4013cee0 msgr2=0x7fce4013d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.472+0000 7fce4851d640 1 --2- 172.21.15.116:0/3962157339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4013cee0 0x7fce4013d600 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fce40143c70 tx=0x7fce3c02b7b0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.472+0000 7fce4851d640 1 -- 172.21.15.116:0/3962157339 shutdown_connections 2023-11-28T17:09:51.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.472+0000 7fce4851d640 1 --2- 172.21.15.116:0/3962157339 >> v2:172.21.15.156:3300/0 conn(0x7fce4013f640 0x7fce4013fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.472+0000 7fce4851d640 1 --2- 172.21.15.116:0/3962157339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4013cee0 0x7fce4013d600 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.472+0000 7fce4851d640 1 -- 172.21.15.116:0/3962157339 >> 172.21.15.116:0/3962157339 conn(0x7fce400b0cf0 msgr2=0x7fce400b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:51.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.473+0000 7fce4851d640 1 -- 172.21.15.116:0/3962157339 shutdown_connections 2023-11-28T17:09:51.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.473+0000 7fce4851d640 1 -- 172.21.15.116:0/3962157339 wait complete. 2023-11-28T17:09:51.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.474+0000 7fce4851d640 1 Processor -- start 2023-11-28T17:09:51.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.474+0000 7fce4851d640 1 -- start start 2023-11-28T17:09:51.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.475+0000 7fce4851d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce40071d20 0x7fce400b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.475+0000 7fce4851d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4013cee0 0x7fce400b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.475+0000 7fce4851d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce4013f640 0x7fce400ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.475+0000 7fce4851d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4013c030 con 0x7fce4013cee0 2023-11-28T17:09:51.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.475+0000 7fce4851d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4013ab90 con 0x7fce4013f640 2023-11-28T17:09:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.475+0000 7fce4851d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce40143000 con 0x7fce40071d20 2023-11-28T17:09:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.476+0000 7fce45ab8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4013cee0 0x7fce400b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.476+0000 7fce45ab8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4013cee0 0x7fce400b3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56424/0 (socket says 172.21.15.116:56424) 2023-11-28T17:09:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.476+0000 7fce45ab8640 1 -- 172.21.15.116:0/978596547 learned_addr learned my addr 172.21.15.116:0/978596547 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.476+0000 7fce462b9640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce40071d20 0x7fce400b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.476+0000 7fce46aba640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce4013f640 0x7fce400ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.476+0000 7fce45ab8640 1 -- 172.21.15.116:0/978596547 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce40071d20 msgr2=0x7fce400b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.476+0000 7fce45ab8640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce40071d20 0x7fce400b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.476+0000 7fce45ab8640 1 -- 172.21.15.116:0/978596547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce4013f640 msgr2=0x7fce400ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.476+0000 7fce45ab8640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce4013f640 0x7fce400ad8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.476+0000 7fce45ab8640 1 -- 172.21.15.116:0/978596547 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c009400 con 0x7fce4013cee0 2023-11-28T17:09:51.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.476+0000 7fce462b9640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce40071d20 0x7fce400b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:51.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.476+0000 7fce45ab8640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4013cee0 0x7fce400b3850 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fce3000c7f0 tx=0x7fce3000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:51.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.477+0000 7fce36ffd640 1 -- 172.21.15.116:0/978596547 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30007820 con 0x7fce4013cee0 2023-11-28T17:09:51.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.477+0000 7fce36ffd640 1 -- 172.21.15.116:0/978596547 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce30007980 con 0x7fce4013cee0 2023-11-28T17:09:51.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.477+0000 7fce36ffd640 1 -- 172.21.15.116:0/978596547 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce30007cf0 con 0x7fce4013cee0 2023-11-28T17:09:51.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.477+0000 7fce4851d640 1 -- 172.21.15.116:0/978596547 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce401432e0 con 0x7fce4013cee0 2023-11-28T17:09:51.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.477+0000 7fce4851d640 1 -- 172.21.15.116:0/978596547 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce401435c0 con 0x7fce4013cee0 2023-11-28T17:09:51.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.478+0000 7fce4851d640 1 -- 172.21.15.116:0/978596547 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4013ad20 con 0x7fce4013cee0 2023-11-28T17:09:51.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.479+0000 7fce36ffd640 1 -- 172.21.15.116:0/978596547 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fce300113f0 con 0x7fce4013cee0 2023-11-28T17:09:51.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.480+0000 7fce36ffd640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fce2406b2d0 0x7fce2406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.480+0000 7fce36ffd640 1 -- 172.21.15.116:0/978596547 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fce30089f20 con 0x7fce4013cee0 2023-11-28T17:09:51.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.480+0000 7fce462b9640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fce2406b2d0 0x7fce2406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.481+0000 7fce462b9640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fce2406b2d0 0x7fce2406d790 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fce3c035000 tx=0x7fce3c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:51.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.485+0000 7fce36ffd640 1 -- 172.21.15.116:0/978596547 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce30054d60 con 0x7fce4013cee0 2023-11-28T17:09:51.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.674+0000 7fce4851d640 1 -- 172.21.15.116:0/978596547 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fce40069360 con 0x7fce4013cee0 2023-11-28T17:09:51.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.675+0000 7fce36ffd640 1 -- 172.21.15.116:0/978596547 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fce3001f0a0 con 0x7fce4013cee0 2023-11-28T17:09:51.677 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:09:51.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.678+0000 7fce4851d640 1 -- 172.21.15.116:0/978596547 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fce2406b2d0 msgr2=0x7fce2406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.678+0000 7fce4851d640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fce2406b2d0 0x7fce2406d790 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fce3c035000 tx=0x7fce3c008030 comp rx=0 tx=0).stop 2023-11-28T17:09:51.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.678+0000 7fce4851d640 1 -- 172.21.15.116:0/978596547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4013cee0 msgr2=0x7fce400b3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.678+0000 7fce4851d640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4013cee0 0x7fce400b3850 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fce3000c7f0 tx=0x7fce3000ccb0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.678+0000 7fce4851d640 1 -- 172.21.15.116:0/978596547 shutdown_connections 2023-11-28T17:09:51.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.678+0000 7fce4851d640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fce2406b2d0 0x7fce2406d790 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.678+0000 7fce4851d640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce4013f640 0x7fce400ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.678+0000 7fce4851d640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce4013cee0 0x7fce400b3850 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.678+0000 7fce4851d640 1 --2- 172.21.15.116:0/978596547 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce40071d20 0x7fce400b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.678+0000 7fce4851d640 1 -- 172.21.15.116:0/978596547 >> 172.21.15.116:0/978596547 conn(0x7fce400b0cf0 msgr2=0x7fce4013ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:51.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.678+0000 7fce4851d640 1 -- 172.21.15.116:0/978596547 shutdown_connections 2023-11-28T17:09:51.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.678+0000 7fce4851d640 1 -- 172.21.15.116:0/978596547 wait complete. 2023-11-28T17:09:51.699 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:09:51.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.912+0000 7fdea1589640 1 Processor -- start 2023-11-28T17:09:51.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.912+0000 7fdea1589640 1 -- start start 2023-11-28T17:09:51.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.913+0000 7fdea1589640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde9c15d2c0 0x7fde9c0cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.913+0000 7fdea1589640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde9c0d2f40 0x7fde9c0b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.913+0000 7fdea1589640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9c0b6640 con 0x7fde9c0cdf70 2023-11-28T17:09:51.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.913+0000 7fdea1589640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde9c0ce420 con 0x7fde9c15d2c0 2023-11-28T17:09:51.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.913+0000 7fdea1589640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde9c0d3480 con 0x7fde9c0d2f40 2023-11-28T17:09:51.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.914+0000 7fde9affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde9c15d2c0 0x7fde9c0cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.914+0000 7fde9a7fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fde9c0cdf70 0x7fde9c0d2830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51478/0 (socket says 172.21.15.116:51478) 2023-11-28T17:09:51.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.914+0000 7fde9a7fc640 1 -- 172.21.15.116:0/1798067316 learned_addr learned my addr 172.21.15.116:0/1798067316 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:51.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.914+0000 7fde9b7fe640 1 --2- 172.21.15.116:0/1798067316 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde9c0d2f40 0x7fde9c0b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.914+0000 7fde9affd640 1 -- 172.21.15.116:0/1798067316 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde9c0d2f40 msgr2=0x7fde9c0b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:51.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.914+0000 7fde9affd640 1 --2- 172.21.15.116:0/1798067316 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde9c0d2f40 0x7fde9c0b6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.914+0000 7fde9affd640 1 -- 172.21.15.116:0/1798067316 >> v1:172.21.15.116:6789/0 conn(0x7fde9c0cdf70 legacy=0x7fde9c0d2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.914+0000 7fde9affd640 1 -- 172.21.15.116:0/1798067316 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9c0d3760 con 0x7fde9c15d2c0 2023-11-28T17:09:51.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.914+0000 7fde9b7fe640 1 --2- 172.21.15.116:0/1798067316 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde9c0d2f40 0x7fde9c0b6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:51.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.916+0000 7fde9affd640 1 --2- 172.21.15.116:0/1798067316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9c15d2c0 0x7fde9c0cda30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0097a0 tx=0x7fde8c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3246585487897ccb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:51.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.916+0000 7fde99ffb640 1 -- 172.21.15.116:0/1798067316 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c02d040 con 0x7fde9c15d2c0 2023-11-28T17:09:51.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.916+0000 7fde99ffb640 1 -- 172.21.15.116:0/1798067316 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8c031810 con 0x7fde9c15d2c0 2023-11-28T17:09:51.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.916+0000 7fde99ffb640 1 -- 172.21.15.116:0/1798067316 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c031ba0 con 0x7fde9c15d2c0 2023-11-28T17:09:51.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.917+0000 7fdea1589640 1 -- 172.21.15.116:0/1798067316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9c15d2c0 msgr2=0x7fde9c0cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.917+0000 7fdea1589640 1 --2- 172.21.15.116:0/1798067316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9c15d2c0 0x7fde9c0cda30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0097a0 tx=0x7fde8c02b8b0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.917+0000 7fdea1589640 1 -- 172.21.15.116:0/1798067316 shutdown_connections 2023-11-28T17:09:51.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.917+0000 7fdea1589640 1 --2- 172.21.15.116:0/1798067316 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde9c0d2f40 0x7fde9c0b6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.917+0000 7fdea1589640 1 --2- 172.21.15.116:0/1798067316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9c15d2c0 0x7fde9c0cda30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.917+0000 7fdea1589640 1 -- 172.21.15.116:0/1798067316 >> 172.21.15.116:0/1798067316 conn(0x7fde9c072010 msgr2=0x7fde9c072420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:51.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.917+0000 7fdea1589640 1 -- 172.21.15.116:0/1798067316 shutdown_connections 2023-11-28T17:09:51.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.917+0000 7fdea1589640 1 -- 172.21.15.116:0/1798067316 wait complete. 2023-11-28T17:09:51.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.919+0000 7fdea1589640 1 Processor -- start 2023-11-28T17:09:51.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.919+0000 7fdea1589640 1 -- start start 2023-11-28T17:09:51.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.920+0000 7fdea1589640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9c0cdf70 0x7fde9c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.920+0000 7fdea1589640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde9c0d2f40 0x7fde9c15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.920+0000 7fdea1589640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde9c15d2c0 0x7fde9c0c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.920+0000 7fdea1589640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9c0c9420 con 0x7fde9c0cdf70 2023-11-28T17:09:51.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.920+0000 7fdea1589640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9c0c87a0 con 0x7fde9c0d2f40 2023-11-28T17:09:51.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.920+0000 7fdea1589640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde9c0c8a80 con 0x7fde9c15d2c0 2023-11-28T17:09:51.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.920+0000 7fde9affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9c0cdf70 0x7fde9c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.920+0000 7fde9affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9c0cdf70 0x7fde9c0d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56456/0 (socket says 172.21.15.116:56456) 2023-11-28T17:09:51.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.920+0000 7fde9affd640 1 -- 172.21.15.116:0/2889012690 learned_addr learned my addr 172.21.15.116:0/2889012690 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:51.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.920+0000 7fde9affd640 1 -- 172.21.15.116:0/2889012690 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde9c15d2c0 msgr2=0x7fde9c0c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.920+0000 7fde9affd640 1 --2- 172.21.15.116:0/2889012690 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde9c15d2c0 0x7fde9c0c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.921+0000 7fde9affd640 1 -- 172.21.15.116:0/2889012690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde9c0d2f40 msgr2=0x7fde9c15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:51.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.921+0000 7fde9affd640 1 --2- 172.21.15.116:0/2889012690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde9c0d2f40 0x7fde9c15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:51.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.921+0000 7fde9affd640 1 -- 172.21.15.116:0/2889012690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c009400 con 0x7fde9c0cdf70 2023-11-28T17:09:51.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.921+0000 7fde9affd640 1 --2- 172.21.15.116:0/2889012690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9c0cdf70 0x7fde9c0d1f20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fde8c000c00 tx=0x7fde8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:51.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.921+0000 7fde937fe640 1 -- 172.21.15.116:0/2889012690 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c03ad60 con 0x7fde9c0cdf70 2023-11-28T17:09:51.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.921+0000 7fde937fe640 1 -- 172.21.15.116:0/2889012690 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8c0443f0 con 0x7fde9c0cdf70 2023-11-28T17:09:51.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.921+0000 7fdea1589640 1 -- 172.21.15.116:0/2889012690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde9c0c8d60 con 0x7fde9c0cdf70 2023-11-28T17:09:51.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.921+0000 7fde937fe640 1 -- 172.21.15.116:0/2889012690 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c0447a0 con 0x7fde9c0cdf70 2023-11-28T17:09:51.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.921+0000 7fdea1589640 1 -- 172.21.15.116:0/2889012690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde9c0d2460 con 0x7fde9c0cdf70 2023-11-28T17:09:51.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.923+0000 7fdea1589640 1 -- 172.21.15.116:0/2889012690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde9c0693d0 con 0x7fde9c0cdf70 2023-11-28T17:09:51.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.923+0000 7fde937fe640 1 -- 172.21.15.116:0/2889012690 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fde8c04d450 con 0x7fde9c0cdf70 2023-11-28T17:09:51.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.925+0000 7fde937fe640 1 --2- 172.21.15.116:0/2889012690 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fde8006b2a0 0x7fde8006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:51.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.926+0000 7fde937fe640 1 -- 172.21.15.116:0/2889012690 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fde8c0abf70 con 0x7fde9c0cdf70 2023-11-28T17:09:51.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.926+0000 7fde9a7fc640 1 --2- 172.21.15.116:0/2889012690 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fde8006b2a0 0x7fde8006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:51.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.926+0000 7fde9a7fc640 1 --2- 172.21.15.116:0/2889012690 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fde8006b2a0 0x7fde8006d760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fde9c1f6500 tx=0x7fde84001660 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:51.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:51.929+0000 7fde937fe640 1 -- 172.21.15.116:0/2889012690 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde8c0dee90 con 0x7fde9c0cdf70 2023-11-28T17:09:52.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.135+0000 7fdea1589640 1 -- 172.21.15.116:0/2889012690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fde9c15cb50 con 0x7fde9c0cdf70 2023-11-28T17:09:52.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.136+0000 7fde937fe640 1 -- 172.21.15.116:0/2889012690 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fde8c076dc0 con 0x7fde9c0cdf70 2023-11-28T17:09:52.138 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-28T17:09:52.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.139+0000 7fdea1589640 1 -- 172.21.15.116:0/2889012690 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fde8006b2a0 msgr2=0x7fde8006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:52.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.139+0000 7fdea1589640 1 --2- 172.21.15.116:0/2889012690 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fde8006b2a0 0x7fde8006d760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fde9c1f6500 tx=0x7fde84001660 comp rx=0 tx=0).stop 2023-11-28T17:09:52.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.139+0000 7fdea1589640 1 -- 172.21.15.116:0/2889012690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9c0cdf70 msgr2=0x7fde9c0d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:52.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.139+0000 7fdea1589640 1 --2- 172.21.15.116:0/2889012690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9c0cdf70 0x7fde9c0d1f20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fde8c000c00 tx=0x7fde8c0046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.139+0000 7fdea1589640 1 -- 172.21.15.116:0/2889012690 shutdown_connections 2023-11-28T17:09:52.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.139+0000 7fdea1589640 1 --2- 172.21.15.116:0/2889012690 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fde8006b2a0 0x7fde8006d760 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.139+0000 7fdea1589640 1 --2- 172.21.15.116:0/2889012690 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde9c15d2c0 0x7fde9c0c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.139+0000 7fdea1589640 1 --2- 172.21.15.116:0/2889012690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde9c0d2f40 0x7fde9c15abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.139+0000 7fdea1589640 1 --2- 172.21.15.116:0/2889012690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde9c0cdf70 0x7fde9c0d1f20 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.139+0000 7fdea1589640 1 -- 172.21.15.116:0/2889012690 >> 172.21.15.116:0/2889012690 conn(0x7fde9c072010 msgr2=0x7fde9c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:52.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.139+0000 7fdea1589640 1 -- 172.21.15.116:0/2889012690 shutdown_connections 2023-11-28T17:09:52.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.139+0000 7fdea1589640 1 -- 172.21.15.116:0/2889012690 wait complete. 2023-11-28T17:09:52.160 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2023-11-28T17:09:52.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:52.161 DEBUG:teuthology.orchestra.run.smithi156:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-11-28T17:09:52.263 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:09:52.263 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:09:52.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:52.263 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-28T17:09:52.336 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:09:52.337 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:09:52.396 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:09:52.396 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:09:52.450 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:52.530 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:09:52.536 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:09:52.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.653+0000 7f84bd031640 1 Processor -- start 2023-11-28T17:09:52.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.653+0000 7f84bd031640 1 -- start start 2023-11-28T17:09:52.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.655+0000 7f84bd031640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b806c250 0x7f84b806c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:52.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.655+0000 7f84bd031640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b815fc10 con 0x7f84b815f7b0 2023-11-28T17:09:52.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.655+0000 7f84bd031640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b8070bd0 con 0x7f84b8144430 2023-11-28T17:09:52.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.655+0000 7f84bd031640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b8144810 con 0x7f84b806c250 2023-11-28T17:09:52.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.655+0000 7f84affff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f84b815f7b0 0x7f84b8071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51480/0 (socket says 172.21.15.116:51480) 2023-11-28T17:09:52.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.655+0000 7f84affff640 1 -- 172.21.15.116:0/1735654334 learned_addr learned my addr 172.21.15.116:0/1735654334 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:52.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b759e640 1 --2- 172.21.15.116:0/1735654334 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b806c250 0x7f84b806c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:52.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 -- 172.21.15.116:0/1735654334 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462196042 0 0) 0x7f84b815fc10 con 0x7f84b815f7b0 2023-11-28T17:09:52.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 -- 172.21.15.116:0/1735654334 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a4003520 con 0x7f84b815f7b0 2023-11-28T17:09:52.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 -- 172.21.15.116:0/1735654334 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3544322595 0 0) 0x7f84a4003520 con 0x7f84b815f7b0 2023-11-28T17:09:52.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 -- 172.21.15.116:0/1735654334 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b806c250 msgr2=0x7f84b806c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:52.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 --2- 172.21.15.116:0/1735654334 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b806c250 0x7f84b806c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 -- 172.21.15.116:0/1735654334 >> v1:172.21.15.156:6789/0 conn(0x7f84b8144430 legacy=0x7f84b80703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:52.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 -- 172.21.15.116:0/1735654334 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b8070eb0 con 0x7f84b815f7b0 2023-11-28T17:09:52.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 -- 172.21.15.116:0/1735654334 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f84a0001670 con 0x7f84b815f7b0 2023-11-28T17:09:52.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 -- 172.21.15.116:0/1735654334 >> v1:172.21.15.116:6789/0 conn(0x7f84b815f7b0 legacy=0x7f84b8071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 --2- 172.21.15.116:0/1735654334 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b806c250 0x7f84a40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 --2- 172.21.15.116:0/1735654334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84a4003ee0 0x7f84a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 --2- 172.21.15.116:0/1735654334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a4004d00 0x7f84a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 -- 172.21.15.116:0/1735654334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b8144810 con 0x7f84a4004d00 2023-11-28T17:09:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 -- 172.21.15.116:0/1735654334 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a400ba30 con 0x7f84a4003ee0 2023-11-28T17:09:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.656+0000 7f84b659c640 1 -- 172.21.15.116:0/1735654334 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a400bbf0 con 0x7f84b806c250 2023-11-28T17:09:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.657+0000 7f84b759e640 1 --2- 172.21.15.116:0/1735654334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a4004d00 0x7f84a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.657+0000 7f84affff640 1 --2- 172.21.15.116:0/1735654334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84a4003ee0 0x7f84a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.657+0000 7f84b6d9d640 1 --2- 172.21.15.116:0/1735654334 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b806c250 0x7f84a40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.657+0000 7f84b759e640 1 -- 172.21.15.116:0/1735654334 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b806c250 msgr2=0x7f84a40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.657+0000 7f84b759e640 1 --2- 172.21.15.116:0/1735654334 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84b806c250 0x7f84a40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.657+0000 7f84b759e640 1 -- 172.21.15.116:0/1735654334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84a4003ee0 msgr2=0x7f84a40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.657+0000 7f84b759e640 1 --2- 172.21.15.116:0/1735654334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84a4003ee0 0x7f84a40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.657+0000 7f84b759e640 1 -- 172.21.15.116:0/1735654334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a4003520 con 0x7f84a4004d00 2023-11-28T17:09:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.658+0000 7f84b759e640 1 --2- 172.21.15.116:0/1735654334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a4004d00 0x7f84a4005180 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009fc0 tx=0x7f84a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b0ee921d3a6172 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.658+0000 7f84b659c640 1 -- 172.21.15.116:0/1735654334 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b002cc20 con 0x7f84a4004d00 2023-11-28T17:09:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.659+0000 7f84b659c640 1 -- 172.21.15.116:0/1735654334 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b000f040 con 0x7f84a4004d00 2023-11-28T17:09:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.659+0000 7f84b659c640 1 -- 172.21.15.116:0/1735654334 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b0034440 con 0x7f84a4004d00 2023-11-28T17:09:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.659+0000 7f84bd031640 1 -- 172.21.15.116:0/1735654334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a4004d00 msgr2=0x7f84a4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.659+0000 7f84bd031640 1 --2- 172.21.15.116:0/1735654334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84a4004d00 0x7f84a4005180 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009fc0 tx=0x7f84a40057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.659+0000 7f84b759e640 1 -- 172.21.15.116:0/1735654334 reap_dead start 2023-11-28T17:09:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.659+0000 7f84bd031640 1 -- 172.21.15.116:0/1735654334 shutdown_connections 2023-11-28T17:09:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.659+0000 7f84bd031640 1 -- 172.21.15.116:0/1735654334 >> 172.21.15.116:0/1735654334 conn(0x7f84b806a4c0 msgr2=0x7f84b806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.659+0000 7f84bd031640 1 -- 172.21.15.116:0/1735654334 shutdown_connections 2023-11-28T17:09:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.659+0000 7f84bd031640 1 -- 172.21.15.116:0/1735654334 wait complete. 2023-11-28T17:09:52.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.661+0000 7f84bd031640 1 Processor -- start 2023-11-28T17:09:52.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.661+0000 7f84bd031640 1 -- start start 2023-11-28T17:09:52.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.662+0000 7f84bd031640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84a4004d00 0x7f84b80a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:52.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.662+0000 7f84bd031640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84b815cfa0 0x7f84b815e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:52.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.662+0000 7f84bd031640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84b815a990 0x7f84b81414d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:52.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.662+0000 7f84bd031640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b815e040 con 0x7f84b815cfa0 2023-11-28T17:09:52.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.662+0000 7f84bd031640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b815d430 con 0x7f84b815a990 2023-11-28T17:09:52.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.662+0000 7f84bd031640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b815d710 con 0x7f84a4004d00 2023-11-28T17:09:52.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.662+0000 7f84affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84b815cfa0 0x7f84b815e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:52.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.662+0000 7f84affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84b815cfa0 0x7f84b815e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56466/0 (socket says 172.21.15.116:56466) 2023-11-28T17:09:52.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.662+0000 7f84affff640 1 -- 172.21.15.116:0/3661937635 learned_addr learned my addr 172.21.15.116:0/3661937635 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:52.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.663+0000 7f84b6d9d640 1 --2- 172.21.15.116:0/3661937635 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84a4004d00 0x7f84b80a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.663+0000 7f84affff640 1 -- 172.21.15.116:0/3661937635 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84a4004d00 msgr2=0x7f84b80a8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.663+0000 7f84affff640 1 --2- 172.21.15.116:0/3661937635 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84a4004d00 0x7f84b80a8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.663+0000 7f84affff640 1 -- 172.21.15.116:0/3661937635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84b815a990 msgr2=0x7f84b81414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.663+0000 7f84affff640 1 --2- 172.21.15.116:0/3661937635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84b815a990 0x7f84b81414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.663+0000 7f84affff640 1 -- 172.21.15.116:0/3661937635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b815d9f0 con 0x7f84b815cfa0 2023-11-28T17:09:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.663+0000 7f84affff640 1 --2- 172.21.15.116:0/3661937635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84b815cfa0 0x7f84b815e4d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f84a0001050 tx=0x7f84a000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.663+0000 7f84af7fe640 1 -- 172.21.15.116:0/3661937635 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a000c7f0 con 0x7f84b815cfa0 2023-11-28T17:09:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.663+0000 7f84af7fe640 1 -- 172.21.15.116:0/3661937635 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a000c950 con 0x7f84b815cfa0 2023-11-28T17:09:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.663+0000 7f84bd031640 1 -- 172.21.15.116:0/3661937635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b815dcd0 con 0x7f84b815cfa0 2023-11-28T17:09:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.663+0000 7f84af7fe640 1 -- 172.21.15.116:0/3661937635 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a000cd00 con 0x7f84b815cfa0 2023-11-28T17:09:52.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.664+0000 7f84bd031640 1 -- 172.21.15.116:0/3661937635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b813ebf0 con 0x7f84b815cfa0 2023-11-28T17:09:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.665+0000 7f84bd031640 1 -- 172.21.15.116:0/3661937635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b815e1f0 con 0x7f84b815cfa0 2023-11-28T17:09:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.666+0000 7f84af7fe640 1 -- 172.21.15.116:0/3661937635 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f84a001d460 con 0x7f84b815cfa0 2023-11-28T17:09:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.666+0000 7f84af7fe640 1 --2- 172.21.15.116:0/3661937635 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f849806b2a0 0x7f849806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:52.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.667+0000 7f84af7fe640 1 -- 172.21.15.116:0/3661937635 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f84a0089bb0 con 0x7f84b815cfa0 2023-11-28T17:09:52.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.667+0000 7f84b6d9d640 1 --2- 172.21.15.116:0/3661937635 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f849806b2a0 0x7f849806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:52.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.667+0000 7f84b6d9d640 1 --2- 172.21.15.116:0/3661937635 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f849806b2a0 0x7f849806d760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84a8005540 tx=0x7f84a8007c50 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:52.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.671+0000 7f84af7fe640 1 -- 172.21.15.116:0/3661937635 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f84b815e1f0 con 0x7f84b815cfa0 2023-11-28T17:09:52.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.863+0000 7f84bd031640 1 -- 172.21.15.116:0/3661937635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84b80693d0 con 0x7f84b815cfa0 2023-11-28T17:09:52.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.865+0000 7f84af7fe640 1 -- 172.21.15.116:0/3661937635 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f84a0054a00 con 0x7f84b815cfa0 2023-11-28T17:09:52.867 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:52.867 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:43.049465+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.868+0000 7f84bd031640 1 -- 172.21.15.116:0/3661937635 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f849806b2a0 msgr2=0x7f849806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.868+0000 7f84bd031640 1 --2- 172.21.15.116:0/3661937635 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f849806b2a0 0x7f849806d760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84a8005540 tx=0x7f84a8007c50 comp rx=0 tx=0).stop 2023-11-28T17:09:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.868+0000 7f84bd031640 1 -- 172.21.15.116:0/3661937635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84b815cfa0 msgr2=0x7f84b815e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.868+0000 7f84bd031640 1 --2- 172.21.15.116:0/3661937635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84b815cfa0 0x7f84b815e4d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f84a0001050 tx=0x7f84a000ed50 comp rx=0 tx=0).stop 2023-11-28T17:09:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.868+0000 7f84bd031640 1 -- 172.21.15.116:0/3661937635 shutdown_connections 2023-11-28T17:09:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.868+0000 7f84bd031640 1 --2- 172.21.15.116:0/3661937635 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f849806b2a0 0x7f849806d760 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.868+0000 7f84bd031640 1 --2- 172.21.15.116:0/3661937635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84b815a990 0x7f84b81414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.868+0000 7f84bd031640 1 --2- 172.21.15.116:0/3661937635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84b815cfa0 0x7f84b815e4d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.868+0000 7f84bd031640 1 --2- 172.21.15.116:0/3661937635 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84a4004d00 0x7f84b80a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:52.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.868+0000 7f84bd031640 1 -- 172.21.15.116:0/3661937635 >> 172.21.15.116:0/3661937635 conn(0x7f84b806a4c0 msgr2=0x7f84b80c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.868+0000 7f84bd031640 1 -- 172.21.15.116:0/3661937635 shutdown_connections 2023-11-28T17:09:52.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:52.868+0000 7f84bd031640 1 -- 172.21.15.116:0/3661937635 wait complete. 2023-11-28T17:09:52.887 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:09:52.887 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:09:53.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.146+0000 7f63f41d5640 1 Processor -- start 2023-11-28T17:09:53.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.146+0000 7f63f41d5640 1 -- start start 2023-11-28T17:09:53.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.147+0000 7f63f41d5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f63ec072250 0x7f63ec15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.147+0000 7f63f41d5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ec0cd320 con 0x7f63ec15b0c0 2023-11-28T17:09:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.147+0000 7f63f41d5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ec0ced90 con 0x7f63ec0ce940 2023-11-28T17:09:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.147+0000 7f63f41d5640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63ec0cc710 con 0x7f63ec072250 2023-11-28T17:09:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.148+0000 7f63f1f71640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f63ec15b0c0 0x7f63ec071b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51484/0 (socket says 172.21.15.116:51484) 2023-11-28T17:09:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.148+0000 7f63f1f71640 1 -- 172.21.15.116:0/741453643 learned_addr learned my addr 172.21.15.116:0/741453643 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.148+0000 7f63f1770640 1 --2- 172.21.15.116:0/741453643 >> v2:172.21.15.116:3300/0 conn(0x7f63ec072250 0x7f63ec15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:53.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.148+0000 7f63f0f6f640 1 -- 172.21.15.116:0/741453643 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820487554 0 0) 0x7f63ec0cd320 con 0x7f63ec15b0c0 2023-11-28T17:09:53.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.148+0000 7f63f0f6f640 1 -- 172.21.15.116:0/741453643 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d8003520 con 0x7f63ec15b0c0 2023-11-28T17:09:53.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.148+0000 7f63f1770640 1 -- 172.21.15.116:0/741453643 >> v1:172.21.15.156:6789/0 conn(0x7f63ec0ce940 legacy=0x7f63ec15b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:53.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.148+0000 7f63f1770640 1 -- 172.21.15.116:0/741453643 >> v1:172.21.15.116:6789/0 conn(0x7f63ec15b0c0 legacy=0x7f63ec071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.148+0000 7f63f1770640 1 -- 172.21.15.116:0/741453643 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ec0cc9f0 con 0x7f63ec072250 2023-11-28T17:09:53.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.149+0000 7f63f1770640 1 --2- 172.21.15.116:0/741453643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec072250 0x7f63ec15ed10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f63dc0097a0 tx=0x7f63dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd82f7614c763cdc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:53.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.150+0000 7f63f0f6f640 1 -- 172.21.15.116:0/741453643 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc031830 con 0x7f63ec072250 2023-11-28T17:09:53.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.150+0000 7f63f0f6f640 1 -- 172.21.15.116:0/741453643 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63dc031990 con 0x7f63ec072250 2023-11-28T17:09:53.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.150+0000 7f63f0f6f640 1 -- 172.21.15.116:0/741453643 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc031d20 con 0x7f63ec072250 2023-11-28T17:09:53.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.150+0000 7f63f41d5640 1 -- 172.21.15.116:0/741453643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec072250 msgr2=0x7f63ec15ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:53.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.150+0000 7f63f41d5640 1 --2- 172.21.15.116:0/741453643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec072250 0x7f63ec15ed10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f63dc0097a0 tx=0x7f63dc02b8d0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.150+0000 7f63f41d5640 1 -- 172.21.15.116:0/741453643 shutdown_connections 2023-11-28T17:09:53.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.150+0000 7f63f41d5640 1 --2- 172.21.15.116:0/741453643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec072250 0x7f63ec15ed10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.150+0000 7f63f41d5640 1 -- 172.21.15.116:0/741453643 >> 172.21.15.116:0/741453643 conn(0x7f63ec0b0cf0 msgr2=0x7f63ec0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:53.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.150+0000 7f63f41d5640 1 -- 172.21.15.116:0/741453643 shutdown_connections 2023-11-28T17:09:53.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.150+0000 7f63f41d5640 1 -- 172.21.15.116:0/741453643 wait complete. 2023-11-28T17:09:53.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.152+0000 7f63f41d5640 1 Processor -- start 2023-11-28T17:09:53.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.152+0000 7f63f41d5640 1 -- start start 2023-11-28T17:09:53.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.153+0000 7f63f41d5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec072250 0x7f63ec1f2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:53.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.153+0000 7f63f41d5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f63ec0ce940 0x7f63ec0afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:53.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.153+0000 7f63f41d5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec15b0c0 0x7f63ec0b01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:53.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.153+0000 7f63f41d5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec1569a0 con 0x7f63ec072250 2023-11-28T17:09:53.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.153+0000 7f63f41d5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec155af0 con 0x7f63ec15b0c0 2023-11-28T17:09:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.153+0000 7f63f41d5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec155d90 con 0x7f63ec0ce940 2023-11-28T17:09:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.153+0000 7f63f1f71640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec072250 0x7f63ec1f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.153+0000 7f63f1f71640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec072250 0x7f63ec1f2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56482/0 (socket says 172.21.15.116:56482) 2023-11-28T17:09:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.153+0000 7f63f1f71640 1 -- 172.21.15.116:0/4270973372 learned_addr learned my addr 172.21.15.116:0/4270973372 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.154+0000 7f63f2772640 1 --2- 172.21.15.116:0/4270973372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec15b0c0 0x7f63ec0b01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.154+0000 7f63f1770640 1 --2- 172.21.15.116:0/4270973372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f63ec0ce940 0x7f63ec0afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.154+0000 7f63f1f71640 1 -- 172.21.15.116:0/4270973372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f63ec0ce940 msgr2=0x7f63ec0afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.154+0000 7f63f1f71640 1 --2- 172.21.15.116:0/4270973372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f63ec0ce940 0x7f63ec0afc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.154+0000 7f63f1f71640 1 -- 172.21.15.116:0/4270973372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec15b0c0 msgr2=0x7f63ec0b01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:53.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.154+0000 7f63f1f71640 1 --2- 172.21.15.116:0/4270973372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec15b0c0 0x7f63ec0b01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.154+0000 7f63f1f71640 1 -- 172.21.15.116:0/4270973372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63dc009400 con 0x7f63ec072250 2023-11-28T17:09:53.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.154+0000 7f63f1f71640 1 --2- 172.21.15.116:0/4270973372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec072250 0x7f63ec1f2950 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f63e800c500 tx=0x7f63e800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:53.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.154+0000 7f63e27fc640 1 -- 172.21.15.116:0/4270973372 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8005ce0 con 0x7f63ec072250 2023-11-28T17:09:53.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.154+0000 7f63e27fc640 1 -- 172.21.15.116:0/4270973372 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e8005e40 con 0x7f63ec072250 2023-11-28T17:09:53.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.154+0000 7f63f41d5640 1 -- 172.21.15.116:0/4270973372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ec15b620 con 0x7f63ec072250 2023-11-28T17:09:53.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.154+0000 7f63e27fc640 1 -- 172.21.15.116:0/4270973372 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8007330 con 0x7f63ec072250 2023-11-28T17:09:53.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.155+0000 7f63f41d5640 1 -- 172.21.15.116:0/4270973372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63ec156350 con 0x7f63ec072250 2023-11-28T17:09:53.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.156+0000 7f63f41d5640 1 -- 172.21.15.116:0/4270973372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63ec073c80 con 0x7f63ec072250 2023-11-28T17:09:53.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.157+0000 7f63e27fc640 1 -- 172.21.15.116:0/4270973372 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f63e8007550 con 0x7f63ec072250 2023-11-28T17:09:53.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.157+0000 7f63e27fc640 1 --2- 172.21.15.116:0/4270973372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f63d006b2a0 0x7f63d006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:53.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.157+0000 7f63e27fc640 1 -- 172.21.15.116:0/4270973372 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f63e80892a0 con 0x7f63ec072250 2023-11-28T17:09:53.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.158+0000 7f63f1770640 1 --2- 172.21.15.116:0/4270973372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f63d006b2a0 0x7f63d006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:53.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.159+0000 7f63f1770640 1 --2- 172.21.15.116:0/4270973372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f63d006b2a0 0x7f63d006d760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f63dc036000 tx=0x7f63dc03c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:53.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.162+0000 7f63e27fc640 1 -- 172.21.15.116:0/4270973372 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f63ec073c80 con 0x7f63ec072250 2023-11-28T17:09:53.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.345+0000 7f63f41d5640 1 -- 172.21.15.116:0/4270973372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63ec073c80 con 0x7f63ec072250 2023-11-28T17:09:53.347 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.346+0000 7f63e27fc640 1 -- 172.21.15.116:0/4270973372 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f63ec073c80 con 0x7f63ec072250 2023-11-28T17:09:53.348 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:09:53.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.349+0000 7f63f41d5640 1 -- 172.21.15.116:0/4270973372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f63d006b2a0 msgr2=0x7f63d006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:53.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.349+0000 7f63f41d5640 1 --2- 172.21.15.116:0/4270973372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f63d006b2a0 0x7f63d006d760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f63dc036000 tx=0x7f63dc03c040 comp rx=0 tx=0).stop 2023-11-28T17:09:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.349+0000 7f63f41d5640 1 -- 172.21.15.116:0/4270973372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec072250 msgr2=0x7f63ec1f2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.349+0000 7f63f41d5640 1 --2- 172.21.15.116:0/4270973372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec072250 0x7f63ec1f2950 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f63e800c500 tx=0x7f63e800c9c0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.349+0000 7f63f41d5640 1 -- 172.21.15.116:0/4270973372 shutdown_connections 2023-11-28T17:09:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.349+0000 7f63f41d5640 1 --2- 172.21.15.116:0/4270973372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f63d006b2a0 0x7f63d006d760 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.349+0000 7f63f41d5640 1 --2- 172.21.15.116:0/4270973372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec15b0c0 0x7f63ec0b01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.349+0000 7f63f41d5640 1 --2- 172.21.15.116:0/4270973372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f63ec0ce940 0x7f63ec0afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.349+0000 7f63f41d5640 1 --2- 172.21.15.116:0/4270973372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec072250 0x7f63ec1f2950 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.349+0000 7f63f41d5640 1 -- 172.21.15.116:0/4270973372 >> 172.21.15.116:0/4270973372 conn(0x7f63ec0b0cf0 msgr2=0x7f63ec0ad040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.349+0000 7f63f41d5640 1 -- 172.21.15.116:0/4270973372 shutdown_connections 2023-11-28T17:09:53.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.349+0000 7f63f41d5640 1 -- 172.21.15.116:0/4270973372 wait complete. 2023-11-28T17:09:53.369 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:09:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.582+0000 7fbf40c39640 1 Processor -- start 2023-11-28T17:09:53.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.582+0000 7fbf40c39640 1 -- start start 2023-11-28T17:09:53.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.584+0000 7fbf40c39640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf3c152050 0x7fbf3c15bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.584+0000 7fbf40c39640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf3c0b2fe0 0x7fbf3c0ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.584+0000 7fbf40c39640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbf3c0abd30 0x7fbf3c0ade70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.584+0000 7fbf40c39640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3c0b2b50 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.584+0000 7fbf40c39640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3c0ae3b0 con 0x7fbf3c0abd30 2023-11-28T17:09:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.584+0000 7fbf40c39640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c15c520 con 0x7fbf3c152050 2023-11-28T17:09:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.584+0000 7fbf3affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf3c0b2fe0 0x7fbf3c0ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.584+0000 7fbf3affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf3c0b2fe0 0x7fbf3c0ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56494/0 (socket says 172.21.15.116:56494) 2023-11-28T17:09:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.584+0000 7fbf3affd640 1 -- 172.21.15.116:0/3800318790 learned_addr learned my addr 172.21.15.116:0/3800318790 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:53.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.585+0000 7fbf3bfff640 1 --2- 172.21.15.116:0/3800318790 >> v2:172.21.15.156:3300/0 conn(0x7fbf3c0abd30 0x7fbf3c0ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:53.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.585+0000 7fbf3b7fe640 1 --2- 172.21.15.116:0/3800318790 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf3c152050 0x7fbf3c15bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.585+0000 7fbf3affd640 1 -- 172.21.15.116:0/3800318790 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf3c152050 msgr2=0x7fbf3c15bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.585+0000 7fbf3affd640 1 --2- 172.21.15.116:0/3800318790 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf3c152050 0x7fbf3c15bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.585+0000 7fbf3affd640 1 -- 172.21.15.116:0/3800318790 >> v2:172.21.15.156:3300/0 conn(0x7fbf3c0abd30 msgr2=0x7fbf3c0ade70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.585+0000 7fbf3affd640 1 --2- 172.21.15.116:0/3800318790 >> v2:172.21.15.156:3300/0 conn(0x7fbf3c0abd30 0x7fbf3c0ade70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.585+0000 7fbf3affd640 1 -- 172.21.15.116:0/3800318790 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c0ac2d0 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.586+0000 7fbf3affd640 1 --2- 172.21.15.116:0/3800318790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf3c0b2fe0 0x7fbf3c0ad930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c0ad720 tx=0x7fbf3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8e79f2479a2fdf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.586+0000 7fbf3a7fc640 1 -- 172.21.15.116:0/3800318790 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30031870 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.586+0000 7fbf3a7fc640 1 -- 172.21.15.116:0/3800318790 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf300319d0 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.586+0000 7fbf3a7fc640 1 -- 172.21.15.116:0/3800318790 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30031d60 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.587+0000 7fbf40c39640 1 -- 172.21.15.116:0/3800318790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf3c0b2fe0 msgr2=0x7fbf3c0ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.587+0000 7fbf40c39640 1 --2- 172.21.15.116:0/3800318790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf3c0b2fe0 0x7fbf3c0ad930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c0ad720 tx=0x7fbf3002b910 comp rx=0 tx=0).stop 2023-11-28T17:09:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.587+0000 7fbf40c39640 1 -- 172.21.15.116:0/3800318790 shutdown_connections 2023-11-28T17:09:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.587+0000 7fbf40c39640 1 --2- 172.21.15.116:0/3800318790 >> v2:172.21.15.156:3300/0 conn(0x7fbf3c0abd30 0x7fbf3c0ade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.587+0000 7fbf40c39640 1 --2- 172.21.15.116:0/3800318790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf3c0b2fe0 0x7fbf3c0ad930 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.587+0000 7fbf40c39640 1 --2- 172.21.15.116:0/3800318790 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf3c152050 0x7fbf3c15bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.587+0000 7fbf40c39640 1 -- 172.21.15.116:0/3800318790 >> 172.21.15.116:0/3800318790 conn(0x7fbf3c0b0950 msgr2=0x7fbf3c0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.587+0000 7fbf40c39640 1 -- 172.21.15.116:0/3800318790 shutdown_connections 2023-11-28T17:09:53.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.587+0000 7fbf40c39640 1 -- 172.21.15.116:0/3800318790 wait complete. 2023-11-28T17:09:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.589+0000 7fbf40c39640 1 Processor -- start 2023-11-28T17:09:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.589+0000 7fbf40c39640 1 -- start start 2023-11-28T17:09:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.590+0000 7fbf40c39640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf3c0abd30 0x7fbf3c070c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:53.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.590+0000 7fbf40c39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf3c0b2fe0 0x7fbf3c1475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:53.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.590+0000 7fbf40c39640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf3c152050 0x7fbf3c159cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:53.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.590+0000 7fbf40c39640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c0711e0 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.590+0000 7fbf40c39640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c0714c0 con 0x7fbf3c152050 2023-11-28T17:09:53.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.590+0000 7fbf40c39640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c0717a0 con 0x7fbf3c0abd30 2023-11-28T17:09:53.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.590+0000 7fbf3affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf3c0b2fe0 0x7fbf3c1475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:53.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.590+0000 7fbf3affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf3c0b2fe0 0x7fbf3c1475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56504/0 (socket says 172.21.15.116:56504) 2023-11-28T17:09:53.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.590+0000 7fbf3affd640 1 -- 172.21.15.116:0/1014796759 learned_addr learned my addr 172.21.15.116:0/1014796759 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:53.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.591+0000 7fbf3affd640 1 -- 172.21.15.116:0/1014796759 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf3c0abd30 msgr2=0x7fbf3c070c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:53.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.591+0000 7fbf3affd640 1 --2- 172.21.15.116:0/1014796759 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf3c0abd30 0x7fbf3c070c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.591+0000 7fbf3affd640 1 -- 172.21.15.116:0/1014796759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf3c152050 msgr2=0x7fbf3c159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:53.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.591+0000 7fbf3affd640 1 --2- 172.21.15.116:0/1014796759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf3c152050 0x7fbf3c159cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.591+0000 7fbf3affd640 1 -- 172.21.15.116:0/1014796759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf300094d0 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.591+0000 7fbf3affd640 1 --2- 172.21.15.116:0/1014796759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf3c0b2fe0 0x7fbf3c1475d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbf3002b950 tx=0x7fbf300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:53.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.591+0000 7fbf23fff640 1 -- 172.21.15.116:0/1014796759 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3003ad30 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.591+0000 7fbf23fff640 1 -- 172.21.15.116:0/1014796759 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf300443f0 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.591+0000 7fbf23fff640 1 -- 172.21.15.116:0/1014796759 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf300447a0 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.591+0000 7fbf40c39640 1 -- 172.21.15.116:0/1014796759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3c071a80 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.591+0000 7fbf40c39640 1 -- 172.21.15.116:0/1014796759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3c1471b0 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.592+0000 7fbf40c39640 1 -- 172.21.15.116:0/1014796759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3c14bb50 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.593+0000 7fbf23fff640 1 -- 172.21.15.116:0/1014796759 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fbf3004d450 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.595+0000 7fbf23fff640 1 --2- 172.21.15.116:0/1014796759 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf2806b2a0 0x7fbf2806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.595+0000 7fbf23fff640 1 -- 172.21.15.116:0/1014796759 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbf3002d040 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.595+0000 7fbf3b7fe640 1 --2- 172.21.15.116:0/1014796759 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf2806b2a0 0x7fbf2806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:53.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.596+0000 7fbf3b7fe640 1 --2- 172.21.15.116:0/1014796759 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf2806b2a0 0x7fbf2806d760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c006fd0 tx=0x7fbf2c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:53.600 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.599+0000 7fbf23fff640 1 -- 172.21.15.116:0/1014796759 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf300dee90 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.787+0000 7fbf40c39640 1 -- 172.21.15.116:0/1014796759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbf3c1473a0 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.788+0000 7fbf23fff640 1 -- 172.21.15.116:0/1014796759 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbf30035020 con 0x7fbf3c0b2fe0 2023-11-28T17:09:53.789 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:09:53.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.790+0000 7fbf40c39640 1 -- 172.21.15.116:0/1014796759 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf2806b2a0 msgr2=0x7fbf2806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:53.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.790+0000 7fbf40c39640 1 --2- 172.21.15.116:0/1014796759 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf2806b2a0 0x7fbf2806d760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c006fd0 tx=0x7fbf2c008040 comp rx=0 tx=0).stop 2023-11-28T17:09:53.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.790+0000 7fbf40c39640 1 -- 172.21.15.116:0/1014796759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf3c0b2fe0 msgr2=0x7fbf3c1475d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:53.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.791+0000 7fbf40c39640 1 --2- 172.21.15.116:0/1014796759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf3c0b2fe0 0x7fbf3c1475d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbf3002b950 tx=0x7fbf300046c0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.791+0000 7fbf40c39640 1 -- 172.21.15.116:0/1014796759 shutdown_connections 2023-11-28T17:09:53.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.791+0000 7fbf40c39640 1 --2- 172.21.15.116:0/1014796759 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf2806b2a0 0x7fbf2806d760 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.791+0000 7fbf40c39640 1 --2- 172.21.15.116:0/1014796759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf3c152050 0x7fbf3c159cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.791+0000 7fbf40c39640 1 --2- 172.21.15.116:0/1014796759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf3c0b2fe0 0x7fbf3c1475d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.791+0000 7fbf40c39640 1 --2- 172.21.15.116:0/1014796759 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf3c0abd30 0x7fbf3c070c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:53.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.791+0000 7fbf40c39640 1 -- 172.21.15.116:0/1014796759 >> 172.21.15.116:0/1014796759 conn(0x7fbf3c0b0950 msgr2=0x7fbf3c15c690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:53.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.791+0000 7fbf40c39640 1 -- 172.21.15.116:0/1014796759 shutdown_connections 2023-11-28T17:09:53.793 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:53.791+0000 7fbf40c39640 1 -- 172.21.15.116:0/1014796759 wait complete. 2023-11-28T17:09:53.811 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:09:54.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.018+0000 7f65c22bb640 1 Processor -- start 2023-11-28T17:09:54.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.018+0000 7f65c22bb640 1 -- start start 2023-11-28T17:09:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.019+0000 7f65c22bb640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f65bc15b0c0 0x7f65bc15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.019+0000 7f65c22bb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f65bc0ce940 0x7f65bc0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.020+0000 7f65c22bb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65bc0cd320 con 0x7f65bc071b40 2023-11-28T17:09:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.020+0000 7f65c22bb640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65bc0ced80 con 0x7f65bc15b0c0 2023-11-28T17:09:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.020+0000 7f65c22bb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0cbee0 con 0x7f65bc0ce940 2023-11-28T17:09:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.020+0000 7f65c0858640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f65bc071b40 0x7f65bc15b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51492/0 (socket says 172.21.15.116:51492) 2023-11-28T17:09:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.020+0000 7f65c0858640 1 -- 172.21.15.116:0/3458286830 learned_addr learned my addr 172.21.15.116:0/3458286830 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:54.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.020+0000 7f65bbfff640 1 --2- 172.21.15.116:0/3458286830 >> v2:172.21.15.156:3300/0 conn(0x7f65bc15b0c0 0x7f65bc15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.020+0000 7f65b3fff640 1 --2- 172.21.15.116:0/3458286830 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f65bc0ce940 0x7f65bc0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.020+0000 7f65bb7fe640 1 -- 172.21.15.116:0/3458286830 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837548996 0 0) 0x7f65bc0cd320 con 0x7f65bc071b40 2023-11-28T17:09:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 -- 172.21.15.116:0/3458286830 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65a8003520 con 0x7f65bc071b40 2023-11-28T17:09:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 -- 172.21.15.116:0/3458286830 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 838907189 0 0) 0x7f65a8003520 con 0x7f65bc071b40 2023-11-28T17:09:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 -- 172.21.15.116:0/3458286830 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f65bc0ce940 msgr2=0x7f65bc0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 --2- 172.21.15.116:0/3458286830 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f65bc0ce940 0x7f65bc0cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 -- 172.21.15.116:0/3458286830 >> v2:172.21.15.156:3300/0 conn(0x7f65bc15b0c0 msgr2=0x7f65bc15ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 --2- 172.21.15.116:0/3458286830 >> v2:172.21.15.156:3300/0 conn(0x7f65bc15b0c0 0x7f65bc15ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 -- 172.21.15.116:0/3458286830 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc0cc710 con 0x7f65bc071b40 2023-11-28T17:09:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 -- 172.21.15.116:0/3458286830 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f65b40019f0 con 0x7f65bc071b40 2023-11-28T17:09:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bbfff640 1 --2- 172.21.15.116:0/3458286830 >> v2:172.21.15.156:3300/0 conn(0x7f65bc15b0c0 0x7f65bc15ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:09:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65b3fff640 1 --2- 172.21.15.116:0/3458286830 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f65bc0ce940 0x7f65bc0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:09:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 -- 172.21.15.116:0/3458286830 >> v1:172.21.15.116:6789/0 conn(0x7f65bc071b40 legacy=0x7f65bc15b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 --2- 172.21.15.116:0/3458286830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65a8003970 0x7f65a8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 --2- 172.21.15.116:0/3458286830 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f65bc0ce940 0x7f65a80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 --2- 172.21.15.116:0/3458286830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65a8004d00 0x7f65a8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 -- 172.21.15.116:0/3458286830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0ced80 con 0x7f65a8004d00 2023-11-28T17:09:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 -- 172.21.15.116:0/3458286830 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0cbee0 con 0x7f65a8003970 2023-11-28T17:09:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.021+0000 7f65bb7fe640 1 -- 172.21.15.116:0/3458286830 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a800b980 con 0x7f65bc0ce940 2023-11-28T17:09:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.022+0000 7f65c0858640 1 --2- 172.21.15.116:0/3458286830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65a8004d00 0x7f65a8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.022+0000 7f65bbfff640 1 --2- 172.21.15.116:0/3458286830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65a8003970 0x7f65a8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.022+0000 7f65b3fff640 1 --2- 172.21.15.116:0/3458286830 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f65bc0ce940 0x7f65a80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.022+0000 7f65c0858640 1 -- 172.21.15.116:0/3458286830 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f65bc0ce940 msgr2=0x7f65a80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.022+0000 7f65c0858640 1 --2- 172.21.15.116:0/3458286830 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f65bc0ce940 0x7f65a80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.022+0000 7f65c0858640 1 -- 172.21.15.116:0/3458286830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65a8003970 msgr2=0x7f65a8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.022+0000 7f65c0858640 1 --2- 172.21.15.116:0/3458286830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65a8003970 0x7f65a8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.022+0000 7f65c0858640 1 -- 172.21.15.116:0/3458286830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a4009400 con 0x7f65a8004d00 2023-11-28T17:09:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.023+0000 7f65c0858640 1 --2- 172.21.15.116:0/3458286830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65a8004d00 0x7f65a8005180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f65b400a7c0 tx=0x7f65a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e3c6f236a6563a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.023+0000 7f65bb7fe640 1 -- 172.21.15.116:0/3458286830 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4034630 con 0x7f65a8004d00 2023-11-28T17:09:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.023+0000 7f65bb7fe640 1 -- 172.21.15.116:0/3458286830 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b402f070 con 0x7f65a8004d00 2023-11-28T17:09:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.023+0000 7f65bb7fe640 1 -- 172.21.15.116:0/3458286830 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b40349f0 con 0x7f65a8004d00 2023-11-28T17:09:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.024+0000 7f65c22bb640 1 -- 172.21.15.116:0/3458286830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65a8004d00 msgr2=0x7f65a8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.024+0000 7f65c22bb640 1 --2- 172.21.15.116:0/3458286830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65a8004d00 0x7f65a8005180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f65b400a7c0 tx=0x7f65a80057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.024+0000 7f65c0858640 1 -- 172.21.15.116:0/3458286830 reap_dead start 2023-11-28T17:09:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.024+0000 7f65c22bb640 1 -- 172.21.15.116:0/3458286830 shutdown_connections 2023-11-28T17:09:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.024+0000 7f65c22bb640 1 -- 172.21.15.116:0/3458286830 >> 172.21.15.116:0/3458286830 conn(0x7f65bc0b0cf0 msgr2=0x7f65bc0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.024+0000 7f65c22bb640 1 -- 172.21.15.116:0/3458286830 shutdown_connections 2023-11-28T17:09:54.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.024+0000 7f65c22bb640 1 -- 172.21.15.116:0/3458286830 wait complete. 2023-11-28T17:09:54.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.026+0000 7f65c22bb640 1 Processor -- start 2023-11-28T17:09:54.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.026+0000 7f65c22bb640 1 -- start start 2023-11-28T17:09:54.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.026+0000 7f65c22bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65a8004d00 0x7f65bc1e7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.026+0000 7f65c22bb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f65bc0b0530 0x7f65bc0af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.026+0000 7f65c22bb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65bc1545e0 0x7f65bc154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.027+0000 7f65c22bb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc1e4710 con 0x7f65a8004d00 2023-11-28T17:09:54.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.027+0000 7f65c22bb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc157f00 con 0x7f65bc1545e0 2023-11-28T17:09:54.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.027+0000 7f65c22bb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc0b0aa0 con 0x7f65bc0b0530 2023-11-28T17:09:54.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.027+0000 7f65bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65a8004d00 0x7f65bc1e7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.027+0000 7f65bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65a8004d00 0x7f65bc1e7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56516/0 (socket says 172.21.15.116:56516) 2023-11-28T17:09:54.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.027+0000 7f65bbfff640 1 -- 172.21.15.116:0/291527018 learned_addr learned my addr 172.21.15.116:0/291527018 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:54.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.027+0000 7f65b3fff640 1 --2- 172.21.15.116:0/291527018 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f65bc0b0530 0x7f65bc0af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.027+0000 7f65bbfff640 1 -- 172.21.15.116:0/291527018 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f65bc0b0530 msgr2=0x7f65bc0af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:54.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.027+0000 7f65bbfff640 1 --2- 172.21.15.116:0/291527018 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f65bc0b0530 0x7f65bc0af3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.027+0000 7f65bbfff640 1 -- 172.21.15.116:0/291527018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65bc1545e0 msgr2=0x7f65bc154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:54.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.027+0000 7f65c0858640 1 --2- 172.21.15.116:0/291527018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65bc1545e0 0x7f65bc154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.027+0000 7f65bbfff640 1 --2- 172.21.15.116:0/291527018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65bc1545e0 0x7f65bc154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.027+0000 7f65bbfff640 1 -- 172.21.15.116:0/291527018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc154fb0 con 0x7f65a8004d00 2023-11-28T17:09:54.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.027+0000 7f65bbfff640 1 --2- 172.21.15.116:0/291527018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65a8004d00 0x7f65bc1e7a10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f65ac011930 tx=0x7f65ac011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:54.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.028+0000 7f65b97fa640 1 -- 172.21.15.116:0/291527018 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac00fbd0 con 0x7f65a8004d00 2023-11-28T17:09:54.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.028+0000 7f65b97fa640 1 -- 172.21.15.116:0/291527018 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ac00fd30 con 0x7f65a8004d00 2023-11-28T17:09:54.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.028+0000 7f65b97fa640 1 -- 172.21.15.116:0/291527018 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac015660 con 0x7f65a8004d00 2023-11-28T17:09:54.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.028+0000 7f65c22bb640 1 -- 172.21.15.116:0/291527018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65bc1e4190 con 0x7f65a8004d00 2023-11-28T17:09:54.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.028+0000 7f65c22bb640 1 -- 172.21.15.116:0/291527018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65bc158110 con 0x7f65a8004d00 2023-11-28T17:09:54.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.029+0000 7f65c22bb640 1 -- 172.21.15.116:0/291527018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65bc069360 con 0x7f65a8004d00 2023-11-28T17:09:54.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.030+0000 7f65b97fa640 1 -- 172.21.15.116:0/291527018 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f65ac0157c0 con 0x7f65a8004d00 2023-11-28T17:09:54.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.031+0000 7f65b97fa640 1 --2- 172.21.15.116:0/291527018 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f659c06b2d0 0x7f659c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.031+0000 7f65b97fa640 1 -- 172.21.15.116:0/291527018 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f65ac08a2c0 con 0x7f65a8004d00 2023-11-28T17:09:54.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.031+0000 7f65b3fff640 1 --2- 172.21.15.116:0/291527018 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f659c06b2d0 0x7f659c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.031+0000 7f65b3fff640 1 --2- 172.21.15.116:0/291527018 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f659c06b2d0 0x7f659c06d790 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009ae0 tx=0x7f65a4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:54.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.035+0000 7f65b97fa640 1 -- 172.21.15.116:0/291527018 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65ac055070 con 0x7f65a8004d00 2023-11-28T17:09:54.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.218+0000 7f65c22bb640 1 -- 172.21.15.116:0/291527018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f65bc0b1990 con 0x7f65a8004d00 2023-11-28T17:09:54.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.219+0000 7f65b97fa640 1 -- 172.21.15.116:0/291527018 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f65ac00fea0 con 0x7f65a8004d00 2023-11-28T17:09:54.220 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:09:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.221+0000 7f65c22bb640 1 -- 172.21.15.116:0/291527018 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f659c06b2d0 msgr2=0x7f659c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.221+0000 7f65c22bb640 1 --2- 172.21.15.116:0/291527018 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f659c06b2d0 0x7f659c06d790 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009ae0 tx=0x7f65a4008040 comp rx=0 tx=0).stop 2023-11-28T17:09:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.221+0000 7f65c22bb640 1 -- 172.21.15.116:0/291527018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65a8004d00 msgr2=0x7f65bc1e7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.221+0000 7f65c22bb640 1 --2- 172.21.15.116:0/291527018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65a8004d00 0x7f65bc1e7a10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f65ac011930 tx=0x7f65ac011df0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.221+0000 7f65c22bb640 1 -- 172.21.15.116:0/291527018 shutdown_connections 2023-11-28T17:09:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.221+0000 7f65c22bb640 1 --2- 172.21.15.116:0/291527018 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f659c06b2d0 0x7f659c06d790 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.221+0000 7f65c22bb640 1 --2- 172.21.15.116:0/291527018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65bc1545e0 0x7f65bc154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.221+0000 7f65c22bb640 1 --2- 172.21.15.116:0/291527018 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f65bc0b0530 0x7f65bc0af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.221+0000 7f65c22bb640 1 --2- 172.21.15.116:0/291527018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65a8004d00 0x7f65bc1e7a10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.221+0000 7f65c22bb640 1 -- 172.21.15.116:0/291527018 >> 172.21.15.116:0/291527018 conn(0x7f65bc0b0cf0 msgr2=0x7f65bc156b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:54.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.221+0000 7f65c22bb640 1 -- 172.21.15.116:0/291527018 shutdown_connections 2023-11-28T17:09:54.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.221+0000 7f65c22bb640 1 -- 172.21.15.116:0/291527018 wait complete. 2023-11-28T17:09:54.240 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:09:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.439+0000 7f8571778640 1 Processor -- start 2023-11-28T17:09:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.439+0000 7f8571778640 1 -- start start 2023-11-28T17:09:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.440+0000 7f8571778640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f856c160950 0x7f856c0b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.440+0000 7f8571778640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f856c0d11b0 0x7f856c0b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.440+0000 7f8571778640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f856c0b5110 0x7f856c06b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.440+0000 7f8571778640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856c160d30 con 0x7f856c160950 2023-11-28T17:09:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.440+0000 7f8571778640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856c0b5540 con 0x7f856c0d11b0 2023-11-28T17:09:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.440+0000 7f8571778640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0d1650 con 0x7f856c0b5110 2023-11-28T17:09:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.441+0000 7f856affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f856c160950 0x7f856c0b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.441+0000 7f856affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f856c160950 0x7f856c0b8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56522/0 (socket says 172.21.15.116:56522) 2023-11-28T17:09:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.441+0000 7f856affd640 1 -- 172.21.15.116:0/2141489701 learned_addr learned my addr 172.21.15.116:0/2141489701 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.441+0000 7f856affd640 1 -- 172.21.15.116:0/2141489701 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f856c0b5110 msgr2=0x7f856c06b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.441+0000 7f856affd640 1 --2- 172.21.15.116:0/2141489701 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f856c0b5110 0x7f856c06b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.441+0000 7f856a7fc640 1 --2- 172.21.15.116:0/2141489701 >> v2:172.21.15.156:3300/0 conn(0x7f856c0d11b0 0x7f856c0b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.441+0000 7f856affd640 1 -- 172.21.15.116:0/2141489701 >> v2:172.21.15.156:3300/0 conn(0x7f856c0d11b0 msgr2=0x7f856c0b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.441+0000 7f856affd640 1 --2- 172.21.15.116:0/2141489701 >> v2:172.21.15.156:3300/0 conn(0x7f856c0d11b0 0x7f856c0b4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.441+0000 7f856affd640 1 -- 172.21.15.116:0/2141489701 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c0d1930 con 0x7f856c160950 2023-11-28T17:09:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.442+0000 7f856affd640 1 --2- 172.21.15.116:0/2141489701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f856c160950 0x7f856c0b8a40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f856c0d0cb0 tx=0x7f855c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97fd1de875a5148 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.443+0000 7f8569ffb640 1 -- 172.21.15.116:0/2141489701 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c031870 con 0x7f856c160950 2023-11-28T17:09:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.443+0000 7f8569ffb640 1 -- 172.21.15.116:0/2141489701 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855c0319d0 con 0x7f856c160950 2023-11-28T17:09:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.443+0000 7f8569ffb640 1 -- 172.21.15.116:0/2141489701 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c031d60 con 0x7f856c160950 2023-11-28T17:09:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.443+0000 7f8571778640 1 -- 172.21.15.116:0/2141489701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f856c160950 msgr2=0x7f856c0b8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.443+0000 7f8571778640 1 --2- 172.21.15.116:0/2141489701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f856c160950 0x7f856c0b8a40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f856c0d0cb0 tx=0x7f855c02b910 comp rx=0 tx=0).stop 2023-11-28T17:09:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.443+0000 7f8571778640 1 -- 172.21.15.116:0/2141489701 shutdown_connections 2023-11-28T17:09:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.443+0000 7f8571778640 1 --2- 172.21.15.116:0/2141489701 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f856c0b5110 0x7f856c06b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.443+0000 7f8571778640 1 --2- 172.21.15.116:0/2141489701 >> v2:172.21.15.156:3300/0 conn(0x7f856c0d11b0 0x7f856c0b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.443+0000 7f8571778640 1 --2- 172.21.15.116:0/2141489701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f856c160950 0x7f856c0b8a40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.443+0000 7f8571778640 1 -- 172.21.15.116:0/2141489701 >> 172.21.15.116:0/2141489701 conn(0x7f856c072220 msgr2=0x7f856c0cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.443+0000 7f8571778640 1 -- 172.21.15.116:0/2141489701 shutdown_connections 2023-11-28T17:09:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.443+0000 7f8571778640 1 -- 172.21.15.116:0/2141489701 wait complete. 2023-11-28T17:09:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.445+0000 7f8571778640 1 Processor -- start 2023-11-28T17:09:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.445+0000 7f8571778640 1 -- start start 2023-11-28T17:09:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.446+0000 7f8571778640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f856c0b5110 0x7f856c15e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.446+0000 7f8571778640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c0d11b0 0x7f856c0c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.446+0000 7f8571778640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f856c160950 0x7f856c14e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.446+0000 7f8571778640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0c76b0 con 0x7f856c160950 2023-11-28T17:09:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.446+0000 7f8571778640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0c7990 con 0x7f856c0d11b0 2023-11-28T17:09:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.446+0000 7f8571778640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0c7c70 con 0x7f856c0b5110 2023-11-28T17:09:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.446+0000 7f856b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f856c160950 0x7f856c14e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.447+0000 7f856b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f856c160950 0x7f856c14e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56538/0 (socket says 172.21.15.116:56538) 2023-11-28T17:09:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.447+0000 7f856b7fe640 1 -- 172.21.15.116:0/407574855 learned_addr learned my addr 172.21.15.116:0/407574855 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.447+0000 7f856a7fc640 1 --2- 172.21.15.116:0/407574855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c0d11b0 0x7f856c0c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.447+0000 7f856affd640 1 --2- 172.21.15.116:0/407574855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f856c0b5110 0x7f856c15e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.447+0000 7f856b7fe640 1 -- 172.21.15.116:0/407574855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f856c0b5110 msgr2=0x7f856c15e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.447+0000 7f856b7fe640 1 --2- 172.21.15.116:0/407574855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f856c0b5110 0x7f856c15e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.447+0000 7f856b7fe640 1 -- 172.21.15.116:0/407574855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c0d11b0 msgr2=0x7f856c0c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:54.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.447+0000 7f856b7fe640 1 --2- 172.21.15.116:0/407574855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c0d11b0 0x7f856c0c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.447+0000 7f856b7fe640 1 -- 172.21.15.116:0/407574855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855c0094d0 con 0x7f856c160950 2023-11-28T17:09:54.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.447+0000 7f856b7fe640 1 --2- 172.21.15.116:0/407574855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f856c160950 0x7f856c14e0e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8564009fd0 tx=0x7f856400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:54.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.447+0000 7f85637fe640 1 -- 172.21.15.116:0/407574855 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8564011830 con 0x7f856c160950 2023-11-28T17:09:54.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.448+0000 7f85637fe640 1 -- 172.21.15.116:0/407574855 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8564011990 con 0x7f856c160950 2023-11-28T17:09:54.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.448+0000 7f85637fe640 1 -- 172.21.15.116:0/407574855 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8564011d20 con 0x7f856c160950 2023-11-28T17:09:54.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.448+0000 7f8571778640 1 -- 172.21.15.116:0/407574855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856c0c7f50 con 0x7f856c160950 2023-11-28T17:09:54.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.448+0000 7f8571778640 1 -- 172.21.15.116:0/407574855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856c158fe0 con 0x7f856c160950 2023-11-28T17:09:54.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.449+0000 7f8571778640 1 -- 172.21.15.116:0/407574855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856c1592c0 con 0x7f856c160950 2023-11-28T17:09:54.451 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.450+0000 7f85637fe640 1 -- 172.21.15.116:0/407574855 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f856401b460 con 0x7f856c160950 2023-11-28T17:09:54.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.451+0000 7f85637fe640 1 --2- 172.21.15.116:0/407574855 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f854c06b2a0 0x7f854c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.452+0000 7f85637fe640 1 -- 172.21.15.116:0/407574855 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8564089af0 con 0x7f856c160950 2023-11-28T17:09:54.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.452+0000 7f856affd640 1 --2- 172.21.15.116:0/407574855 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f854c06b2a0 0x7f854c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.452+0000 7f856affd640 1 --2- 172.21.15.116:0/407574855 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f854c06b2a0 0x7f854c06d760 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f855c036000 tx=0x7f855c03c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:54.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.456+0000 7f85637fe640 1 -- 172.21.15.116:0/407574855 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8564058340 con 0x7f856c160950 2023-11-28T17:09:54.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.660+0000 7f8571778640 1 -- 172.21.15.116:0/407574855 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f856c0693d0 con 0x7f856c160950 2023-11-28T17:09:54.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.661+0000 7f85637fe640 1 -- 172.21.15.116:0/407574855 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8564057ed0 con 0x7f856c160950 2023-11-28T17:09:54.662 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-28T17:09:54.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.663+0000 7f8571778640 1 -- 172.21.15.116:0/407574855 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f854c06b2a0 msgr2=0x7f854c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:54.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.663+0000 7f8571778640 1 --2- 172.21.15.116:0/407574855 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f854c06b2a0 0x7f854c06d760 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f855c036000 tx=0x7f855c03c040 comp rx=0 tx=0).stop 2023-11-28T17:09:54.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.663+0000 7f8571778640 1 -- 172.21.15.116:0/407574855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f856c160950 msgr2=0x7f856c14e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:54.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.663+0000 7f8571778640 1 --2- 172.21.15.116:0/407574855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f856c160950 0x7f856c14e0e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8564009fd0 tx=0x7f856400b8e0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.664+0000 7f8571778640 1 -- 172.21.15.116:0/407574855 shutdown_connections 2023-11-28T17:09:54.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.664+0000 7f8571778640 1 --2- 172.21.15.116:0/407574855 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f854c06b2a0 0x7f854c06d760 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.664+0000 7f8571778640 1 --2- 172.21.15.116:0/407574855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f856c160950 0x7f856c14e0e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.664+0000 7f8571778640 1 --2- 172.21.15.116:0/407574855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c0d11b0 0x7f856c0c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.664+0000 7f8571778640 1 --2- 172.21.15.116:0/407574855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f856c0b5110 0x7f856c15e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.664+0000 7f8571778640 1 -- 172.21.15.116:0/407574855 >> 172.21.15.116:0/407574855 conn(0x7f856c072220 msgr2=0x7f856c0d2960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:54.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.664+0000 7f8571778640 1 -- 172.21.15.116:0/407574855 shutdown_connections 2023-11-28T17:09:54.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.664+0000 7f8571778640 1 -- 172.21.15.116:0/407574855 wait complete. 2023-11-28T17:09:54.684 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:09:54.684 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:09:54.734 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:54.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.990+0000 7fd5c7389640 1 Processor -- start 2023-11-28T17:09:54.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.990+0000 7fd5c7389640 1 -- start start 2023-11-28T17:09:54.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.992+0000 7fd5c7389640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd5c013cee0 0x7fd5c006c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.992+0000 7fd5c7389640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd5c0140200 0x7fd5c006c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.992+0000 7fd5c7389640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c00aea20 con 0x7fd5c015a490 2023-11-28T17:09:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.992+0000 7fd5c7389640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5c013d2c0 con 0x7fd5c013cee0 2023-11-28T17:09:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.992+0000 7fd5c7389640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5c015b3a0 con 0x7fd5c0140200 2023-11-28T17:09:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.992+0000 7fd5c5125640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd5c013cee0 0x7fd5c006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.992+0000 7fd5c5926640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd5c015a490 0x7fd5c015a900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51504/0 (socket says 172.21.15.116:51504) 2023-11-28T17:09:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.992+0000 7fd5c5125640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd5c013cee0 0x7fd5c006c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56554/0 (socket says 172.21.15.116:56554) 2023-11-28T17:09:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.992+0000 7fd5c5926640 1 -- 172.21.15.116:0/205710425 learned_addr learned my addr 172.21.15.116:0/205710425 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:54.994 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.993+0000 7fd5c4924640 1 --2- 172.21.15.116:0/205710425 >> v2:172.21.15.156:3300/0 conn(0x7fd5c0140200 0x7fd5c006c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.993+0000 7fd5c5125640 1 -- 172.21.15.116:0/205710425 >> v2:172.21.15.156:3300/0 conn(0x7fd5c0140200 msgr2=0x7fd5c006c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:54.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.993+0000 7fd5c5125640 1 --2- 172.21.15.116:0/205710425 >> v2:172.21.15.156:3300/0 conn(0x7fd5c0140200 0x7fd5c006c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.993+0000 7fd5c5125640 1 -- 172.21.15.116:0/205710425 >> v1:172.21.15.116:6789/0 conn(0x7fd5c015a490 legacy=0x7fd5c015a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:54.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.993+0000 7fd5c5125640 1 -- 172.21.15.116:0/205710425 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c015b0c0 con 0x7fd5c013cee0 2023-11-28T17:09:54.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.994+0000 7fd5c5125640 1 --2- 172.21.15.116:0/205710425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c013cee0 0x7fd5c006c250 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0067f50 tx=0x7fd5b402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b4a0d8de879dcc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:54.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.994+0000 7fd5bffff640 1 -- 172.21.15.116:0/205710425 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b402cbf0 con 0x7fd5c013cee0 2023-11-28T17:09:54.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.994+0000 7fd5bffff640 1 -- 172.21.15.116:0/205710425 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b402cd50 con 0x7fd5c013cee0 2023-11-28T17:09:54.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.994+0000 7fd5bffff640 1 -- 172.21.15.116:0/205710425 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4034660 con 0x7fd5c013cee0 2023-11-28T17:09:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.995+0000 7fd5c7389640 1 -- 172.21.15.116:0/205710425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c013cee0 msgr2=0x7fd5c006c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.995+0000 7fd5c7389640 1 --2- 172.21.15.116:0/205710425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c013cee0 0x7fd5c006c250 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0067f50 tx=0x7fd5b402be40 comp rx=0 tx=0).stop 2023-11-28T17:09:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.995+0000 7fd5c7389640 1 -- 172.21.15.116:0/205710425 shutdown_connections 2023-11-28T17:09:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.995+0000 7fd5c7389640 1 --2- 172.21.15.116:0/205710425 >> v2:172.21.15.156:3300/0 conn(0x7fd5c0140200 0x7fd5c006c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.995+0000 7fd5c7389640 1 --2- 172.21.15.116:0/205710425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c013cee0 0x7fd5c006c250 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.995+0000 7fd5c7389640 1 -- 172.21.15.116:0/205710425 >> 172.21.15.116:0/205710425 conn(0x7fd5c006a4c0 msgr2=0x7fd5c006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.995+0000 7fd5c7389640 1 -- 172.21.15.116:0/205710425 shutdown_connections 2023-11-28T17:09:54.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.995+0000 7fd5c7389640 1 -- 172.21.15.116:0/205710425 wait complete. 2023-11-28T17:09:54.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.997+0000 7fd5c7389640 1 Processor -- start 2023-11-28T17:09:54.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.997+0000 7fd5c7389640 1 -- start start 2023-11-28T17:09:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.998+0000 7fd5c7389640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5c013cee0 0x7fd5c01f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.998+0000 7fd5c7389640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5c0140200 0x7fd5c0071e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.998+0000 7fd5c7389640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c015a490 0x7fd5c0135050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.998+0000 7fd5c7389640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c01f2fb0 con 0x7fd5c015a490 2023-11-28T17:09:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.998+0000 7fd5c7389640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c00723b0 con 0x7fd5c013cee0 2023-11-28T17:09:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.998+0000 7fd5c7389640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c013d460 con 0x7fd5c0140200 2023-11-28T17:09:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.998+0000 7fd5c5926640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c015a490 0x7fd5c0135050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.998+0000 7fd5c5926640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c015a490 0x7fd5c0135050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56566/0 (socket says 172.21.15.116:56566) 2023-11-28T17:09:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.998+0000 7fd5c5926640 1 -- 172.21.15.116:0/296638431 learned_addr learned my addr 172.21.15.116:0/296638431 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.998+0000 7fd5c5926640 1 -- 172.21.15.116:0/296638431 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5c0140200 msgr2=0x7fd5c0071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.998+0000 7fd5c5926640 1 --2- 172.21.15.116:0/296638431 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5c0140200 0x7fd5c0071e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.998+0000 7fd5c5926640 1 -- 172.21.15.116:0/296638431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5c013cee0 msgr2=0x7fd5c01f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.998+0000 7fd5c5926640 1 --2- 172.21.15.116:0/296638431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5c013cee0 0x7fd5c01f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.998+0000 7fd5c5926640 1 -- 172.21.15.116:0/296638431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b4009a70 con 0x7fd5c015a490 2023-11-28T17:09:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.999+0000 7fd5c5926640 1 --2- 172.21.15.116:0/296638431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c015a490 0x7fd5c0135050 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd5b800c370 tx=0x7fd5b800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.999+0000 7fd5bdffb640 1 -- 172.21.15.116:0/296638431 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8005ce0 con 0x7fd5c015a490 2023-11-28T17:09:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.999+0000 7fd5bdffb640 1 -- 172.21.15.116:0/296638431 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b8005e40 con 0x7fd5c015a490 2023-11-28T17:09:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.999+0000 7fd5bdffb640 1 -- 172.21.15.116:0/296638431 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b80072d0 con 0x7fd5c015a490 2023-11-28T17:09:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.999+0000 7fd5c7389640 1 -- 172.21.15.116:0/296638431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c0070570 con 0x7fd5c015a490 2023-11-28T17:09:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:54.999+0000 7fd5c7389640 1 -- 172.21.15.116:0/296638431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c013d720 con 0x7fd5c015a490 2023-11-28T17:09:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.000+0000 7fd5c7389640 1 -- 172.21.15.116:0/296638431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c00b0af0 con 0x7fd5c015a490 2023-11-28T17:09:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.001+0000 7fd5bdffb640 1 -- 172.21.15.116:0/296638431 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fd5b80074f0 con 0x7fd5c015a490 2023-11-28T17:09:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.002+0000 7fd5bdffb640 1 --2- 172.21.15.116:0/296638431 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd5a406b2a0 0x7fd5a406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.002+0000 7fd5bdffb640 1 -- 172.21.15.116:0/296638431 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd5b8089130 con 0x7fd5c015a490 2023-11-28T17:09:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.002+0000 7fd5c5125640 1 --2- 172.21.15.116:0/296638431 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd5a406b2a0 0x7fd5a406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.003+0000 7fd5c5125640 1 --2- 172.21.15.116:0/296638431 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd5a406b2a0 0x7fd5a406d760 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4036000 tx=0x7fd5b402e000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.006+0000 7fd5bdffb640 1 -- 172.21.15.116:0/296638431 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5b8053f00 con 0x7fd5c015a490 2023-11-28T17:09:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.186+0000 7fd5c7389640 1 -- 172.21.15.116:0/296638431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5c00693d0 con 0x7fd5c015a490 2023-11-28T17:09:55.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.189+0000 7fd5bdffb640 1 -- 172.21.15.116:0/296638431 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16570 (secure 0 0 0) 0x7fd5b8057520 con 0x7fd5c015a490 2023-11-28T17:09:55.190 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:55.190 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:43.049465+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:55.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.191+0000 7fd5c7389640 1 -- 172.21.15.116:0/296638431 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd5a406b2a0 msgr2=0x7fd5a406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:55.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.191+0000 7fd5c7389640 1 --2- 172.21.15.116:0/296638431 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd5a406b2a0 0x7fd5a406d760 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4036000 tx=0x7fd5b402e000 comp rx=0 tx=0).stop 2023-11-28T17:09:55.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.191+0000 7fd5c7389640 1 -- 172.21.15.116:0/296638431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c015a490 msgr2=0x7fd5c0135050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:55.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.191+0000 7fd5c7389640 1 --2- 172.21.15.116:0/296638431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c015a490 0x7fd5c0135050 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd5b800c370 tx=0x7fd5b800c830 comp rx=0 tx=0).stop 2023-11-28T17:09:55.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.191+0000 7fd5c7389640 1 -- 172.21.15.116:0/296638431 shutdown_connections 2023-11-28T17:09:55.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.191+0000 7fd5c7389640 1 --2- 172.21.15.116:0/296638431 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd5a406b2a0 0x7fd5a406d760 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.191+0000 7fd5c7389640 1 --2- 172.21.15.116:0/296638431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c015a490 0x7fd5c0135050 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.191+0000 7fd5c7389640 1 --2- 172.21.15.116:0/296638431 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5c0140200 0x7fd5c0071e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.191+0000 7fd5c7389640 1 --2- 172.21.15.116:0/296638431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5c013cee0 0x7fd5c01f2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.191+0000 7fd5c7389640 1 -- 172.21.15.116:0/296638431 >> 172.21.15.116:0/296638431 conn(0x7fd5c006a4c0 msgr2=0x7fd5c00ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:55.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.192+0000 7fd5c7389640 1 -- 172.21.15.116:0/296638431 shutdown_connections 2023-11-28T17:09:55.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.192+0000 7fd5c7389640 1 -- 172.21.15.116:0/296638431 wait complete. 2023-11-28T17:09:55.210 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:09:55.210 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:09:55.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.409+0000 7fee9f690640 1 Processor -- start 2023-11-28T17:09:55.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.409+0000 7fee9f690640 1 -- start start 2023-11-28T17:09:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.411+0000 7fee9f690640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fee980b5920 0x7fee98167ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.411+0000 7fee9f690640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee9815d800 con 0x7fee9815d420 2023-11-28T17:09:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.411+0000 7fee9f690640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee980d2fa0 con 0x7fee980d2710 2023-11-28T17:09:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.411+0000 7fee9f690640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee980d2280 con 0x7fee980b5920 2023-11-28T17:09:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.411+0000 7fee9d42c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fee9815d420 0x7fee980b8b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51516/0 (socket says 172.21.15.116:51516) 2023-11-28T17:09:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.411+0000 7fee9d42c640 1 -- 172.21.15.116:0/770941224 learned_addr learned my addr 172.21.15.116:0/770941224 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.411+0000 7fee9dc2d640 1 --2- 172.21.15.116:0/770941224 >> v2:172.21.15.116:3300/0 conn(0x7fee980b5920 0x7fee98167ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.412+0000 7fee8ffff640 1 -- 172.21.15.116:0/770941224 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349134301 0 0) 0x7fee9815d800 con 0x7fee9815d420 2023-11-28T17:09:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.412+0000 7fee8ffff640 1 -- 172.21.15.116:0/770941224 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee84003520 con 0x7fee9815d420 2023-11-28T17:09:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.412+0000 7fee9dc2d640 1 -- 172.21.15.116:0/770941224 >> v1:172.21.15.156:6789/0 conn(0x7fee980d2710 legacy=0x7fee980b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.412+0000 7fee9dc2d640 1 -- 172.21.15.116:0/770941224 >> v1:172.21.15.116:6789/0 conn(0x7fee9815d420 legacy=0x7fee980b8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.412+0000 7fee9dc2d640 1 -- 172.21.15.116:0/770941224 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee980d1670 con 0x7fee980b5920 2023-11-28T17:09:55.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.413+0000 7fee9dc2d640 1 --2- 172.21.15.116:0/770941224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee980b5920 0x7fee98167ed0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fee90009d90 tx=0x7fee9002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604b8f87004b3d0b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:55.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.413+0000 7fee8ffff640 1 -- 172.21.15.116:0/770941224 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9002cc60 con 0x7fee980b5920 2023-11-28T17:09:55.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.413+0000 7fee8ffff640 1 -- 172.21.15.116:0/770941224 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9002cdc0 con 0x7fee980b5920 2023-11-28T17:09:55.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.413+0000 7fee8ffff640 1 -- 172.21.15.116:0/770941224 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee90034660 con 0x7fee980b5920 2023-11-28T17:09:55.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.414+0000 7fee9f690640 1 -- 172.21.15.116:0/770941224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee980b5920 msgr2=0x7fee98167ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:55.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.414+0000 7fee9f690640 1 --2- 172.21.15.116:0/770941224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee980b5920 0x7fee98167ed0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fee90009d90 tx=0x7fee9002beb0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.414+0000 7fee9f690640 1 -- 172.21.15.116:0/770941224 shutdown_connections 2023-11-28T17:09:55.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.414+0000 7fee9f690640 1 --2- 172.21.15.116:0/770941224 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee980b5920 0x7fee98167ed0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.414+0000 7fee9f690640 1 -- 172.21.15.116:0/770941224 >> 172.21.15.116:0/770941224 conn(0x7fee98072010 msgr2=0x7fee98072420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:55.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.414+0000 7fee9f690640 1 -- 172.21.15.116:0/770941224 shutdown_connections 2023-11-28T17:09:55.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.414+0000 7fee9f690640 1 -- 172.21.15.116:0/770941224 wait complete. 2023-11-28T17:09:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.416+0000 7fee9f690640 1 Processor -- start 2023-11-28T17:09:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.416+0000 7fee9f690640 1 -- start start 2023-11-28T17:09:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.417+0000 7fee9f690640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee980b5920 0x7fee9815fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.417+0000 7fee9f690640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fee980d2710 0x7fee981579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.417+0000 7fee9f690640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee9815d420 0x7fee981f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.417+0000 7fee9f690640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9814ef50 con 0x7fee980b5920 2023-11-28T17:09:55.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.417+0000 7fee9f690640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee980cd040 con 0x7fee9815d420 2023-11-28T17:09:55.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.417+0000 7fee9f690640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee981571d0 con 0x7fee980d2710 2023-11-28T17:09:55.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.417+0000 7fee9d42c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee980b5920 0x7fee9815fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:55.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.417+0000 7fee9d42c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee980b5920 0x7fee9815fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56586/0 (socket says 172.21.15.116:56586) 2023-11-28T17:09:55.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.417+0000 7fee9d42c640 1 -- 172.21.15.116:0/2525283314 learned_addr learned my addr 172.21.15.116:0/2525283314 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:55.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.417+0000 7fee9d42c640 1 -- 172.21.15.116:0/2525283314 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fee980d2710 msgr2=0x7fee981579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:55.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.417+0000 7fee9d42c640 1 --2- 172.21.15.116:0/2525283314 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fee980d2710 0x7fee981579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.417+0000 7fee9d42c640 1 -- 172.21.15.116:0/2525283314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee9815d420 msgr2=0x7fee981f2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:55.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.417+0000 7fee9d42c640 1 --2- 172.21.15.116:0/2525283314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee9815d420 0x7fee981f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.417+0000 7fee9d42c640 1 -- 172.21.15.116:0/2525283314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee90009a70 con 0x7fee980b5920 2023-11-28T17:09:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.418+0000 7fee9d42c640 1 --2- 172.21.15.116:0/2525283314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee980b5920 0x7fee9815fde0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fee9400c410 tx=0x7fee9400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.418+0000 7fee8dffb640 1 -- 172.21.15.116:0/2525283314 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee94005ce0 con 0x7fee980b5920 2023-11-28T17:09:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.418+0000 7fee8dffb640 1 -- 172.21.15.116:0/2525283314 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee94005e40 con 0x7fee980b5920 2023-11-28T17:09:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.418+0000 7fee8dffb640 1 -- 172.21.15.116:0/2525283314 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee940072f0 con 0x7fee980b5920 2023-11-28T17:09:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.418+0000 7fee9f690640 1 -- 172.21.15.116:0/2525283314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee98156930 con 0x7fee980b5920 2023-11-28T17:09:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.418+0000 7fee9f690640 1 -- 172.21.15.116:0/2525283314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee98156ef0 con 0x7fee980b5920 2023-11-28T17:09:55.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.419+0000 7fee9f690640 1 -- 172.21.15.116:0/2525283314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee98069360 con 0x7fee980b5920 2023-11-28T17:09:55.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.420+0000 7fee8dffb640 1 -- 172.21.15.116:0/2525283314 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fee94007510 con 0x7fee980b5920 2023-11-28T17:09:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.422+0000 7fee8dffb640 1 --2- 172.21.15.116:0/2525283314 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fee7c06b2a0 0x7fee7c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.422+0000 7fee8dffb640 1 -- 172.21.15.116:0/2525283314 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fee94089230 con 0x7fee980b5920 2023-11-28T17:09:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.422+0000 7fee9cc2b640 1 --2- 172.21.15.116:0/2525283314 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fee7c06b2a0 0x7fee7c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:55.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.423+0000 7fee9cc2b640 1 --2- 172.21.15.116:0/2525283314 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fee7c06b2a0 0x7fee7c06d760 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fee980d3f10 tx=0x7fee880079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:55.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.426+0000 7fee8dffb640 1 -- 172.21.15.116:0/2525283314 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fee940541d0 con 0x7fee980b5920 2023-11-28T17:09:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.617+0000 7fee9f690640 1 -- 172.21.15.116:0/2525283314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fee980d3c60 con 0x7fee980b5920 2023-11-28T17:09:55.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.617+0000 7fee8dffb640 1 -- 172.21.15.116:0/2525283314 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fee940577f0 con 0x7fee980b5920 2023-11-28T17:09:55.619 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:09:55.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.620+0000 7fee9f690640 1 -- 172.21.15.116:0/2525283314 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fee7c06b2a0 msgr2=0x7fee7c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:55.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.620+0000 7fee9f690640 1 --2- 172.21.15.116:0/2525283314 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fee7c06b2a0 0x7fee7c06d760 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fee980d3f10 tx=0x7fee880079b0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.620+0000 7fee9f690640 1 -- 172.21.15.116:0/2525283314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee980b5920 msgr2=0x7fee9815fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:55.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.620+0000 7fee9f690640 1 --2- 172.21.15.116:0/2525283314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee980b5920 0x7fee9815fde0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fee9400c410 tx=0x7fee9400c8d0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.620+0000 7fee9f690640 1 -- 172.21.15.116:0/2525283314 shutdown_connections 2023-11-28T17:09:55.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.620+0000 7fee9f690640 1 --2- 172.21.15.116:0/2525283314 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fee7c06b2a0 0x7fee7c06d760 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.620+0000 7fee9f690640 1 --2- 172.21.15.116:0/2525283314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee9815d420 0x7fee981f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.620+0000 7fee9f690640 1 --2- 172.21.15.116:0/2525283314 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fee980d2710 0x7fee981579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.620+0000 7fee9f690640 1 --2- 172.21.15.116:0/2525283314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee980b5920 0x7fee9815fde0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.620+0000 7fee9f690640 1 -- 172.21.15.116:0/2525283314 >> 172.21.15.116:0/2525283314 conn(0x7fee98072010 msgr2=0x7fee980c8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:55.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.621+0000 7fee9f690640 1 -- 172.21.15.116:0/2525283314 shutdown_connections 2023-11-28T17:09:55.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.621+0000 7fee9f690640 1 -- 172.21.15.116:0/2525283314 wait complete. 2023-11-28T17:09:55.641 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:09:55.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.851+0000 7f4b0c0da640 1 Processor -- start 2023-11-28T17:09:55.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.852+0000 7f4b0c0da640 1 -- start start 2023-11-28T17:09:55.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.853+0000 7f4b0c0da640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4b0415b0c0 0x7f4b0415ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.853+0000 7f4b0c0da640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4b040ce940 0x7f4b040cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.853+0000 7f4b0c0da640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b040cd320 con 0x7f4b04071b40 2023-11-28T17:09:55.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.853+0000 7f4b0c0da640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b040ced80 con 0x7f4b040ce940 2023-11-28T17:09:55.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.853+0000 7f4b0c0da640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b040cbee0 con 0x7f4b0415b0c0 2023-11-28T17:09:55.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.854+0000 7f4b09e76640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4b0415b0c0 0x7f4b0415ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:55.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.854+0000 7f4b09675640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4b040ce940 0x7f4b040cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:55.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.854+0000 7f4b09675640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4b040ce940 0x7f4b040cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:57788/0 (socket says 172.21.15.116:57788) 2023-11-28T17:09:55.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.854+0000 7f4b09675640 1 -- 172.21.15.116:0/2755871872 learned_addr learned my addr 172.21.15.116:0/2755871872 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:55.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.854+0000 7f4b09e76640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4b0415b0c0 0x7f4b0415ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:47196/0 (socket says 172.21.15.116:47196) 2023-11-28T17:09:55.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.854+0000 7f4b0a677640 1 --1- 172.21.15.116:0/2755871872 >> v1:172.21.15.156:6789/0 conn(0x7f4b04071b40 0x7f4b0415b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.116:60410/0 (socket says 172.21.15.116:60410) 2023-11-28T17:09:55.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.855+0000 7f4b09e76640 1 -- 172.21.15.116:0/2755871872 >> v2:172.21.15.156:3300/0 conn(0x7f4b040ce940 msgr2=0x7f4b040cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:55.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.855+0000 7f4b09e76640 1 --2- 172.21.15.116:0/2755871872 >> v2:172.21.15.156:3300/0 conn(0x7f4b040ce940 0x7f4b040cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.855+0000 7f4b09e76640 1 -- 172.21.15.116:0/2755871872 >> v1:172.21.15.156:6789/0 conn(0x7f4b04071b40 legacy=0x7f4b0415b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:55.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.855+0000 7f4b09e76640 1 -- 172.21.15.116:0/2755871872 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b040cc710 con 0x7f4b0415b0c0 2023-11-28T17:09:55.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.856+0000 7f4b09e76640 1 --2- 172.21.15.116:0/2755871872 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4b0415b0c0 0x7f4b0415ed10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4b000097a0 tx=0x7f4b0002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a3a7c5a028a5c92 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:55.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.857+0000 7f4b08e74640 1 -- 172.21.15.116:0/2755871872 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b00031780 con 0x7f4b0415b0c0 2023-11-28T17:09:55.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.857+0000 7f4b08e74640 1 -- 172.21.15.116:0/2755871872 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b000318e0 con 0x7f4b0415b0c0 2023-11-28T17:09:55.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.857+0000 7f4b08e74640 1 -- 172.21.15.116:0/2755871872 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b00031c70 con 0x7f4b0415b0c0 2023-11-28T17:09:55.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.857+0000 7f4b0c0da640 1 -- 172.21.15.116:0/2755871872 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4b0415b0c0 msgr2=0x7f4b0415ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:55.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.857+0000 7f4b0c0da640 1 --2- 172.21.15.116:0/2755871872 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4b0415b0c0 0x7f4b0415ed10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4b000097a0 tx=0x7f4b0002b820 comp rx=0 tx=0).stop 2023-11-28T17:09:55.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.857+0000 7f4b0c0da640 1 -- 172.21.15.116:0/2755871872 shutdown_connections 2023-11-28T17:09:55.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.857+0000 7f4b0c0da640 1 --2- 172.21.15.116:0/2755871872 >> v2:172.21.15.156:3300/0 conn(0x7f4b040ce940 0x7f4b040cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.857+0000 7f4b0c0da640 1 --2- 172.21.15.116:0/2755871872 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4b0415b0c0 0x7f4b0415ed10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.857+0000 7f4b0c0da640 1 -- 172.21.15.116:0/2755871872 >> 172.21.15.116:0/2755871872 conn(0x7f4b040b0cf0 msgr2=0x7f4b040b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:55.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.858+0000 7f4b0c0da640 1 -- 172.21.15.116:0/2755871872 shutdown_connections 2023-11-28T17:09:55.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.858+0000 7f4b0c0da640 1 -- 172.21.15.116:0/2755871872 wait complete. 2023-11-28T17:09:55.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.860+0000 7f4b0c0da640 1 Processor -- start 2023-11-28T17:09:55.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.860+0000 7f4b0c0da640 1 -- start start 2023-11-28T17:09:55.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.861+0000 7f4b0c0da640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b04071b40 0x7f4b04155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.861+0000 7f4b0c0da640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4b040ce940 0x7f4b041f4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.861+0000 7f4b0c0da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b0415b0c0 0x7f4b0415da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.861+0000 7f4b0c0da640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b04157e20 con 0x7f4b0415b0c0 2023-11-28T17:09:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.861+0000 7f4b0c0da640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b041585f0 con 0x7f4b04071b40 2023-11-28T17:09:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.861+0000 7f4b0c0da640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b041588d0 con 0x7f4b040ce940 2023-11-28T17:09:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.861+0000 7f4b0a677640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b0415b0c0 0x7f4b0415da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.861+0000 7f4b0a677640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b0415b0c0 0x7f4b0415da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56590/0 (socket says 172.21.15.116:56590) 2023-11-28T17:09:55.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.861+0000 7f4b0a677640 1 -- 172.21.15.116:0/1082274651 learned_addr learned my addr 172.21.15.116:0/1082274651 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:55.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.862+0000 7f4b09675640 1 --2- 172.21.15.116:0/1082274651 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4b040ce940 0x7f4b041f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:55.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.862+0000 7f4b09e76640 1 --2- 172.21.15.116:0/1082274651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b04071b40 0x7f4b04155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:55.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.862+0000 7f4b0a677640 1 -- 172.21.15.116:0/1082274651 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4b040ce940 msgr2=0x7f4b041f4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:55.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.862+0000 7f4b0a677640 1 --2- 172.21.15.116:0/1082274651 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4b040ce940 0x7f4b041f4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.862+0000 7f4b0a677640 1 -- 172.21.15.116:0/1082274651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b04071b40 msgr2=0x7f4b04155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:55.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.862+0000 7f4b0a677640 1 --2- 172.21.15.116:0/1082274651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b04071b40 0x7f4b04155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:55.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.862+0000 7f4b0a677640 1 -- 172.21.15.116:0/1082274651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b00009400 con 0x7f4b0415b0c0 2023-11-28T17:09:55.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.862+0000 7f4b0a677640 1 --2- 172.21.15.116:0/1082274651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b0415b0c0 0x7f4b0415da70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4afc00c780 tx=0x7f4afc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:55.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.863+0000 7f4afa7fc640 1 -- 172.21.15.116:0/1082274651 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc00ce90 con 0x7f4b0415b0c0 2023-11-28T17:09:55.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.863+0000 7f4afa7fc640 1 -- 172.21.15.116:0/1082274651 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4afc00a640 con 0x7f4b0415b0c0 2023-11-28T17:09:55.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.863+0000 7f4afa7fc640 1 -- 172.21.15.116:0/1082274651 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc00a9f0 con 0x7f4b0415b0c0 2023-11-28T17:09:55.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.863+0000 7f4b0c0da640 1 -- 172.21.15.116:0/1082274651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b04158bb0 con 0x7f4b0415b0c0 2023-11-28T17:09:55.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.863+0000 7f4b0c0da640 1 -- 172.21.15.116:0/1082274651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b041561d0 con 0x7f4b0415b0c0 2023-11-28T17:09:55.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.864+0000 7f4b0c0da640 1 -- 172.21.15.116:0/1082274651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b041590d0 con 0x7f4b0415b0c0 2023-11-28T17:09:55.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.865+0000 7f4afa7fc640 1 -- 172.21.15.116:0/1082274651 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4afc00ab50 con 0x7f4b0415b0c0 2023-11-28T17:09:55.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.866+0000 7f4afa7fc640 1 --2- 172.21.15.116:0/1082274651 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4ae806b2a0 0x7f4ae806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:55.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.866+0000 7f4afa7fc640 1 -- 172.21.15.116:0/1082274651 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4afc089310 con 0x7f4b0415b0c0 2023-11-28T17:09:55.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.866+0000 7f4b09e76640 1 --2- 172.21.15.116:0/1082274651 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4ae806b2a0 0x7f4ae806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:55.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.867+0000 7f4b09e76640 1 --2- 172.21.15.116:0/1082274651 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4ae806b2a0 0x7f4ae806d760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4b00036000 tx=0x7f4b0003d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:55.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:55.870+0000 7f4afa7fc640 1 -- 172.21.15.116:0/1082274651 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b041590d0 con 0x7f4b0415b0c0 2023-11-28T17:09:56.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.061+0000 7f4b0c0da640 1 -- 172.21.15.116:0/1082274651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4b041590d0 con 0x7f4b0415b0c0 2023-11-28T17:09:56.063 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.062+0000 7f4afa7fc640 1 -- 172.21.15.116:0/1082274651 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4b041590d0 con 0x7f4b0415b0c0 2023-11-28T17:09:56.063 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:09:56.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.064+0000 7f4b0c0da640 1 -- 172.21.15.116:0/1082274651 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4ae806b2a0 msgr2=0x7f4ae806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:56.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.064+0000 7f4b0c0da640 1 --2- 172.21.15.116:0/1082274651 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4ae806b2a0 0x7f4ae806d760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4b00036000 tx=0x7f4b0003d040 comp rx=0 tx=0).stop 2023-11-28T17:09:56.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.065+0000 7f4b0c0da640 1 -- 172.21.15.116:0/1082274651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b0415b0c0 msgr2=0x7f4b0415da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:56.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.065+0000 7f4b0c0da640 1 --2- 172.21.15.116:0/1082274651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b0415b0c0 0x7f4b0415da70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4afc00c780 tx=0x7f4afc00cc40 comp rx=0 tx=0).stop 2023-11-28T17:09:56.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.065+0000 7f4b0c0da640 1 -- 172.21.15.116:0/1082274651 shutdown_connections 2023-11-28T17:09:56.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.065+0000 7f4b0c0da640 1 --2- 172.21.15.116:0/1082274651 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4ae806b2a0 0x7f4ae806d760 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.065+0000 7f4b0c0da640 1 --2- 172.21.15.116:0/1082274651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b0415b0c0 0x7f4b0415da70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.065+0000 7f4b0c0da640 1 --2- 172.21.15.116:0/1082274651 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4b040ce940 0x7f4b041f4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.065+0000 7f4b0c0da640 1 --2- 172.21.15.116:0/1082274651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b04071b40 0x7f4b04155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.065+0000 7f4b0c0da640 1 -- 172.21.15.116:0/1082274651 >> 172.21.15.116:0/1082274651 conn(0x7f4b040b0cf0 msgr2=0x7f4b0406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:56.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.065+0000 7f4b0c0da640 1 -- 172.21.15.116:0/1082274651 shutdown_connections 2023-11-28T17:09:56.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.065+0000 7f4b0c0da640 1 -- 172.21.15.116:0/1082274651 wait complete. 2023-11-28T17:09:56.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:09:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.299+0000 7f3625be9640 1 Processor -- start 2023-11-28T17:09:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.299+0000 7f3625be9640 1 -- start start 2023-11-28T17:09:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.301+0000 7f3625be9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3620152050 0x7f362015bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.301+0000 7f3625be9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f36200abd30 0x7f36200cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.301+0000 7f3625be9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36200b2b50 con 0x7f36200b2fe0 2023-11-28T17:09:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.301+0000 7f3625be9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f362015c520 con 0x7f3620152050 2023-11-28T17:09:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.301+0000 7f3625be9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36200ac2d0 con 0x7f36200abd30 2023-11-28T17:09:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.301+0000 7f3624be7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3620152050 0x7f362015bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.301+0000 7f3624be7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3620152050 0x7f362015bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56600/0 (socket says 172.21.15.116:56600) 2023-11-28T17:09:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.301+0000 7f361ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f36200b2fe0 0x7f36200ad930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51526/0 (socket says 172.21.15.116:51526) 2023-11-28T17:09:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.301+0000 7f3624be7640 1 -- 172.21.15.116:0/3809082882 learned_addr learned my addr 172.21.15.116:0/3809082882 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.302+0000 7f36253e8640 1 --2- 172.21.15.116:0/3809082882 >> v2:172.21.15.156:3300/0 conn(0x7f36200abd30 0x7f36200cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.302+0000 7f3624be7640 1 -- 172.21.15.116:0/3809082882 >> v2:172.21.15.156:3300/0 conn(0x7f36200abd30 msgr2=0x7f36200cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.302+0000 7f3624be7640 1 --2- 172.21.15.116:0/3809082882 >> v2:172.21.15.156:3300/0 conn(0x7f36200abd30 0x7f36200cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.302+0000 7f3624be7640 1 -- 172.21.15.116:0/3809082882 >> v1:172.21.15.116:6789/0 conn(0x7f36200b2fe0 legacy=0x7f36200ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.302+0000 7f3624be7640 1 -- 172.21.15.116:0/3809082882 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36200b2220 con 0x7f3620152050 2023-11-28T17:09:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.302+0000 7f361f7fe640 1 -- 172.21.15.116:0/3809082882 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786377352 0 0) 0x7f36200b2b50 con 0x7f36200b2fe0 2023-11-28T17:09:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.303+0000 7f3624be7640 1 --2- 172.21.15.116:0/3809082882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3620152050 0x7f362015bfe0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f36200b1930 tx=0x7f361402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=813d69b048d7ecfc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.303+0000 7f361f7fe640 1 -- 172.21.15.116:0/3809082882 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3614030860 con 0x7f3620152050 2023-11-28T17:09:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.303+0000 7f361f7fe640 1 -- 172.21.15.116:0/3809082882 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36140309c0 con 0x7f3620152050 2023-11-28T17:09:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.303+0000 7f361f7fe640 1 -- 172.21.15.116:0/3809082882 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3614030d50 con 0x7f3620152050 2023-11-28T17:09:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.304+0000 7f3625be9640 1 -- 172.21.15.116:0/3809082882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3620152050 msgr2=0x7f362015bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.304+0000 7f3625be9640 1 --2- 172.21.15.116:0/3809082882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3620152050 0x7f362015bfe0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f36200b1930 tx=0x7f361402b9a0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.304+0000 7f3625be9640 1 -- 172.21.15.116:0/3809082882 shutdown_connections 2023-11-28T17:09:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.304+0000 7f3625be9640 1 --2- 172.21.15.116:0/3809082882 >> v2:172.21.15.156:3300/0 conn(0x7f36200abd30 0x7f36200cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.304+0000 7f3625be9640 1 --2- 172.21.15.116:0/3809082882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3620152050 0x7f362015bfe0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.304+0000 7f3625be9640 1 -- 172.21.15.116:0/3809082882 >> 172.21.15.116:0/3809082882 conn(0x7f36200b0950 msgr2=0x7f36200b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.304+0000 7f3625be9640 1 -- 172.21.15.116:0/3809082882 shutdown_connections 2023-11-28T17:09:56.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.304+0000 7f3625be9640 1 -- 172.21.15.116:0/3809082882 wait complete. 2023-11-28T17:09:56.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.306+0000 7f3625be9640 1 Processor -- start 2023-11-28T17:09:56.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.306+0000 7f3625be9640 1 -- start start 2023-11-28T17:09:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.307+0000 7f3625be9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36200abd30 0x7f362013e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.307+0000 7f3625be9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36200b2fe0 0x7f3620071a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.307+0000 7f3625be9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3620152050 0x7f362015a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.307+0000 7f3625be9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362009ab10 con 0x7f36200b2fe0 2023-11-28T17:09:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.307+0000 7f3625be9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362009af30 con 0x7f3620152050 2023-11-28T17:09:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.307+0000 7f3625be9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3620099e90 con 0x7f36200abd30 2023-11-28T17:09:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.307+0000 7f361ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36200b2fe0 0x7f3620071a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.307+0000 7f361ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36200b2fe0 0x7f3620071a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56608/0 (socket says 172.21.15.116:56608) 2023-11-28T17:09:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.307+0000 7f361ffff640 1 -- 172.21.15.116:0/2740947527 learned_addr learned my addr 172.21.15.116:0/2740947527 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.308+0000 7f36253e8640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3620152050 0x7f362015a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.308+0000 7f3624be7640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36200abd30 0x7f362013e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.308+0000 7f361ffff640 1 -- 172.21.15.116:0/2740947527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36200abd30 msgr2=0x7f362013e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.308+0000 7f361ffff640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36200abd30 0x7f362013e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.308+0000 7f361ffff640 1 -- 172.21.15.116:0/2740947527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3620152050 msgr2=0x7f362015a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.308+0000 7f361ffff640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3620152050 0x7f362015a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.308+0000 7f361ffff640 1 -- 172.21.15.116:0/2740947527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36140094d0 con 0x7f36200b2fe0 2023-11-28T17:09:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.308+0000 7f36253e8640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3620152050 0x7f362015a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.308+0000 7f361ffff640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36200b2fe0 0x7f3620071a70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f360c00c4d0 tx=0x7f360c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.308+0000 7f361d7fa640 1 -- 172.21.15.116:0/2740947527 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c0074e0 con 0x7f36200b2fe0 2023-11-28T17:09:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.308+0000 7f361d7fa640 1 -- 172.21.15.116:0/2740947527 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360c007660 con 0x7f36200b2fe0 2023-11-28T17:09:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.308+0000 7f3625be9640 1 -- 172.21.15.116:0/2740947527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362009a170 con 0x7f36200b2fe0 2023-11-28T17:09:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.308+0000 7f361d7fa640 1 -- 172.21.15.116:0/2740947527 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c004a20 con 0x7f36200b2fe0 2023-11-28T17:09:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.308+0000 7f3625be9640 1 -- 172.21.15.116:0/2740947527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362009a730 con 0x7f36200b2fe0 2023-11-28T17:09:56.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.310+0000 7f3625be9640 1 -- 172.21.15.116:0/2740947527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362009ac50 con 0x7f36200b2fe0 2023-11-28T17:09:56.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.311+0000 7f361d7fa640 1 -- 172.21.15.116:0/2740947527 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f360c004b80 con 0x7f36200b2fe0 2023-11-28T17:09:56.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.311+0000 7f361d7fa640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f360406b2a0 0x7f360406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:56.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.312+0000 7f361d7fa640 1 -- 172.21.15.116:0/2740947527 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f360c088da0 con 0x7f36200b2fe0 2023-11-28T17:09:56.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.312+0000 7f3624be7640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f360406b2a0 0x7f360406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:56.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.312+0000 7f3624be7640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f360406b2a0 0x7f360406d760 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3614036000 tx=0x7f361403b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:56.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.316+0000 7f361d7fa640 1 -- 172.21.15.116:0/2740947527 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f360c053bf0 con 0x7f36200b2fe0 2023-11-28T17:09:56.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.507+0000 7f3625be9640 1 -- 172.21.15.116:0/2740947527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f362009d0b0 con 0x7f36200b2fe0 2023-11-28T17:09:56.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.508+0000 7f361d7fa640 1 -- 172.21.15.116:0/2740947527 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f360c018020 con 0x7f36200b2fe0 2023-11-28T17:09:56.510 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:09:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.511+0000 7f3625be9640 1 -- 172.21.15.116:0/2740947527 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f360406b2a0 msgr2=0x7f360406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.511+0000 7f3625be9640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f360406b2a0 0x7f360406d760 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3614036000 tx=0x7f361403b040 comp rx=0 tx=0).stop 2023-11-28T17:09:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.511+0000 7f3625be9640 1 -- 172.21.15.116:0/2740947527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36200b2fe0 msgr2=0x7f3620071a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:56.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.511+0000 7f3625be9640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36200b2fe0 0x7f3620071a70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f360c00c4d0 tx=0x7f360c00c990 comp rx=0 tx=0).stop 2023-11-28T17:09:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.511+0000 7f3625be9640 1 -- 172.21.15.116:0/2740947527 shutdown_connections 2023-11-28T17:09:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.511+0000 7f3625be9640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f360406b2a0 0x7f360406d760 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.511+0000 7f3625be9640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3620152050 0x7f362015a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.511+0000 7f3625be9640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36200b2fe0 0x7f3620071a70 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.511+0000 7f3625be9640 1 --2- 172.21.15.116:0/2740947527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36200abd30 0x7f362013e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.511+0000 7f3625be9640 1 -- 172.21.15.116:0/2740947527 >> 172.21.15.116:0/2740947527 conn(0x7f36200b0950 msgr2=0x7f362015c700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.511+0000 7f3625be9640 1 -- 172.21.15.116:0/2740947527 shutdown_connections 2023-11-28T17:09:56.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.511+0000 7f3625be9640 1 -- 172.21.15.116:0/2740947527 wait complete. 2023-11-28T17:09:56.530 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:09:56.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.745+0000 7f4f99261640 1 Processor -- start 2023-11-28T17:09:56.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.746+0000 7f4f99261640 1 -- start start 2023-11-28T17:09:56.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.747+0000 7f4f99261640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4f9415f7b0 0x7f4f94071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:56.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.747+0000 7f4f99261640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4f940721b0 0x7f4f9406c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:56.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.747+0000 7f4f99261640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f9415fc10 con 0x7f4f94144430 2023-11-28T17:09:56.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.747+0000 7f4f99261640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f94144810 con 0x7f4f940721b0 2023-11-28T17:09:56.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.747+0000 7f4f99261640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f94070bd0 con 0x7f4f9415f7b0 2023-11-28T17:09:56.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.747+0000 7f4f92ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4f94144430 0x7f4f940703f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51536/0 (socket says 172.21.15.116:51536) 2023-11-28T17:09:56.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.748+0000 7f4f92ffd640 1 -- 172.21.15.116:0/1399073252 learned_addr learned my addr 172.21.15.116:0/1399073252 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:56.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.748+0000 7f4f937fe640 1 --2- 172.21.15.116:0/1399073252 >> v2:172.21.15.116:3300/0 conn(0x7f4f940721b0 0x7f4f9406c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:56.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.748+0000 7f4f927fc640 1 -- 172.21.15.116:0/1399073252 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909724938 0 0) 0x7f4f9415fc10 con 0x7f4f94144430 2023-11-28T17:09:56.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.748+0000 7f4f8bfff640 1 --2- 172.21.15.116:0/1399073252 >> v2:172.21.15.156:3300/0 conn(0x7f4f9415f7b0 0x7f4f94071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:56.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.748+0000 7f4f927fc640 1 -- 172.21.15.116:0/1399073252 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f80003520 con 0x7f4f94144430 2023-11-28T17:09:56.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.748+0000 7f4f937fe640 1 -- 172.21.15.116:0/1399073252 >> v2:172.21.15.156:3300/0 conn(0x7f4f9415f7b0 msgr2=0x7f4f94071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:56.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.748+0000 7f4f937fe640 1 --2- 172.21.15.116:0/1399073252 >> v2:172.21.15.156:3300/0 conn(0x7f4f9415f7b0 0x7f4f94071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.748+0000 7f4f937fe640 1 -- 172.21.15.116:0/1399073252 >> v1:172.21.15.116:6789/0 conn(0x7f4f94144430 legacy=0x7f4f940703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:56.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.748+0000 7f4f927fc640 1 -- 172.21.15.116:0/1399073252 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2074179392 0 0) 0x7f4f80003520 con 0x7f4f94144430 2023-11-28T17:09:56.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.748+0000 7f4f937fe640 1 -- 172.21.15.116:0/1399073252 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f94070eb0 con 0x7f4f940721b0 2023-11-28T17:09:56.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.749+0000 7f4f937fe640 1 --2- 172.21.15.116:0/1399073252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f940721b0 0x7f4f9406c250 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4f94068020 tx=0x7f4f8c02bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1adf249bf43cc64f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:56.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.750+0000 7f4f927fc640 1 -- 172.21.15.116:0/1399073252 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84001670 con 0x7f4f940721b0 2023-11-28T17:09:56.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.750+0000 7f4f927fc640 1 -- 172.21.15.116:0/1399073252 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8c02c8c0 con 0x7f4f940721b0 2023-11-28T17:09:56.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.750+0000 7f4f927fc640 1 -- 172.21.15.116:0/1399073252 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c02cc50 con 0x7f4f940721b0 2023-11-28T17:09:56.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.750+0000 7f4f99261640 1 -- 172.21.15.116:0/1399073252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f940721b0 msgr2=0x7f4f9406c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:56.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.750+0000 7f4f99261640 1 --2- 172.21.15.116:0/1399073252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f940721b0 0x7f4f9406c250 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4f94068020 tx=0x7f4f8c02bd90 comp rx=0 tx=0).stop 2023-11-28T17:09:56.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.750+0000 7f4f99261640 1 -- 172.21.15.116:0/1399073252 shutdown_connections 2023-11-28T17:09:56.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.750+0000 7f4f99261640 1 --2- 172.21.15.116:0/1399073252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f940721b0 0x7f4f9406c250 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.750+0000 7f4f99261640 1 --2- 172.21.15.116:0/1399073252 >> v2:172.21.15.156:3300/0 conn(0x7f4f9415f7b0 0x7f4f94071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.750+0000 7f4f99261640 1 -- 172.21.15.116:0/1399073252 >> 172.21.15.116:0/1399073252 conn(0x7f4f9406a4c0 msgr2=0x7f4f9406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:56.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.750+0000 7f4f99261640 1 -- 172.21.15.116:0/1399073252 shutdown_connections 2023-11-28T17:09:56.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.751+0000 7f4f99261640 1 -- 172.21.15.116:0/1399073252 wait complete. 2023-11-28T17:09:56.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.753+0000 7f4f99261640 1 Processor -- start 2023-11-28T17:09:56.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.753+0000 7f4f99261640 1 -- start start 2023-11-28T17:09:56.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.754+0000 7f4f99261640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f940721b0 0x7f4f940c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:56.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.754+0000 7f4f99261640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f94144430 0x7f4f941f2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:56.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.754+0000 7f4f99261640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4f9415f7b0 0x7f4f9413e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:56.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.754+0000 7f4f99261640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f940c93b0 con 0x7f4f940721b0 2023-11-28T17:09:56.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.754+0000 7f4f99261640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f940c9d10 con 0x7f4f94144430 2023-11-28T17:09:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.754+0000 7f4f99261640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f940c8730 con 0x7f4f9415f7b0 2023-11-28T17:09:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.754+0000 7f4f92ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f940721b0 0x7f4f940c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.754+0000 7f4f92ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f940721b0 0x7f4f940c97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56626/0 (socket says 172.21.15.116:56626) 2023-11-28T17:09:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.754+0000 7f4f92ffd640 1 -- 172.21.15.116:0/2032718549 learned_addr learned my addr 172.21.15.116:0/2032718549 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.755+0000 7f4f92ffd640 1 -- 172.21.15.116:0/2032718549 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4f9415f7b0 msgr2=0x7f4f9413e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:56.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.755+0000 7f4f92ffd640 1 --2- 172.21.15.116:0/2032718549 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4f9415f7b0 0x7f4f9413e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.755+0000 7f4f92ffd640 1 -- 172.21.15.116:0/2032718549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f94144430 msgr2=0x7f4f941f2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.755+0000 7f4f92ffd640 1 --2- 172.21.15.116:0/2032718549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f94144430 0x7f4f941f2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.755+0000 7f4f92ffd640 1 -- 172.21.15.116:0/2032718549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c009a70 con 0x7f4f940721b0 2023-11-28T17:09:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.755+0000 7f4f92ffd640 1 --2- 172.21.15.116:0/2032718549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f940721b0 0x7f4f940c97d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4f8400c710 tx=0x7f4f8400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.755+0000 7f4f8b7fe640 1 -- 172.21.15.116:0/2032718549 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84005ad0 con 0x7f4f940721b0 2023-11-28T17:09:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.755+0000 7f4f8b7fe640 1 -- 172.21.15.116:0/2032718549 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f84005c30 con 0x7f4f940721b0 2023-11-28T17:09:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.755+0000 7f4f8b7fe640 1 -- 172.21.15.116:0/2032718549 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84007070 con 0x7f4f940721b0 2023-11-28T17:09:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.755+0000 7f4f99261640 1 -- 172.21.15.116:0/2032718549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f940c8a10 con 0x7f4f940721b0 2023-11-28T17:09:56.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.755+0000 7f4f99261640 1 -- 172.21.15.116:0/2032718549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f940c8fd0 con 0x7f4f940721b0 2023-11-28T17:09:56.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.757+0000 7f4f99261640 1 -- 172.21.15.116:0/2032718549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f940693d0 con 0x7f4f940721b0 2023-11-28T17:09:56.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.757+0000 7f4f8b7fe640 1 -- 172.21.15.116:0/2032718549 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4f840071f0 con 0x7f4f940721b0 2023-11-28T17:09:56.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.758+0000 7f4f8b7fe640 1 --2- 172.21.15.116:0/2032718549 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4f7406b2a0 0x7f4f7406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:56.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.758+0000 7f4f8b7fe640 1 -- 172.21.15.116:0/2032718549 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4f84089a30 con 0x7f4f940721b0 2023-11-28T17:09:56.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.759+0000 7f4f8bfff640 1 --2- 172.21.15.116:0/2032718549 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4f7406b2a0 0x7f4f7406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:56.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.759+0000 7f4f8bfff640 1 --2- 172.21.15.116:0/2032718549 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4f7406b2a0 0x7f4f7406d760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c006fd0 tx=0x7f4f7c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:56.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.764+0000 7f4f8b7fe640 1 -- 172.21.15.116:0/2032718549 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f84054870 con 0x7f4f940721b0 2023-11-28T17:09:56.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.967+0000 7f4f99261640 1 -- 172.21.15.116:0/2032718549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4f940cabf0 con 0x7f4f940721b0 2023-11-28T17:09:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.968+0000 7f4f8b7fe640 1 -- 172.21.15.116:0/2032718549 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4f84057e90 con 0x7f4f940721b0 2023-11-28T17:09:56.969 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-28T17:09:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.970+0000 7f4f99261640 1 -- 172.21.15.116:0/2032718549 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4f7406b2a0 msgr2=0x7f4f7406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.970+0000 7f4f99261640 1 --2- 172.21.15.116:0/2032718549 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4f7406b2a0 0x7f4f7406d760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c006fd0 tx=0x7f4f7c008040 comp rx=0 tx=0).stop 2023-11-28T17:09:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.970+0000 7f4f99261640 1 -- 172.21.15.116:0/2032718549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f940721b0 msgr2=0x7f4f940c97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:56.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.970+0000 7f4f99261640 1 --2- 172.21.15.116:0/2032718549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f940721b0 0x7f4f940c97d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4f8400c710 tx=0x7f4f8400cbd0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.970+0000 7f4f99261640 1 -- 172.21.15.116:0/2032718549 shutdown_connections 2023-11-28T17:09:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.970+0000 7f4f99261640 1 --2- 172.21.15.116:0/2032718549 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4f7406b2a0 0x7f4f7406d760 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.970+0000 7f4f99261640 1 --2- 172.21.15.116:0/2032718549 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4f9415f7b0 0x7f4f9413e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.970+0000 7f4f99261640 1 --2- 172.21.15.116:0/2032718549 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f94144430 0x7f4f941f2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.970+0000 7f4f99261640 1 --2- 172.21.15.116:0/2032718549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f940721b0 0x7f4f940c97d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.970+0000 7f4f99261640 1 -- 172.21.15.116:0/2032718549 >> 172.21.15.116:0/2032718549 conn(0x7f4f9406a4c0 msgr2=0x7f4f94143ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.971+0000 7f4f99261640 1 -- 172.21.15.116:0/2032718549 shutdown_connections 2023-11-28T17:09:56.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:56.971+0000 7f4f99261640 1 -- 172.21.15.116:0/2032718549 wait complete. 2023-11-28T17:09:56.991 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2023-11-28T17:09:56.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:56.991 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2023-11-28T17:09:57.084 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:09:57.084 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:09:57.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:09:57.084 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-11-28T17:09:57.138 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:09:57.139 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:09:57.200 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:09:57.200 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:09:57.254 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:57.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.475+0000 7f4eafbdc640 1 Processor -- start 2023-11-28T17:09:57.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.475+0000 7f4eafbdc640 1 -- start start 2023-11-28T17:09:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.477+0000 7f4eafbdc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea806c960 0x7f4ea815a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.477+0000 7f4eafbdc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4ea8140200 0x7f4ea815a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.477+0000 7f4eafbdc640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea80aea20 con 0x7f4ea813cee0 2023-11-28T17:09:57.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.477+0000 7f4eafbdc640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea813d2c0 con 0x7f4ea8140200 2023-11-28T17:09:57.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.477+0000 7f4eafbdc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea815b3a0 con 0x7f4ea806c960 2023-11-28T17:09:57.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.477+0000 7f4ead978640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4ea813cee0 0x7f4ea806c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51544/0 (socket says 172.21.15.116:51544) 2023-11-28T17:09:57.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.477+0000 7f4ead978640 1 -- 172.21.15.116:0/1136061091 learned_addr learned my addr 172.21.15.116:0/1136061091 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:57.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.477+0000 7f4ead177640 1 --2- 172.21.15.116:0/1136061091 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea806c960 0x7f4ea815a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.477+0000 7f4eae179640 1 --2- 172.21.15.116:0/1136061091 >> v2:172.21.15.156:3300/0 conn(0x7f4ea8140200 0x7f4ea815a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 -- 172.21.15.116:0/1136061091 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492020435 0 0) 0x7f4ea80aea20 con 0x7f4ea813cee0 2023-11-28T17:09:57.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 -- 172.21.15.116:0/1136061091 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e94003520 con 0x7f4ea813cee0 2023-11-28T17:09:57.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 -- 172.21.15.116:0/1136061091 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4085830438 0 0) 0x7f4e94003520 con 0x7f4ea813cee0 2023-11-28T17:09:57.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 -- 172.21.15.116:0/1136061091 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea806c960 msgr2=0x7f4ea815a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:57.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 --2- 172.21.15.116:0/1136061091 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea806c960 0x7f4ea815a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 -- 172.21.15.116:0/1136061091 >> v2:172.21.15.156:3300/0 conn(0x7f4ea8140200 msgr2=0x7f4ea815a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:57.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 --2- 172.21.15.116:0/1136061091 >> v2:172.21.15.156:3300/0 conn(0x7f4ea8140200 0x7f4ea815a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 -- 172.21.15.116:0/1136061091 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea815b0c0 con 0x7f4ea813cee0 2023-11-28T17:09:57.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 -- 172.21.15.116:0/1136061091 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f4ea4001670 con 0x7f4ea813cee0 2023-11-28T17:09:57.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 -- 172.21.15.116:0/1136061091 >> v1:172.21.15.116:6789/0 conn(0x7f4ea813cee0 legacy=0x7f4ea806c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:57.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 --2- 172.21.15.116:0/1136061091 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea806c960 0x7f4e94003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 --2- 172.21.15.116:0/1136061091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e94003ef0 0x7f4e940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 --2- 172.21.15.116:0/1136061091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e94004d00 0x7f4e94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 -- 172.21.15.116:0/1136061091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea813d2c0 con 0x7f4e94004d00 2023-11-28T17:09:57.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 -- 172.21.15.116:0/1136061091 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea815b3a0 con 0x7f4e94003ef0 2023-11-28T17:09:57.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eac976640 1 -- 172.21.15.116:0/1136061091 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9400b9d0 con 0x7f4ea806c960 2023-11-28T17:09:57.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.478+0000 7f4eae179640 1 --2- 172.21.15.116:0/1136061091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e94004d00 0x7f4e94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.479+0000 7f4ead978640 1 --2- 172.21.15.116:0/1136061091 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea806c960 0x7f4e94003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.479+0000 7f4ead177640 1 --2- 172.21.15.116:0/1136061091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e94003ef0 0x7f4e940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.479+0000 7f4eae179640 1 -- 172.21.15.116:0/1136061091 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea806c960 msgr2=0x7f4e94003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:57.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.479+0000 7f4eae179640 1 --2- 172.21.15.116:0/1136061091 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea806c960 0x7f4e94003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.479+0000 7f4eae179640 1 -- 172.21.15.116:0/1136061091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e94003ef0 msgr2=0x7f4e940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:57.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.479+0000 7f4eae179640 1 --2- 172.21.15.116:0/1136061091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e94003ef0 0x7f4e940047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.479+0000 7f4eae179640 1 -- 172.21.15.116:0/1136061091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e94003520 con 0x7f4e94004d00 2023-11-28T17:09:57.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.480+0000 7f4eae179640 1 --2- 172.21.15.116:0/1136061091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e94004d00 0x7f4e94005180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0009fc0 tx=0x7f4e940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b218f3903fdeef5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:57.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.480+0000 7f4eac976640 1 -- 172.21.15.116:0/1136061091 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea002cbf0 con 0x7f4e94004d00 2023-11-28T17:09:57.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.480+0000 7f4eac976640 1 -- 172.21.15.116:0/1136061091 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea000f040 con 0x7f4e94004d00 2023-11-28T17:09:57.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.480+0000 7f4eac976640 1 -- 172.21.15.116:0/1136061091 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0034440 con 0x7f4e94004d00 2023-11-28T17:09:57.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.481+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1136061091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e94004d00 msgr2=0x7f4e94005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:57.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.481+0000 7f4eafbdc640 1 --2- 172.21.15.116:0/1136061091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e94004d00 0x7f4e94005180 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0009fc0 tx=0x7f4e940057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.481+0000 7f4eae179640 1 -- 172.21.15.116:0/1136061091 reap_dead start 2023-11-28T17:09:57.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.481+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1136061091 shutdown_connections 2023-11-28T17:09:57.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.481+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1136061091 >> 172.21.15.116:0/1136061091 conn(0x7f4ea806a4c0 msgr2=0x7f4ea806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:57.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.481+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1136061091 shutdown_connections 2023-11-28T17:09:57.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.481+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1136061091 wait complete. 2023-11-28T17:09:57.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.483+0000 7f4eafbdc640 1 Processor -- start 2023-11-28T17:09:57.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.483+0000 7f4eafbdc640 1 -- start start 2023-11-28T17:09:57.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.484+0000 7f4eafbdc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e94004d00 0x7f4ea812eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.484+0000 7f4eafbdc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea813b8c0 0x7f4ea813e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.484+0000 7f4eafbdc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea813e9f0 0x7f4ea80b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.484+0000 7f4eafbdc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea812e790 con 0x7f4ea813e9f0 2023-11-28T17:09:57.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.484+0000 7f4eafbdc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea812e160 con 0x7f4e94004d00 2023-11-28T17:09:57.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.484+0000 7f4eafbdc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea813a1f0 con 0x7f4ea813b8c0 2023-11-28T17:09:57.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.484+0000 7f4eae179640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea813e9f0 0x7f4ea80b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.485+0000 7f4eae179640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea813e9f0 0x7f4ea80b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56652/0 (socket says 172.21.15.116:56652) 2023-11-28T17:09:57.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.485+0000 7f4eae179640 1 -- 172.21.15.116:0/1126964324 learned_addr learned my addr 172.21.15.116:0/1126964324 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:57.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.485+0000 7f4ead177640 1 --2- 172.21.15.116:0/1126964324 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea813b8c0 0x7f4ea813e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.485+0000 7f4ead978640 1 --2- 172.21.15.116:0/1126964324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e94004d00 0x7f4ea812eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.485+0000 7f4eae179640 1 -- 172.21.15.116:0/1126964324 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea813b8c0 msgr2=0x7f4ea813e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:57.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.485+0000 7f4eae179640 1 --2- 172.21.15.116:0/1126964324 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea813b8c0 0x7f4ea813e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.485+0000 7f4eae179640 1 -- 172.21.15.116:0/1126964324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e94004d00 msgr2=0x7f4ea812eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:57.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.485+0000 7f4eae179640 1 --2- 172.21.15.116:0/1126964324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e94004d00 0x7f4ea812eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.485+0000 7f4eae179640 1 -- 172.21.15.116:0/1126964324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea813ee90 con 0x7f4ea813e9f0 2023-11-28T17:09:57.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.485+0000 7f4eae179640 1 --2- 172.21.15.116:0/1126964324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea813e9f0 0x7f4ea80b30c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0037040 tx=0x7f4ea0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.486+0000 7f4e9e7fc640 1 -- 172.21.15.116:0/1126964324 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0034bc0 con 0x7f4ea813e9f0 2023-11-28T17:09:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.486+0000 7f4e9e7fc640 1 -- 172.21.15.116:0/1126964324 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea0034d40 con 0x7f4ea813e9f0 2023-11-28T17:09:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.486+0000 7f4e9e7fc640 1 -- 172.21.15.116:0/1126964324 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea003e680 con 0x7f4ea813e9f0 2023-11-28T17:09:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.486+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1126964324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea813f170 con 0x7f4ea813e9f0 2023-11-28T17:09:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.486+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1126964324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea812f0d0 con 0x7f4ea813e9f0 2023-11-28T17:09:57.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.487+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1126964324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea813c6c0 con 0x7f4ea813e9f0 2023-11-28T17:09:57.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.488+0000 7f4e9e7fc640 1 -- 172.21.15.116:0/1126964324 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4ea003e7e0 con 0x7f4ea813e9f0 2023-11-28T17:09:57.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.489+0000 7f4e9e7fc640 1 --2- 172.21.15.116:0/1126964324 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4e8c06b2a0 0x7f4e8c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.489+0000 7f4e9e7fc640 1 -- 172.21.15.116:0/1126964324 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4ea00abf40 con 0x7f4ea813e9f0 2023-11-28T17:09:57.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.489+0000 7f4ead978640 1 --2- 172.21.15.116:0/1126964324 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4e8c06b2a0 0x7f4e8c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.490+0000 7f4ead978640 1 --2- 172.21.15.116:0/1126964324 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4e8c06b2a0 0x7f4e8c06d760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4001050 tx=0x7f4ea4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:57.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.494+0000 7f4e9e7fc640 1 -- 172.21.15.116:0/1126964324 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ea0076d80 con 0x7f4ea813e9f0 2023-11-28T17:09:57.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.686+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1126964324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ea80693d0 con 0x7f4ea813e9f0 2023-11-28T17:09:57.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.689+0000 7f4e9e7fc640 1 -- 172.21.15.116:0/1126964324 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f4ea0042090 con 0x7f4ea813e9f0 2023-11-28T17:09:57.690 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:57.690 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:43.049465+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:57.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.691+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1126964324 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4e8c06b2a0 msgr2=0x7f4e8c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:57.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.691+0000 7f4eafbdc640 1 --2- 172.21.15.116:0/1126964324 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4e8c06b2a0 0x7f4e8c06d760 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4001050 tx=0x7f4ea4009040 comp rx=0 tx=0).stop 2023-11-28T17:09:57.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.691+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1126964324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea813e9f0 msgr2=0x7f4ea80b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:57.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.691+0000 7f4eafbdc640 1 --2- 172.21.15.116:0/1126964324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea813e9f0 0x7f4ea80b30c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0037040 tx=0x7f4ea0009750 comp rx=0 tx=0).stop 2023-11-28T17:09:57.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.691+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1126964324 shutdown_connections 2023-11-28T17:09:57.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.691+0000 7f4eafbdc640 1 --2- 172.21.15.116:0/1126964324 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4e8c06b2a0 0x7f4e8c06d760 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.691+0000 7f4eafbdc640 1 --2- 172.21.15.116:0/1126964324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ea813e9f0 0x7f4ea80b30c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.691+0000 7f4eafbdc640 1 --2- 172.21.15.116:0/1126964324 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4ea813b8c0 0x7f4ea813e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.691+0000 7f4eafbdc640 1 --2- 172.21.15.116:0/1126964324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e94004d00 0x7f4ea812eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.691+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1126964324 >> 172.21.15.116:0/1126964324 conn(0x7f4ea806a4c0 msgr2=0x7f4ea80ce030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:57.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.692+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1126964324 shutdown_connections 2023-11-28T17:09:57.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.692+0000 7f4eafbdc640 1 -- 172.21.15.116:0/1126964324 wait complete. 2023-11-28T17:09:57.713 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:09:57.714 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:09:57.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.929+0000 7f0a188f1640 1 Processor -- start 2023-11-28T17:09:57.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.929+0000 7f0a188f1640 1 -- start start 2023-11-28T17:09:57.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.930+0000 7f0a188f1640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0a14160b80 0x7f0a140d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.930+0000 7f0a188f1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a140b4010 0x7f0a140d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.930+0000 7f0a188f1640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a140d0da0 con 0x7f0a140cf6f0 2023-11-28T17:09:57.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.930+0000 7f0a188f1640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a14160f60 con 0x7f0a140b4010 2023-11-28T17:09:57.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.930+0000 7f0a188f1640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a140d1080 con 0x7f0a14160b80 2023-11-28T17:09:57.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.931+0000 7f0a0affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a140b4010 0x7f0a140d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.931+0000 7f0a0affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a140b4010 0x7f0a140d18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59958/0 (socket says 172.21.15.116:59958) 2023-11-28T17:09:57.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.931+0000 7f0a0affd640 1 -- 172.21.15.116:0/2134534409 learned_addr learned my addr 172.21.15.116:0/2134534409 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:57.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.931+0000 7f0a137fe640 1 --2- 172.21.15.116:0/2134534409 >> v2:172.21.15.156:3300/0 conn(0x7f0a14160b80 0x7f0a140d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.931+0000 7f0a0affd640 1 -- 172.21.15.116:0/2134534409 >> v2:172.21.15.156:3300/0 conn(0x7f0a14160b80 msgr2=0x7f0a140d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:57.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.931+0000 7f0a0affd640 1 --2- 172.21.15.116:0/2134534409 >> v2:172.21.15.156:3300/0 conn(0x7f0a14160b80 0x7f0a140d1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.931+0000 7f0a0affd640 1 -- 172.21.15.116:0/2134534409 >> v1:172.21.15.156:6789/0 conn(0x7f0a140cf6f0 legacy=0x7f0a140cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:57.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.931+0000 7f0a0affd640 1 -- 172.21.15.116:0/2134534409 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a140ce780 con 0x7f0a140b4010 2023-11-28T17:09:57.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.932+0000 7f0a0affd640 1 --2- 172.21.15.116:0/2134534409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a140b4010 0x7f0a140d18a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0a140d2f90 tx=0x7f09fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25e785f46ea1c33 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.933+0000 7f0a12ffd640 1 -- 172.21.15.116:0/2134534409 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc031870 con 0x7f0a140b4010 2023-11-28T17:09:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.933+0000 7f0a12ffd640 1 -- 172.21.15.116:0/2134534409 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09fc0319d0 con 0x7f0a140b4010 2023-11-28T17:09:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.933+0000 7f0a12ffd640 1 -- 172.21.15.116:0/2134534409 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc031d60 con 0x7f0a140b4010 2023-11-28T17:09:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.933+0000 7f0a188f1640 1 -- 172.21.15.116:0/2134534409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a140b4010 msgr2=0x7f0a140d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.933+0000 7f0a188f1640 1 --2- 172.21.15.116:0/2134534409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a140b4010 0x7f0a140d18a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0a140d2f90 tx=0x7f09fc02b910 comp rx=0 tx=0).stop 2023-11-28T17:09:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.933+0000 7f0a188f1640 1 -- 172.21.15.116:0/2134534409 shutdown_connections 2023-11-28T17:09:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.933+0000 7f0a188f1640 1 --2- 172.21.15.116:0/2134534409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a140b4010 0x7f0a140d18a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.933+0000 7f0a188f1640 1 --2- 172.21.15.116:0/2134534409 >> v2:172.21.15.156:3300/0 conn(0x7f0a14160b80 0x7f0a140d1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.933+0000 7f0a188f1640 1 -- 172.21.15.116:0/2134534409 >> 172.21.15.116:0/2134534409 conn(0x7f0a140d2fc0 msgr2=0x7f0a140d33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.933+0000 7f0a188f1640 1 -- 172.21.15.116:0/2134534409 shutdown_connections 2023-11-28T17:09:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.933+0000 7f0a188f1640 1 -- 172.21.15.116:0/2134534409 wait complete. 2023-11-28T17:09:57.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.935+0000 7f0a188f1640 1 Processor -- start 2023-11-28T17:09:57.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.935+0000 7f0a188f1640 1 -- start start 2023-11-28T17:09:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.936+0000 7f0a188f1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a140b4010 0x7f0a140b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.936+0000 7f0a188f1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a140cf6f0 0x7f0a14158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.936+0000 7f0a188f1640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a14160b80 0x7f0a140b7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.936+0000 7f0a188f1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a140ca770 con 0x7f0a140b4010 2023-11-28T17:09:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.936+0000 7f0a188f1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a140c7500 con 0x7f0a140cf6f0 2023-11-28T17:09:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.936+0000 7f0a188f1640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a140b61c0 con 0x7f0a14160b80 2023-11-28T17:09:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.936+0000 7f0a137fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a140b4010 0x7f0a140b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.936+0000 7f0a137fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a140b4010 0x7f0a140b5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59966/0 (socket says 172.21.15.116:59966) 2023-11-28T17:09:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.936+0000 7f0a137fe640 1 -- 172.21.15.116:0/3397348917 learned_addr learned my addr 172.21.15.116:0/3397348917 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.937+0000 7f0a0affd640 1 --2- 172.21.15.116:0/3397348917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a140cf6f0 0x7f0a14158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.937+0000 7f0a13fff640 1 --2- 172.21.15.116:0/3397348917 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a14160b80 0x7f0a140b7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.937+0000 7f0a137fe640 1 -- 172.21.15.116:0/3397348917 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a14160b80 msgr2=0x7f0a140b7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.937+0000 7f0a137fe640 1 --2- 172.21.15.116:0/3397348917 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a14160b80 0x7f0a140b7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.937+0000 7f0a137fe640 1 -- 172.21.15.116:0/3397348917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a140cf6f0 msgr2=0x7f0a14158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.937+0000 7f0a137fe640 1 --2- 172.21.15.116:0/3397348917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a140cf6f0 0x7f0a14158220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.937+0000 7f0a137fe640 1 -- 172.21.15.116:0/3397348917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc0094d0 con 0x7f0a140b4010 2023-11-28T17:09:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.937+0000 7f0a137fe640 1 --2- 172.21.15.116:0/3397348917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a140b4010 0x7f0a140b5be0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0a0400c7f0 tx=0x7f0a0400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.937+0000 7f0a10ff9640 1 -- 172.21.15.116:0/3397348917 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a04004220 con 0x7f0a140b4010 2023-11-28T17:09:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.937+0000 7f0a10ff9640 1 -- 172.21.15.116:0/3397348917 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a04004380 con 0x7f0a140b4010 2023-11-28T17:09:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.937+0000 7f0a10ff9640 1 -- 172.21.15.116:0/3397348917 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a04007840 con 0x7f0a140b4010 2023-11-28T17:09:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.938+0000 7f0a188f1640 1 -- 172.21.15.116:0/3397348917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a140c7750 con 0x7f0a140b4010 2023-11-28T17:09:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.938+0000 7f0a188f1640 1 -- 172.21.15.116:0/3397348917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a140c7d10 con 0x7f0a140b4010 2023-11-28T17:09:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.939+0000 7f0a188f1640 1 -- 172.21.15.116:0/3397348917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a14067980 con 0x7f0a140b4010 2023-11-28T17:09:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.940+0000 7f0a10ff9640 1 -- 172.21.15.116:0/3397348917 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0a04007bb0 con 0x7f0a140b4010 2023-11-28T17:09:57.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.941+0000 7f0a10ff9640 1 --2- 172.21.15.116:0/3397348917 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f09f406b2d0 0x7f09f406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:57.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.941+0000 7f0a10ff9640 1 -- 172.21.15.116:0/3397348917 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0a04089b00 con 0x7f0a140b4010 2023-11-28T17:09:57.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.941+0000 7f0a0affd640 1 --2- 172.21.15.116:0/3397348917 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f09f406b2d0 0x7f09f406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:57.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.942+0000 7f0a0affd640 1 --2- 172.21.15.116:0/3397348917 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f09f406b2d0 0x7f09f406d790 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f09fc036000 tx=0x7f09fc03c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:57.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:57.945+0000 7f0a10ff9640 1 -- 172.21.15.116:0/3397348917 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a0408be60 con 0x7f0a140b4010 2023-11-28T17:09:58.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.135+0000 7f0a188f1640 1 -- 172.21.15.116:0/3397348917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a140a0eb0 con 0x7f0a140b4010 2023-11-28T17:09:58.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.136+0000 7f0a10ff9640 1 -- 172.21.15.116:0/3397348917 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0a0401e020 con 0x7f0a140b4010 2023-11-28T17:09:58.138 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:09:58.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.139+0000 7f0a188f1640 1 -- 172.21.15.116:0/3397348917 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f09f406b2d0 msgr2=0x7f09f406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:58.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.139+0000 7f0a188f1640 1 --2- 172.21.15.116:0/3397348917 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f09f406b2d0 0x7f09f406d790 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f09fc036000 tx=0x7f09fc03c040 comp rx=0 tx=0).stop 2023-11-28T17:09:58.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.140+0000 7f0a188f1640 1 -- 172.21.15.116:0/3397348917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a140b4010 msgr2=0x7f0a140b5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:58.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.140+0000 7f0a188f1640 1 --2- 172.21.15.116:0/3397348917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a140b4010 0x7f0a140b5be0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0a0400c7f0 tx=0x7f0a0400ccb0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.140+0000 7f0a188f1640 1 -- 172.21.15.116:0/3397348917 shutdown_connections 2023-11-28T17:09:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.140+0000 7f0a188f1640 1 --2- 172.21.15.116:0/3397348917 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f09f406b2d0 0x7f09f406d790 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.140+0000 7f0a188f1640 1 --2- 172.21.15.116:0/3397348917 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a14160b80 0x7f0a140b7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.140+0000 7f0a188f1640 1 --2- 172.21.15.116:0/3397348917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a140cf6f0 0x7f0a14158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.140+0000 7f0a188f1640 1 --2- 172.21.15.116:0/3397348917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a140b4010 0x7f0a140b5be0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.140+0000 7f0a188f1640 1 -- 172.21.15.116:0/3397348917 >> 172.21.15.116:0/3397348917 conn(0x7f0a140d2fc0 msgr2=0x7f0a140733f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:58.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.140+0000 7f0a188f1640 1 -- 172.21.15.116:0/3397348917 shutdown_connections 2023-11-28T17:09:58.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.140+0000 7f0a188f1640 1 -- 172.21.15.116:0/3397348917 wait complete. 2023-11-28T17:09:58.159 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:09:58.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.360+0000 7f85c1b16640 1 Processor -- start 2023-11-28T17:09:58.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.361+0000 7f85c1b16640 1 -- start start 2023-11-28T17:09:58.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.362+0000 7f85c1b16640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f85bc0b20a0 0x7f85bc0d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:58.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.362+0000 7f85c1b16640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85bc0b2ad0 0x7f85bc0ce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:58.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.362+0000 7f85c1b16640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85bc0cdcb0 con 0x7f85bc0d1780 2023-11-28T17:09:58.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.362+0000 7f85c1b16640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85bc0d39e0 con 0x7f85bc0b20a0 2023-11-28T17:09:58.364 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.362+0000 7f85c1b16640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc0d3cc0 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.362+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85bc0b2ad0 0x7f85bc0ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:58.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.363+0000 7f85bbfff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85bc0b2ad0 0x7f85bc0ce750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:42612/0 (socket says 172.21.15.116:42612) 2023-11-28T17:09:58.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.363+0000 7f85bbfff640 1 -- 172.21.15.116:0/950143042 learned_addr learned my addr 172.21.15.116:0/950143042 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:58.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.363+0000 7f85b2ffd640 1 --2- 172.21.15.116:0/950143042 >> v2:172.21.15.156:3300/0 conn(0x7f85bc0b20a0 0x7f85bc0d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:58.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.363+0000 7f85bbfff640 1 -- 172.21.15.116:0/950143042 >> v2:172.21.15.156:3300/0 conn(0x7f85bc0b20a0 msgr2=0x7f85bc0d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:58.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.363+0000 7f85bbfff640 1 --2- 172.21.15.116:0/950143042 >> v2:172.21.15.156:3300/0 conn(0x7f85bc0b20a0 0x7f85bc0d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.363+0000 7f85bbfff640 1 -- 172.21.15.116:0/950143042 >> v1:172.21.15.156:6789/0 conn(0x7f85bc0d1780 legacy=0x7f85bc0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:58.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.363+0000 7f85bbfff640 1 -- 172.21.15.116:0/950143042 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85bc0cdf50 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.365+0000 7f85bbfff640 1 --2- 172.21.15.116:0/950143042 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85bc0b2ad0 0x7f85bc0ce750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f85b4009e50 tx=0x7f85b402bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb38c8272cd8df27 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.365+0000 7f85baffd640 1 -- 172.21.15.116:0/950143042 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b402ca80 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.365+0000 7f85baffd640 1 -- 172.21.15.116:0/950143042 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b402cbe0 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.365+0000 7f85baffd640 1 -- 172.21.15.116:0/950143042 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b40343f0 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.366+0000 7f85c1b16640 1 -- 172.21.15.116:0/950143042 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85bc0b2ad0 msgr2=0x7f85bc0ce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.366+0000 7f85c1b16640 1 --2- 172.21.15.116:0/950143042 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85bc0b2ad0 0x7f85bc0ce750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f85b4009e50 tx=0x7f85b402bee0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.366+0000 7f85c1b16640 1 -- 172.21.15.116:0/950143042 shutdown_connections 2023-11-28T17:09:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.366+0000 7f85c1b16640 1 --2- 172.21.15.116:0/950143042 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85bc0b2ad0 0x7f85bc0ce750 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.366+0000 7f85c1b16640 1 --2- 172.21.15.116:0/950143042 >> v2:172.21.15.156:3300/0 conn(0x7f85bc0b20a0 0x7f85bc0d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.366+0000 7f85c1b16640 1 -- 172.21.15.116:0/950143042 >> 172.21.15.116:0/950143042 conn(0x7f85bc09c4e0 msgr2=0x7f85bc0d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.366+0000 7f85c1b16640 1 -- 172.21.15.116:0/950143042 shutdown_connections 2023-11-28T17:09:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.366+0000 7f85c1b16640 1 -- 172.21.15.116:0/950143042 wait complete. 2023-11-28T17:09:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.368+0000 7f85c1b16640 1 Processor -- start 2023-11-28T17:09:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.368+0000 7f85c1b16640 1 -- start start 2023-11-28T17:09:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85c1b16640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85bc0b20a0 0x7f85bc0b5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85c1b16640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85bc0b2ad0 0x7f85bc1f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85c1b16640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85bc0d1780 0x7f85bc0b7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85c1b16640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc0b7220 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85c1b16640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc0b6610 con 0x7f85bc0d1780 2023-11-28T17:09:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85c1b16640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85bc0b68f0 con 0x7f85bc0b20a0 2023-11-28T17:09:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85b2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85bc0b2ad0 0x7f85bc1f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85b2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85bc0b2ad0 0x7f85bc1f24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59978/0 (socket says 172.21.15.116:59978) 2023-11-28T17:09:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85b2ffd640 1 -- 172.21.15.116:0/1725083243 learned_addr learned my addr 172.21.15.116:0/1725083243 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85bb7fe640 1 --2- 172.21.15.116:0/1725083243 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85bc0b20a0 0x7f85bc0b5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85bbfff640 1 --2- 172.21.15.116:0/1725083243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85bc0d1780 0x7f85bc0b7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85b2ffd640 1 -- 172.21.15.116:0/1725083243 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85bc0b20a0 msgr2=0x7f85bc0b5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85b2ffd640 1 --2- 172.21.15.116:0/1725083243 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85bc0b20a0 0x7f85bc0b5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85b2ffd640 1 -- 172.21.15.116:0/1725083243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85bc0d1780 msgr2=0x7f85bc0b7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85b2ffd640 1 --2- 172.21.15.116:0/1725083243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85bc0d1780 0x7f85bc0b7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.369+0000 7f85b2ffd640 1 -- 172.21.15.116:0/1725083243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b4009b30 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.370+0000 7f85b2ffd640 1 --2- 172.21.15.116:0/1725083243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85bc0b2ad0 0x7f85bc1f24c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f85a400e980 tx=0x7f85a400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.370+0000 7f85b8ff9640 1 -- 172.21.15.116:0/1725083243 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a4009670 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.370+0000 7f85b8ff9640 1 -- 172.21.15.116:0/1725083243 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a400ca00 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.370+0000 7f85b8ff9640 1 -- 172.21.15.116:0/1725083243 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a400cd90 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.370+0000 7f85c1b16640 1 -- 172.21.15.116:0/1725083243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc0b6bd0 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.370+0000 7f85c1b16640 1 -- 172.21.15.116:0/1725083243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85bc0b6300 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.371+0000 7f85c1b16640 1 -- 172.21.15.116:0/1725083243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85bc069360 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.372+0000 7f85b8ff9640 1 -- 172.21.15.116:0/1725083243 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f85a4013470 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.374+0000 7f85b8ff9640 1 --2- 172.21.15.116:0/1725083243 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f859c06b2a0 0x7f859c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.374+0000 7f85b8ff9640 1 -- 172.21.15.116:0/1725083243 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f85a4089e10 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.374+0000 7f85bb7fe640 1 --2- 172.21.15.116:0/1725083243 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f859c06b2a0 0x7f859c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.375+0000 7f85bb7fe640 1 --2- 172.21.15.116:0/1725083243 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f859c06b2a0 0x7f859c06d760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f85ac000f60 tx=0x7f85ac009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.378+0000 7f85b8ff9640 1 -- 172.21.15.116:0/1725083243 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f85a4054c60 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.568+0000 7f85c1b16640 1 -- 172.21.15.116:0/1725083243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f85bc0d40b0 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.569+0000 7f85b8ff9640 1 -- 172.21.15.116:0/1725083243 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f85a4017020 con 0x7f85bc0b2ad0 2023-11-28T17:09:58.570 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:09:58.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.571+0000 7f85c1b16640 1 -- 172.21.15.116:0/1725083243 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f859c06b2a0 msgr2=0x7f859c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:58.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.571+0000 7f85c1b16640 1 --2- 172.21.15.116:0/1725083243 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f859c06b2a0 0x7f859c06d760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f85ac000f60 tx=0x7f85ac009040 comp rx=0 tx=0).stop 2023-11-28T17:09:58.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.571+0000 7f85c1b16640 1 -- 172.21.15.116:0/1725083243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85bc0b2ad0 msgr2=0x7f85bc1f24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:58.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.571+0000 7f85c1b16640 1 --2- 172.21.15.116:0/1725083243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85bc0b2ad0 0x7f85bc1f24c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f85a400e980 tx=0x7f85a400ee40 comp rx=0 tx=0).stop 2023-11-28T17:09:58.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.571+0000 7f85c1b16640 1 -- 172.21.15.116:0/1725083243 shutdown_connections 2023-11-28T17:09:58.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.571+0000 7f85c1b16640 1 --2- 172.21.15.116:0/1725083243 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f859c06b2a0 0x7f859c06d760 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.571+0000 7f85c1b16640 1 --2- 172.21.15.116:0/1725083243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85bc0d1780 0x7f85bc0b7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.571+0000 7f85c1b16640 1 --2- 172.21.15.116:0/1725083243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85bc0b2ad0 0x7f85bc1f24c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.571+0000 7f85c1b16640 1 --2- 172.21.15.116:0/1725083243 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f85bc0b20a0 0x7f85bc0b5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.571+0000 7f85c1b16640 1 -- 172.21.15.116:0/1725083243 >> 172.21.15.116:0/1725083243 conn(0x7f85bc09c4e0 msgr2=0x7f85bc0d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:58.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.572+0000 7f85c1b16640 1 -- 172.21.15.116:0/1725083243 shutdown_connections 2023-11-28T17:09:58.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.572+0000 7f85c1b16640 1 -- 172.21.15.116:0/1725083243 wait complete. 2023-11-28T17:09:58.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:09:58.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.804+0000 7f0f9b22b640 1 Processor -- start 2023-11-28T17:09:58.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.804+0000 7f0f9b22b640 1 -- start start 2023-11-28T17:09:58.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.806+0000 7f0f9b22b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0f9414a740 0x7f0f940cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:58.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.806+0000 7f0f9b22b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0f9414ac30 0x7f0f940b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:58.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.806+0000 7f0f9b22b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f94070420 con 0x7f0f940b2a00 2023-11-28T17:09:58.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.806+0000 7f0f9b22b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f94070aa0 con 0x7f0f9414a740 2023-11-28T17:09:58.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.806+0000 7f0f9b22b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f940b2fc0 con 0x7f0f9414ac30 2023-11-28T17:09:58.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.806+0000 7f0f98fc7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0f9414a740 0x7f0f940cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:58.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.806+0000 7f0f98fc7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0f9414a740 0x7f0f940cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59982/0 (socket says 172.21.15.116:59982) 2023-11-28T17:09:58.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.806+0000 7f0f98fc7640 1 -- 172.21.15.116:0/1893444805 learned_addr learned my addr 172.21.15.116:0/1893444805 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:58.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.806+0000 7f0f93fff640 1 --2- 172.21.15.116:0/1893444805 >> v2:172.21.15.156:3300/0 conn(0x7f0f9414ac30 0x7f0f940b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:58.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.806+0000 7f0f98fc7640 1 -- 172.21.15.116:0/1893444805 >> v2:172.21.15.156:3300/0 conn(0x7f0f9414ac30 msgr2=0x7f0f940b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:58.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.806+0000 7f0f98fc7640 1 --2- 172.21.15.116:0/1893444805 >> v2:172.21.15.156:3300/0 conn(0x7f0f9414ac30 0x7f0f940b3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.806+0000 7f0f98fc7640 1 -- 172.21.15.116:0/1893444805 >> v1:172.21.15.116:6789/0 conn(0x7f0f940b2a00 legacy=0x7f0f94070ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:58.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.807+0000 7f0f98fc7640 1 -- 172.21.15.116:0/1893444805 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f940706c0 con 0x7f0f9414a740 2023-11-28T17:09:58.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.807+0000 7f0f937fe640 1 -- 172.21.15.116:0/1893444805 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153110862 0 0) 0x7f0f94070420 con 0x7f0f940b2a00 2023-11-28T17:09:58.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.808+0000 7f0f98fc7640 1 --2- 172.21.15.116:0/1893444805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9414a740 0x7f0f940cc7a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0f940cc230 tx=0x7f0f8802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68f6672c6e390c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:58.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.808+0000 7f0f937fe640 1 -- 172.21.15.116:0/1893444805 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88031870 con 0x7f0f9414a740 2023-11-28T17:09:58.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.808+0000 7f0f937fe640 1 -- 172.21.15.116:0/1893444805 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f880319d0 con 0x7f0f9414a740 2023-11-28T17:09:58.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.808+0000 7f0f937fe640 1 -- 172.21.15.116:0/1893444805 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88031d60 con 0x7f0f9414a740 2023-11-28T17:09:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.808+0000 7f0f9b22b640 1 -- 172.21.15.116:0/1893444805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9414a740 msgr2=0x7f0f940cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.808+0000 7f0f9b22b640 1 --2- 172.21.15.116:0/1893444805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9414a740 0x7f0f940cc7a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0f940cc230 tx=0x7f0f8802b910 comp rx=0 tx=0).stop 2023-11-28T17:09:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.808+0000 7f0f9b22b640 1 -- 172.21.15.116:0/1893444805 shutdown_connections 2023-11-28T17:09:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.808+0000 7f0f9b22b640 1 --2- 172.21.15.116:0/1893444805 >> v2:172.21.15.156:3300/0 conn(0x7f0f9414ac30 0x7f0f940b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.808+0000 7f0f9b22b640 1 --2- 172.21.15.116:0/1893444805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9414a740 0x7f0f940cc7a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.808+0000 7f0f9b22b640 1 -- 172.21.15.116:0/1893444805 >> 172.21.15.116:0/1893444805 conn(0x7f0f940b0cf0 msgr2=0x7f0f940b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.809+0000 7f0f9b22b640 1 -- 172.21.15.116:0/1893444805 shutdown_connections 2023-11-28T17:09:58.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.809+0000 7f0f9b22b640 1 -- 172.21.15.116:0/1893444805 wait complete. 2023-11-28T17:09:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.810+0000 7f0f9b22b640 1 Processor -- start 2023-11-28T17:09:58.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.810+0000 7f0f9b22b640 1 -- start start 2023-11-28T17:09:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.811+0000 7f0f9b22b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0f940b2a00 0x7f0f94161220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.811+0000 7f0f9b22b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f9414a740 0x7f0f94144af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:58.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.811+0000 7f0f9b22b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9414ac30 0x7f0f940b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.811+0000 7f0f9b22b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f94140930 con 0x7f0f9414ac30 2023-11-28T17:09:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.811+0000 7f0f9b22b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f94140b40 con 0x7f0f9414a740 2023-11-28T17:09:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.811+0000 7f0f9b22b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f94140e20 con 0x7f0f940b2a00 2023-11-28T17:09:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.812+0000 7f0f997c8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9414ac30 0x7f0f940b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.812+0000 7f0f997c8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9414ac30 0x7f0f940b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59990/0 (socket says 172.21.15.116:59990) 2023-11-28T17:09:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.812+0000 7f0f997c8640 1 -- 172.21.15.116:0/921971038 learned_addr learned my addr 172.21.15.116:0/921971038 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:58.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.812+0000 7f0f93fff640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f9414a740 0x7f0f94144af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.812+0000 7f0f98fc7640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0f940b2a00 0x7f0f94161220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.812+0000 7f0f997c8640 1 -- 172.21.15.116:0/921971038 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0f940b2a00 msgr2=0x7f0f94161220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.812+0000 7f0f997c8640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0f940b2a00 0x7f0f94161220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.812+0000 7f0f997c8640 1 -- 172.21.15.116:0/921971038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f9414a740 msgr2=0x7f0f94144af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:58.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.812+0000 7f0f997c8640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f9414a740 0x7f0f94144af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.812+0000 7f0f997c8640 1 -- 172.21.15.116:0/921971038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f880094d0 con 0x7f0f9414ac30 2023-11-28T17:09:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.812+0000 7f0f98fc7640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0f940b2a00 0x7f0f94161220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.812+0000 7f0f93fff640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f9414a740 0x7f0f94144af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:09:58.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.812+0000 7f0f997c8640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9414ac30 0x7f0f940b4970 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c001560 tx=0x7f0f8c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.813+0000 7f0f917fa640 1 -- 172.21.15.116:0/921971038 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c003da0 con 0x7f0f9414ac30 2023-11-28T17:09:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.813+0000 7f0f917fa640 1 -- 172.21.15.116:0/921971038 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8c005070 con 0x7f0f9414ac30 2023-11-28T17:09:58.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.813+0000 7f0f917fa640 1 -- 172.21.15.116:0/921971038 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c005420 con 0x7f0f9414ac30 2023-11-28T17:09:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.813+0000 7f0f9b22b640 1 -- 172.21.15.116:0/921971038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f94141100 con 0x7f0f9414ac30 2023-11-28T17:09:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.813+0000 7f0f9b22b640 1 -- 172.21.15.116:0/921971038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f941416c0 con 0x7f0f9414ac30 2023-11-28T17:09:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.814+0000 7f0f9b22b640 1 -- 172.21.15.116:0/921971038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f940cbe70 con 0x7f0f9414ac30 2023-11-28T17:09:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.815+0000 7f0f917fa640 1 -- 172.21.15.116:0/921971038 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0f8c005580 con 0x7f0f9414ac30 2023-11-28T17:09:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.815+0000 7f0f917fa640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0f8006b2d0 0x7f0f8006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.816+0000 7f0f917fa640 1 -- 172.21.15.116:0/921971038 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0f8c0894d0 con 0x7f0f9414ac30 2023-11-28T17:09:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.816+0000 7f0f98fc7640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0f8006b2d0 0x7f0f8006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.816+0000 7f0f98fc7640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0f8006b2d0 0x7f0f8006d790 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0f88036000 tx=0x7f0f8803c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:58.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:58.820+0000 7f0f917fa640 1 -- 172.21.15.116:0/921971038 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0f8c054320 con 0x7f0f9414ac30 2023-11-28T17:09:59.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.009+0000 7f0f9b22b640 1 -- 172.21.15.116:0/921971038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0f941418e0 con 0x7f0f9414ac30 2023-11-28T17:09:59.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.010+0000 7f0f917fa640 1 -- 172.21.15.116:0/921971038 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0f8c017020 con 0x7f0f9414ac30 2023-11-28T17:09:59.012 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:09:59.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.013+0000 7f0f9b22b640 1 -- 172.21.15.116:0/921971038 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0f8006b2d0 msgr2=0x7f0f8006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:59.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.013+0000 7f0f9b22b640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0f8006b2d0 0x7f0f8006d790 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0f88036000 tx=0x7f0f8803c040 comp rx=0 tx=0).stop 2023-11-28T17:09:59.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.013+0000 7f0f9b22b640 1 -- 172.21.15.116:0/921971038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9414ac30 msgr2=0x7f0f940b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:59.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.013+0000 7f0f9b22b640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9414ac30 0x7f0f940b4970 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c001560 tx=0x7f0f8c00cc70 comp rx=0 tx=0).stop 2023-11-28T17:09:59.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.013+0000 7f0f9b22b640 1 -- 172.21.15.116:0/921971038 shutdown_connections 2023-11-28T17:09:59.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.013+0000 7f0f9b22b640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0f8006b2d0 0x7f0f8006d790 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.013+0000 7f0f9b22b640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f9414ac30 0x7f0f940b4970 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.013+0000 7f0f9b22b640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f9414a740 0x7f0f94144af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.013+0000 7f0f9b22b640 1 --2- 172.21.15.116:0/921971038 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0f940b2a00 0x7f0f94161220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.013+0000 7f0f9b22b640 1 -- 172.21.15.116:0/921971038 >> 172.21.15.116:0/921971038 conn(0x7f0f940b0cf0 msgr2=0x7f0f941499c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:59.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.013+0000 7f0f9b22b640 1 -- 172.21.15.116:0/921971038 shutdown_connections 2023-11-28T17:09:59.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.013+0000 7f0f9b22b640 1 -- 172.21.15.116:0/921971038 wait complete. 2023-11-28T17:09:59.033 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:09:59.248 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.247+0000 7ff50c046640 1 Processor -- start 2023-11-28T17:09:59.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.247+0000 7ff50c046640 1 -- start start 2023-11-28T17:09:59.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.248+0000 7ff50c046640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff504144430 0x7ff50415f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:59.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.248+0000 7ff50c046640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff50406c250 0x7ff50406c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:59.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.248+0000 7ff50c046640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff504070bd0 con 0x7ff5040703f0 2023-11-28T17:09:59.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.248+0000 7ff50c046640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff504144810 con 0x7ff50406c250 2023-11-28T17:09:59.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.248+0000 7ff50c046640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50406cc20 con 0x7ff504144430 2023-11-28T17:09:59.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.249+0000 7ff50a5e3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff50406c250 0x7ff50406c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:59.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.249+0000 7ff50a5e3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff50406c250 0x7ff50406c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59994/0 (socket says 172.21.15.116:59994) 2023-11-28T17:09:59.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.249+0000 7ff50a5e3640 1 -- 172.21.15.116:0/892905325 learned_addr learned my addr 172.21.15.116:0/892905325 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:59.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.249+0000 7ff509de2640 1 --2- 172.21.15.116:0/892905325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff504144430 0x7ff50415f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:59.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.249+0000 7ff50a5e3640 1 -- 172.21.15.116:0/892905325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff504144430 msgr2=0x7ff50415f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:59.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.249+0000 7ff50a5e3640 1 --2- 172.21.15.116:0/892905325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff504144430 0x7ff50415f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.249+0000 7ff50a5e3640 1 -- 172.21.15.116:0/892905325 >> v1:172.21.15.156:6789/0 conn(0x7ff5040703f0 legacy=0x7ff504071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:59.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.249+0000 7ff50a5e3640 1 -- 172.21.15.116:0/892905325 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff504070eb0 con 0x7ff50406c250 2023-11-28T17:09:59.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.250+0000 7ff50a5e3640 1 --2- 172.21.15.116:0/892905325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff50406c250 0x7ff50406c6e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009e50 tx=0x7ff4fc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9143b221a1bb02bb server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:59.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.251+0000 7ff508de0640 1 -- 172.21.15.116:0/892905325 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc02cb00 con 0x7ff50406c250 2023-11-28T17:09:59.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.251+0000 7ff508de0640 1 -- 172.21.15.116:0/892905325 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc02cc60 con 0x7ff50406c250 2023-11-28T17:09:59.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.251+0000 7ff508de0640 1 -- 172.21.15.116:0/892905325 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0343f0 con 0x7ff50406c250 2023-11-28T17:09:59.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.251+0000 7ff50c046640 1 -- 172.21.15.116:0/892905325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff50406c250 msgr2=0x7ff50406c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:59.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.251+0000 7ff50c046640 1 --2- 172.21.15.116:0/892905325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff50406c250 0x7ff50406c6e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009e50 tx=0x7ff4fc02bf60 comp rx=0 tx=0).stop 2023-11-28T17:09:59.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.251+0000 7ff50c046640 1 -- 172.21.15.116:0/892905325 shutdown_connections 2023-11-28T17:09:59.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.251+0000 7ff50c046640 1 --2- 172.21.15.116:0/892905325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff50406c250 0x7ff50406c6e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.251+0000 7ff50c046640 1 --2- 172.21.15.116:0/892905325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff504144430 0x7ff50415f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.251+0000 7ff50c046640 1 -- 172.21.15.116:0/892905325 >> 172.21.15.116:0/892905325 conn(0x7ff50406a4c0 msgr2=0x7ff50406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:59.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.251+0000 7ff50c046640 1 -- 172.21.15.116:0/892905325 shutdown_connections 2023-11-28T17:09:59.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.251+0000 7ff50c046640 1 -- 172.21.15.116:0/892905325 wait complete. 2023-11-28T17:09:59.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.253+0000 7ff50c046640 1 Processor -- start 2023-11-28T17:09:59.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.253+0000 7ff50c046640 1 -- start start 2023-11-28T17:09:59.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.254+0000 7ff50c046640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff50406c250 0x7ff5040b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:59.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.254+0000 7ff50c046640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5040703f0 0x7ff50413df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:59.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.254+0000 7ff50c046640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff504144430 0x7ff50414d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:59.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.254+0000 7ff50c046640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5040b2920 con 0x7ff5040703f0 2023-11-28T17:09:59.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.254+0000 7ff50c046640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5040b2c00 con 0x7ff50406c250 2023-11-28T17:09:59.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.254+0000 7ff50c046640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5040b2ee0 con 0x7ff504144430 2023-11-28T17:09:59.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.254+0000 7ff5095e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5040703f0 0x7ff50413df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:59.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.254+0000 7ff5095e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5040703f0 0x7ff50413df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60006/0 (socket says 172.21.15.116:60006) 2023-11-28T17:09:59.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.254+0000 7ff5095e1640 1 -- 172.21.15.116:0/590719527 learned_addr learned my addr 172.21.15.116:0/590719527 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:59.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.255+0000 7ff5095e1640 1 -- 172.21.15.116:0/590719527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff504144430 msgr2=0x7ff50414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:59.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.255+0000 7ff5095e1640 1 --2- 172.21.15.116:0/590719527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff504144430 0x7ff50414d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.255+0000 7ff5095e1640 1 -- 172.21.15.116:0/590719527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff50406c250 msgr2=0x7ff5040b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:59.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.255+0000 7ff5095e1640 1 --2- 172.21.15.116:0/590719527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff50406c250 0x7ff5040b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.255+0000 7ff5095e1640 1 -- 172.21.15.116:0/590719527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc009b30 con 0x7ff5040703f0 2023-11-28T17:09:59.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.255+0000 7ff5095e1640 1 --2- 172.21.15.116:0/590719527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5040703f0 0x7ff50413df50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff4f800c400 tx=0x7ff4f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:59.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.255+0000 7ff4f67fc640 1 -- 172.21.15.116:0/590719527 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f80052d0 con 0x7ff5040703f0 2023-11-28T17:09:59.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.255+0000 7ff4f67fc640 1 -- 172.21.15.116:0/590719527 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f8005430 con 0x7ff5040703f0 2023-11-28T17:09:59.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.255+0000 7ff50c046640 1 -- 172.21.15.116:0/590719527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5040b31c0 con 0x7ff5040703f0 2023-11-28T17:09:59.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.255+0000 7ff4f67fc640 1 -- 172.21.15.116:0/590719527 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8003830 con 0x7ff5040703f0 2023-11-28T17:09:59.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.256+0000 7ff50c046640 1 -- 172.21.15.116:0/590719527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5040b2630 con 0x7ff5040703f0 2023-11-28T17:09:59.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.257+0000 7ff50c046640 1 -- 172.21.15.116:0/590719527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5040b36e0 con 0x7ff5040703f0 2023-11-28T17:09:59.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.258+0000 7ff4f67fc640 1 -- 172.21.15.116:0/590719527 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff4f8003990 con 0x7ff5040703f0 2023-11-28T17:09:59.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.259+0000 7ff4f67fc640 1 --2- 172.21.15.116:0/590719527 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff4e806b2d0 0x7ff4e806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:59.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.259+0000 7ff4f67fc640 1 -- 172.21.15.116:0/590719527 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff4f8089110 con 0x7ff5040703f0 2023-11-28T17:09:59.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.259+0000 7ff509de2640 1 --2- 172.21.15.116:0/590719527 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff4e806b2d0 0x7ff4e806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:59.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.260+0000 7ff509de2640 1 --2- 172.21.15.116:0/590719527 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff4e806b2d0 0x7ff4e806d790 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff5000098f0 tx=0x7ff500008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:59.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.263+0000 7ff4f67fc640 1 -- 172.21.15.116:0/590719527 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff5040b36e0 con 0x7ff5040703f0 2023-11-28T17:09:59.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.467+0000 7ff50c046640 1 -- 172.21.15.116:0/590719527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff5040b36e0 con 0x7ff5040703f0 2023-11-28T17:09:59.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.467+0000 7ff4f67fc640 1 -- 172.21.15.116:0/590719527 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff5040b36e0 con 0x7ff5040703f0 2023-11-28T17:09:59.469 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-28T17:09:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.470+0000 7ff50c046640 1 -- 172.21.15.116:0/590719527 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff4e806b2d0 msgr2=0x7ff4e806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.470+0000 7ff50c046640 1 --2- 172.21.15.116:0/590719527 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff4e806b2d0 0x7ff4e806d790 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff5000098f0 tx=0x7ff500008040 comp rx=0 tx=0).stop 2023-11-28T17:09:59.471 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.470+0000 7ff50c046640 1 -- 172.21.15.116:0/590719527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5040703f0 msgr2=0x7ff50413df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.470+0000 7ff50c046640 1 --2- 172.21.15.116:0/590719527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5040703f0 0x7ff50413df50 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff4f800c400 tx=0x7ff4f800c8c0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.470+0000 7ff50c046640 1 -- 172.21.15.116:0/590719527 shutdown_connections 2023-11-28T17:09:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.470+0000 7ff50c046640 1 --2- 172.21.15.116:0/590719527 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff4e806b2d0 0x7ff4e806d790 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.470+0000 7ff50c046640 1 --2- 172.21.15.116:0/590719527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff504144430 0x7ff50414d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.470+0000 7ff50c046640 1 --2- 172.21.15.116:0/590719527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5040703f0 0x7ff50413df50 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.470+0000 7ff50c046640 1 --2- 172.21.15.116:0/590719527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff50406c250 0x7ff5040b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.470+0000 7ff50c046640 1 -- 172.21.15.116:0/590719527 >> 172.21.15.116:0/590719527 conn(0x7ff50406a4c0 msgr2=0x7ff50415b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:59.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.470+0000 7ff50c046640 1 -- 172.21.15.116:0/590719527 shutdown_connections 2023-11-28T17:09:59.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.470+0000 7ff50c046640 1 -- 172.21.15.116:0/590719527 wait complete. 2023-11-28T17:09:59.491 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:09:59.491 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:09:59.550 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:09:59.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.769+0000 7fecad81e640 1 Processor -- start 2023-11-28T17:09:59.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.769+0000 7fecad81e640 1 -- start start 2023-11-28T17:09:59.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.771+0000 7fecad81e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feca806bbe0 0x7feca8164bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.771+0000 7fecad81e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca815e970 con 0x7feca813f050 2023-11-28T17:09:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.771+0000 7fecad81e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca8140490 con 0x7feca80cdfe0 2023-11-28T17:09:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.771+0000 7fecad81e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca813f880 con 0x7feca806bbe0 2023-11-28T17:09:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.771+0000 7feca6ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7feca813f050 0x7feca815e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57362/0 (socket says 172.21.15.116:57362) 2023-11-28T17:09:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.771+0000 7feca6ffd640 1 -- 172.21.15.116:0/3399037345 learned_addr learned my addr 172.21.15.116:0/3399037345 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:09:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.772+0000 7feca77fe640 1 --2- 172.21.15.116:0/3399037345 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feca806bbe0 0x7feca8164bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.772+0000 7feca67fc640 1 -- 172.21.15.116:0/3399037345 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755011273 0 0) 0x7fec98001110 con 0x7feca813f050 2023-11-28T17:09:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.772+0000 7feca67fc640 1 -- 172.21.15.116:0/3399037345 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec94003520 con 0x7feca813f050 2023-11-28T17:09:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.772+0000 7feca67fc640 1 -- 172.21.15.116:0/3399037345 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3033829052 0 0) 0x7fec94003520 con 0x7feca813f050 2023-11-28T17:09:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.772+0000 7feca67fc640 1 -- 172.21.15.116:0/3399037345 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feca806bbe0 msgr2=0x7feca8164bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.772+0000 7feca67fc640 1 --2- 172.21.15.116:0/3399037345 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feca806bbe0 0x7feca8164bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.772+0000 7feca67fc640 1 -- 172.21.15.116:0/3399037345 >> v1:172.21.15.156:6789/0 conn(0x7feca80cdfe0 legacy=0x7feca806b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.772+0000 7feca67fc640 1 -- 172.21.15.116:0/3399037345 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca813fb60 con 0x7feca813f050 2023-11-28T17:09:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.772+0000 7feca67fc640 1 -- 172.21.15.116:0/3399037345 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fec98001800 con 0x7feca813f050 2023-11-28T17:09:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.772+0000 7feca77fe640 1 --2- 172.21.15.116:0/3399037345 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feca806bbe0 0x7feca8164bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:09:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7feca67fc640 1 -- 172.21.15.116:0/3399037345 >> v1:172.21.15.116:6789/0 conn(0x7feca813f050 legacy=0x7feca815e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7feca67fc640 1 --2- 172.21.15.116:0/3399037345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec940039a0 0x7fec94003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7feca67fc640 1 --2- 172.21.15.116:0/3399037345 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feca806bbe0 0x7fec940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7feca67fc640 1 --2- 172.21.15.116:0/3399037345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec94004d00 0x7fec94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7feca67fc640 1 -- 172.21.15.116:0/3399037345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca813f880 con 0x7fec94004d00 2023-11-28T17:09:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7feca67fc640 1 -- 172.21.15.116:0/3399037345 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9400dcd0 con 0x7fec940039a0 2023-11-28T17:09:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7feca67fc640 1 -- 172.21.15.116:0/3399037345 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9400de90 con 0x7feca806bbe0 2023-11-28T17:09:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7feca77fe640 1 --2- 172.21.15.116:0/3399037345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec94004d00 0x7fec94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7fec9ffff640 1 --2- 172.21.15.116:0/3399037345 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feca806bbe0 0x7fec940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:59.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7feca6ffd640 1 --2- 172.21.15.116:0/3399037345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec940039a0 0x7fec94003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:59.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7feca77fe640 1 -- 172.21.15.116:0/3399037345 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feca806bbe0 msgr2=0x7fec940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:59.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7feca77fe640 1 --2- 172.21.15.116:0/3399037345 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feca806bbe0 0x7fec940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7feca77fe640 1 -- 172.21.15.116:0/3399037345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec940039a0 msgr2=0x7fec94003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:09:59.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7feca77fe640 1 --2- 172.21.15.116:0/3399037345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec940039a0 0x7fec94003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.773+0000 7feca77fe640 1 -- 172.21.15.116:0/3399037345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec94003520 con 0x7fec94004d00 2023-11-28T17:09:59.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.774+0000 7feca77fe640 1 --2- 172.21.15.116:0/3399037345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec94004d00 0x7fec94005180 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7feca00073b0 tx=0x7fec940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46606edcb09c3006 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:59.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.775+0000 7feca67fc640 1 -- 172.21.15.116:0/3399037345 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca002cd30 con 0x7fec94004d00 2023-11-28T17:09:59.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.775+0000 7feca67fc640 1 -- 172.21.15.116:0/3399037345 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca002ce90 con 0x7fec94004d00 2023-11-28T17:09:59.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.775+0000 7feca67fc640 1 -- 172.21.15.116:0/3399037345 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca0034640 con 0x7fec94004d00 2023-11-28T17:09:59.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.775+0000 7fecad81e640 1 -- 172.21.15.116:0/3399037345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec94004d00 msgr2=0x7fec94005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:59.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.775+0000 7fecad81e640 1 --2- 172.21.15.116:0/3399037345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec94004d00 0x7fec94005180 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7feca00073b0 tx=0x7fec940057f0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.775+0000 7feca77fe640 1 -- 172.21.15.116:0/3399037345 reap_dead start 2023-11-28T17:09:59.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.775+0000 7fecad81e640 1 -- 172.21.15.116:0/3399037345 shutdown_connections 2023-11-28T17:09:59.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.775+0000 7fecad81e640 1 -- 172.21.15.116:0/3399037345 >> 172.21.15.116:0/3399037345 conn(0x7feca80b1ab0 msgr2=0x7feca8070bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:59.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.775+0000 7fecad81e640 1 -- 172.21.15.116:0/3399037345 shutdown_connections 2023-11-28T17:09:59.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.775+0000 7fecad81e640 1 -- 172.21.15.116:0/3399037345 wait complete. 2023-11-28T17:09:59.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.777+0000 7fecad81e640 1 Processor -- start 2023-11-28T17:09:59.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.777+0000 7fecad81e640 1 -- start start 2023-11-28T17:09:59.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.778+0000 7fecad81e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feca8156780 0x7feca80b46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.778+0000 7fecad81e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feca80aefd0 0x7feca8156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.778+0000 7fecad81e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feca81570a0 0x7feca80b3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.778+0000 7fecad81e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca80b4c20 con 0x7feca81570a0 2023-11-28T17:09:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.778+0000 7fecad81e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca813e500 con 0x7feca80aefd0 2023-11-28T17:09:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.778+0000 7fecad81e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca80b2e90 con 0x7feca8156780 2023-11-28T17:09:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.779+0000 7feca77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feca81570a0 0x7feca80b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.779+0000 7feca77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feca81570a0 0x7feca80b3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60024/0 (socket says 172.21.15.116:60024) 2023-11-28T17:09:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.779+0000 7feca77fe640 1 -- 172.21.15.116:0/54493237 learned_addr learned my addr 172.21.15.116:0/54493237 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:09:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.779+0000 7feca6ffd640 1 --2- 172.21.15.116:0/54493237 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feca8156780 0x7feca80b46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.779+0000 7fec9ffff640 1 --2- 172.21.15.116:0/54493237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feca80aefd0 0x7feca8156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.779+0000 7feca77fe640 1 -- 172.21.15.116:0/54493237 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feca8156780 msgr2=0x7feca80b46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.779+0000 7feca77fe640 1 --2- 172.21.15.116:0/54493237 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feca8156780 0x7feca80b46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.779+0000 7feca77fe640 1 -- 172.21.15.116:0/54493237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feca80aefd0 msgr2=0x7feca8156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.779+0000 7feca77fe640 1 --2- 172.21.15.116:0/54493237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feca80aefd0 0x7feca8156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.779+0000 7feca77fe640 1 -- 172.21.15.116:0/54493237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca8158d00 con 0x7feca81570a0 2023-11-28T17:09:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.779+0000 7feca77fe640 1 --2- 172.21.15.116:0/54493237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feca81570a0 0x7feca80b3370 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7feca00073b0 tx=0x7feca0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.780+0000 7feca4ff9640 1 -- 172.21.15.116:0/54493237 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca0046070 con 0x7feca81570a0 2023-11-28T17:09:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.780+0000 7feca4ff9640 1 -- 172.21.15.116:0/54493237 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca0035e10 con 0x7feca81570a0 2023-11-28T17:09:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.780+0000 7feca4ff9640 1 -- 172.21.15.116:0/54493237 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca0003bb0 con 0x7feca81570a0 2023-11-28T17:09:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.780+0000 7fecad81e640 1 -- 172.21.15.116:0/54493237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca8158fe0 con 0x7feca81570a0 2023-11-28T17:09:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.780+0000 7fecad81e640 1 -- 172.21.15.116:0/54493237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca8154fa0 con 0x7feca81570a0 2023-11-28T17:09:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.781+0000 7fecad81e640 1 -- 172.21.15.116:0/54493237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca8067460 con 0x7feca81570a0 2023-11-28T17:09:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.782+0000 7feca4ff9640 1 -- 172.21.15.116:0/54493237 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7feca003b050 con 0x7feca81570a0 2023-11-28T17:09:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.783+0000 7feca4ff9640 1 --2- 172.21.15.116:0/54493237 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fec8806b2a0 0x7fec8806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:09:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.783+0000 7feca4ff9640 1 -- 172.21.15.116:0/54493237 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7feca00ab4a0 con 0x7feca81570a0 2023-11-28T17:09:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.783+0000 7feca6ffd640 1 --2- 172.21.15.116:0/54493237 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fec8806b2a0 0x7fec8806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:09:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.784+0000 7feca6ffd640 1 --2- 172.21.15.116:0/54493237 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fec8806b2a0 0x7fec8806d760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fec98001930 tx=0x7fec98009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:09:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.787+0000 7feca4ff9640 1 -- 172.21.15.116:0/54493237 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feca00762e0 con 0x7feca81570a0 2023-11-28T17:09:59.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.977+0000 7fecad81e640 1 -- 172.21.15.116:0/54493237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feca8071bd0 con 0x7feca81570a0 2023-11-28T17:09:59.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.979+0000 7feca4ff9640 1 -- 172.21.15.116:0/54493237 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16570 (secure 0 0 0) 0x7feca0079900 con 0x7feca81570a0 2023-11-28T17:09:59.981 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:09:59.981 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:43.049465+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:09:59.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.981+0000 7fecad81e640 1 -- 172.21.15.116:0/54493237 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fec8806b2a0 msgr2=0x7fec8806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:59.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.982+0000 7fecad81e640 1 --2- 172.21.15.116:0/54493237 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fec8806b2a0 0x7fec8806d760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fec98001930 tx=0x7fec98009040 comp rx=0 tx=0).stop 2023-11-28T17:09:59.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.982+0000 7fecad81e640 1 -- 172.21.15.116:0/54493237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feca81570a0 msgr2=0x7feca80b3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:09:59.983 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.982+0000 7fecad81e640 1 --2- 172.21.15.116:0/54493237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feca81570a0 0x7feca80b3370 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7feca00073b0 tx=0x7feca0004ab0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.982+0000 7fecad81e640 1 -- 172.21.15.116:0/54493237 shutdown_connections 2023-11-28T17:09:59.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.982+0000 7fecad81e640 1 --2- 172.21.15.116:0/54493237 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fec8806b2a0 0x7fec8806d760 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.982+0000 7fecad81e640 1 --2- 172.21.15.116:0/54493237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feca81570a0 0x7feca80b3370 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.982+0000 7fecad81e640 1 --2- 172.21.15.116:0/54493237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feca80aefd0 0x7feca8156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.984 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.982+0000 7fecad81e640 1 --2- 172.21.15.116:0/54493237 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feca8156780 0x7feca80b46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:09:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.982+0000 7fecad81e640 1 -- 172.21.15.116:0/54493237 >> 172.21.15.116:0/54493237 conn(0x7feca80b1ab0 msgr2=0x7feca80add40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:09:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.982+0000 7fecad81e640 1 -- 172.21.15.116:0/54493237 shutdown_connections 2023-11-28T17:09:59.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:09:59.982+0000 7fecad81e640 1 -- 172.21.15.116:0/54493237 wait complete. 2023-11-28T17:10:00.002 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:10:00.002 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:10:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.214+0000 7fb24de48640 1 Processor -- start 2023-11-28T17:10:00.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.214+0000 7fb24de48640 1 -- start start 2023-11-28T17:10:00.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.215+0000 7fb24de48640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb248145ee0 0x7fb2480ac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.215+0000 7fb24de48640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb248146540 0x7fb248160e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.215+0000 7fb24de48640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2480b5110 con 0x7fb248146a90 2023-11-28T17:10:00.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.215+0000 7fb24de48640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb248161dc0 con 0x7fb248146540 2023-11-28T17:10:00.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.215+0000 7fb24de48640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2480b2190 con 0x7fb248145ee0 2023-11-28T17:10:00.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.216+0000 7fb247fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb248146a90 0x7fb248071030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57370/0 (socket says 172.21.15.116:57370) 2023-11-28T17:10:00.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.216+0000 7fb247fff640 1 -- 172.21.15.116:0/1949351378 learned_addr learned my addr 172.21.15.116:0/1949351378 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:00.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.216+0000 7fb2477fe640 1 --2- 172.21.15.116:0/1949351378 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb248145ee0 0x7fb2480ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:00.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.216+0000 7fb246ffd640 1 --2- 172.21.15.116:0/1949351378 >> v2:172.21.15.156:3300/0 conn(0x7fb248146540 0x7fb248160e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:00.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.216+0000 7fb2467fc640 1 -- 172.21.15.116:0/1949351378 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314675806 0 0) 0x7fb2480b5110 con 0x7fb248146a90 2023-11-28T17:10:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.216+0000 7fb2467fc640 1 -- 172.21.15.116:0/1949351378 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb230003520 con 0x7fb248146a90 2023-11-28T17:10:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2467fc640 1 -- 172.21.15.116:0/1949351378 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2372885424 0 0) 0x7fb230003520 con 0x7fb248146a90 2023-11-28T17:10:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2467fc640 1 -- 172.21.15.116:0/1949351378 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb248145ee0 msgr2=0x7fb2480ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2467fc640 1 --2- 172.21.15.116:0/1949351378 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb248145ee0 0x7fb2480ac870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2467fc640 1 -- 172.21.15.116:0/1949351378 >> v2:172.21.15.156:3300/0 conn(0x7fb248146540 msgr2=0x7fb248160e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2467fc640 1 --2- 172.21.15.116:0/1949351378 >> v2:172.21.15.156:3300/0 conn(0x7fb248146540 0x7fb248160e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2467fc640 1 -- 172.21.15.116:0/1949351378 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb248161ae0 con 0x7fb248146a90 2023-11-28T17:10:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2467fc640 1 -- 172.21.15.116:0/1949351378 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fb2400019f0 con 0x7fb248146a90 2023-11-28T17:10:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2467fc640 1 -- 172.21.15.116:0/1949351378 >> v1:172.21.15.116:6789/0 conn(0x7fb248146a90 legacy=0x7fb248071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2467fc640 1 --2- 172.21.15.116:0/1949351378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230003970 0x7fb230003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2467fc640 1 --2- 172.21.15.116:0/1949351378 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb248145ee0 0x7fb2300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2467fc640 1 --2- 172.21.15.116:0/1949351378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb230004d00 0x7fb230005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2467fc640 1 -- 172.21.15.116:0/1949351378 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb248161dc0 con 0x7fb230003970 2023-11-28T17:10:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2467fc640 1 -- 172.21.15.116:0/1949351378 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2480b2190 con 0x7fb230004d00 2023-11-28T17:10:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2467fc640 1 -- 172.21.15.116:0/1949351378 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23000ba00 con 0x7fb248145ee0 2023-11-28T17:10:00.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2477fe640 1 --2- 172.21.15.116:0/1949351378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230003970 0x7fb230003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2477fe640 1 -- 172.21.15.116:0/1949351378 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb248145ee0 msgr2=0x7fb2300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2477fe640 1 --2- 172.21.15.116:0/1949351378 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb248145ee0 0x7fb2300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2477fe640 1 -- 172.21.15.116:0/1949351378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb230004d00 msgr2=0x7fb230005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.217+0000 7fb2477fe640 1 --2- 172.21.15.116:0/1949351378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb230004d00 0x7fb230005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.218+0000 7fb2477fe640 1 -- 172.21.15.116:0/1949351378 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb230003520 con 0x7fb230003970 2023-11-28T17:10:00.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.219+0000 7fb2477fe640 1 --2- 172.21.15.116:0/1949351378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230003970 0x7fb230003d90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb238009ac0 tx=0x7fb2300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b4663a3e7582439 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.219+0000 7fb2467fc640 1 -- 172.21.15.116:0/1949351378 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23802c710 con 0x7fb230003970 2023-11-28T17:10:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.219+0000 7fb2467fc640 1 -- 172.21.15.116:0/1949351378 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23802c870 con 0x7fb230003970 2023-11-28T17:10:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.219+0000 7fb2467fc640 1 -- 172.21.15.116:0/1949351378 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23802cc00 con 0x7fb230003970 2023-11-28T17:10:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.219+0000 7fb24de48640 1 -- 172.21.15.116:0/1949351378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230003970 msgr2=0x7fb230003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:00.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.219+0000 7fb24de48640 1 --2- 172.21.15.116:0/1949351378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230003970 0x7fb230003d90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb238009ac0 tx=0x7fb2300057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.219+0000 7fb247fff640 1 -- 172.21.15.116:0/1949351378 reap_dead start 2023-11-28T17:10:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.219+0000 7fb24de48640 1 -- 172.21.15.116:0/1949351378 shutdown_connections 2023-11-28T17:10:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.219+0000 7fb24de48640 1 -- 172.21.15.116:0/1949351378 >> 172.21.15.116:0/1949351378 conn(0x7fb2480b0cf0 msgr2=0x7fb2480b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.220+0000 7fb24de48640 1 -- 172.21.15.116:0/1949351378 shutdown_connections 2023-11-28T17:10:00.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.220+0000 7fb24de48640 1 -- 172.21.15.116:0/1949351378 wait complete. 2023-11-28T17:10:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.221+0000 7fb24de48640 1 Processor -- start 2023-11-28T17:10:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.222+0000 7fb24de48640 1 -- start start 2023-11-28T17:10:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.222+0000 7fb24de48640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230003970 0x7fb24815caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.222+0000 7fb24de48640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb24813af60 0x7fb24815feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.223+0000 7fb24de48640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2481420e0 0x7fb24813bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.223+0000 7fb24de48640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24815d850 con 0x7fb230003970 2023-11-28T17:10:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.223+0000 7fb24de48640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24815db30 con 0x7fb24813af60 2023-11-28T17:10:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.223+0000 7fb24de48640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb24815de10 con 0x7fb2481420e0 2023-11-28T17:10:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.223+0000 7fb2477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230003970 0x7fb24815caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.223+0000 7fb2477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230003970 0x7fb24815caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60040/0 (socket says 172.21.15.116:60040) 2023-11-28T17:10:00.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.223+0000 7fb2477fe640 1 -- 172.21.15.116:0/3197536118 learned_addr learned my addr 172.21.15.116:0/3197536118 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.223+0000 7fb2477fe640 1 -- 172.21.15.116:0/3197536118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2481420e0 msgr2=0x7fb24813bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.223+0000 7fb2477fe640 1 --2- 172.21.15.116:0/3197536118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2481420e0 0x7fb24813bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.223+0000 7fb2477fe640 1 -- 172.21.15.116:0/3197536118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb24813af60 msgr2=0x7fb24815feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.223+0000 7fb2477fe640 1 --2- 172.21.15.116:0/3197536118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb24813af60 0x7fb24815feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.223+0000 7fb2477fe640 1 -- 172.21.15.116:0/3197536118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24815e0f0 con 0x7fb230003970 2023-11-28T17:10:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.223+0000 7fb2477fe640 1 --2- 172.21.15.116:0/3197536118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230003970 0x7fb24815caa0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb23802c2b0 tx=0x7fb238004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.224+0000 7fb22ffff640 1 -- 172.21.15.116:0/3197536118 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb238003810 con 0x7fb230003970 2023-11-28T17:10:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.224+0000 7fb22ffff640 1 -- 172.21.15.116:0/3197536118 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23802de90 con 0x7fb230003970 2023-11-28T17:10:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.224+0000 7fb22ffff640 1 -- 172.21.15.116:0/3197536118 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb238003b90 con 0x7fb230003970 2023-11-28T17:10:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.224+0000 7fb24de48640 1 -- 172.21.15.116:0/3197536118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24815e3d0 con 0x7fb230003970 2023-11-28T17:10:00.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.224+0000 7fb24de48640 1 -- 172.21.15.116:0/3197536118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24815cfe0 con 0x7fb230003970 2023-11-28T17:10:00.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.225+0000 7fb24de48640 1 -- 172.21.15.116:0/3197536118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24815e560 con 0x7fb230003970 2023-11-28T17:10:00.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.226+0000 7fb22ffff640 1 -- 172.21.15.116:0/3197536118 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb238003cf0 con 0x7fb230003970 2023-11-28T17:10:00.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.227+0000 7fb22ffff640 1 --2- 172.21.15.116:0/3197536118 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb23406b2a0 0x7fb23406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.227+0000 7fb22ffff640 1 -- 172.21.15.116:0/3197536118 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb2380abd20 con 0x7fb230003970 2023-11-28T17:10:00.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.227+0000 7fb246ffd640 1 --2- 172.21.15.116:0/3197536118 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb23406b2a0 0x7fb23406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:00.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.228+0000 7fb246ffd640 1 --2- 172.21.15.116:0/3197536118 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb23406b2a0 0x7fb23406d760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb23c006e80 tx=0x7fb23c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:00.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.231+0000 7fb22ffff640 1 -- 172.21.15.116:0/3197536118 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb238076d40 con 0x7fb230003970 2023-11-28T17:10:00.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.422+0000 7fb24de48640 1 -- 172.21.15.116:0/3197536118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb248069360 con 0x7fb230003970 2023-11-28T17:10:00.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.423+0000 7fb22ffff640 1 -- 172.21.15.116:0/3197536118 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb23807a360 con 0x7fb230003970 2023-11-28T17:10:00.424 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:10:00.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.425+0000 7fb24de48640 1 -- 172.21.15.116:0/3197536118 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb23406b2a0 msgr2=0x7fb23406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:00.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.425+0000 7fb24de48640 1 --2- 172.21.15.116:0/3197536118 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb23406b2a0 0x7fb23406d760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb23c006e80 tx=0x7fb23c008040 comp rx=0 tx=0).stop 2023-11-28T17:10:00.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.425+0000 7fb24de48640 1 -- 172.21.15.116:0/3197536118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230003970 msgr2=0x7fb24815caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:00.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.425+0000 7fb24de48640 1 --2- 172.21.15.116:0/3197536118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230003970 0x7fb24815caa0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb23802c2b0 tx=0x7fb238004730 comp rx=0 tx=0).stop 2023-11-28T17:10:00.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.425+0000 7fb24de48640 1 -- 172.21.15.116:0/3197536118 shutdown_connections 2023-11-28T17:10:00.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.425+0000 7fb24de48640 1 --2- 172.21.15.116:0/3197536118 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb23406b2a0 0x7fb23406d760 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.425+0000 7fb24de48640 1 --2- 172.21.15.116:0/3197536118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2481420e0 0x7fb24813bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.425+0000 7fb24de48640 1 --2- 172.21.15.116:0/3197536118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb24813af60 0x7fb24815feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.425+0000 7fb24de48640 1 --2- 172.21.15.116:0/3197536118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb230003970 0x7fb24815caa0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.425+0000 7fb24de48640 1 -- 172.21.15.116:0/3197536118 >> 172.21.15.116:0/3197536118 conn(0x7fb2480b0cf0 msgr2=0x7fb2480ce210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:00.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.425+0000 7fb24de48640 1 -- 172.21.15.116:0/3197536118 shutdown_connections 2023-11-28T17:10:00.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.425+0000 7fb24de48640 1 -- 172.21.15.116:0/3197536118 wait complete. 2023-11-28T17:10:00.448 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:10:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.665+0000 7f5d7cebb640 1 Processor -- start 2023-11-28T17:10:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.666+0000 7f5d7cebb640 1 -- start start 2023-11-28T17:10:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.667+0000 7f5d7cebb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d7806d910 0x7f5d780ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.667+0000 7f5d7cebb640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5d780b2f90 0x7f5d780709f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.667+0000 7f5d7cebb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7815a650 con 0x7f5d7815a270 2023-11-28T17:10:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.667+0000 7f5d7cebb640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7806de10 con 0x7f5d780b2f90 2023-11-28T17:10:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.667+0000 7f5d7cebb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d780b34e0 con 0x7f5d7806d910 2023-11-28T17:10:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.668+0000 7f5d777fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5d7815a270 0x7f5d780b2880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57372/0 (socket says 172.21.15.116:57372) 2023-11-28T17:10:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.668+0000 7f5d777fe640 1 -- 172.21.15.116:0/4215638174 learned_addr learned my addr 172.21.15.116:0/4215638174 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.668+0000 7f5d77fff640 1 --2- 172.21.15.116:0/4215638174 >> v2:172.21.15.156:3300/0 conn(0x7f5d780b2f90 0x7f5d780709f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.668+0000 7f5d6ffff640 1 --2- 172.21.15.116:0/4215638174 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d7806d910 0x7f5d780ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.668+0000 7f5d76ffd640 1 -- 172.21.15.116:0/4215638174 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447610696 0 0) 0x7f5d7815a650 con 0x7f5d7815a270 2023-11-28T17:10:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.668+0000 7f5d76ffd640 1 -- 172.21.15.116:0/4215638174 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d64003520 con 0x7f5d7815a270 2023-11-28T17:10:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d76ffd640 1 -- 172.21.15.116:0/4215638174 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1433375505 0 0) 0x7f5d64003520 con 0x7f5d7815a270 2023-11-28T17:10:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d76ffd640 1 -- 172.21.15.116:0/4215638174 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d7806d910 msgr2=0x7f5d780ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d76ffd640 1 --2- 172.21.15.116:0/4215638174 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d7806d910 0x7f5d780ae2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d76ffd640 1 -- 172.21.15.116:0/4215638174 >> v2:172.21.15.156:3300/0 conn(0x7f5d780b2f90 msgr2=0x7f5d780709f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d76ffd640 1 --2- 172.21.15.116:0/4215638174 >> v2:172.21.15.156:3300/0 conn(0x7f5d780b2f90 0x7f5d780709f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d76ffd640 1 -- 172.21.15.116:0/4215638174 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d780b37c0 con 0x7f5d7815a270 2023-11-28T17:10:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d76ffd640 1 -- 172.21.15.116:0/4215638174 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f5d68001670 con 0x7f5d7815a270 2023-11-28T17:10:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d76ffd640 1 -- 172.21.15.116:0/4215638174 >> v1:172.21.15.116:6789/0 conn(0x7f5d7815a270 legacy=0x7f5d780b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d76ffd640 1 --2- 172.21.15.116:0/4215638174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d64003970 0x7f5d64003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d76ffd640 1 --2- 172.21.15.116:0/4215638174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d640047c0 0x7f5d64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d76ffd640 1 --2- 172.21.15.116:0/4215638174 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d7806d910 0x7f5d64005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d76ffd640 1 -- 172.21.15.116:0/4215638174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7806de10 con 0x7f5d64003970 2023-11-28T17:10:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d76ffd640 1 -- 172.21.15.116:0/4215638174 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d780b34e0 con 0x7f5d640047c0 2023-11-28T17:10:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d76ffd640 1 -- 172.21.15.116:0/4215638174 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6400b980 con 0x7f5d7806d910 2023-11-28T17:10:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d777fe640 1 --2- 172.21.15.116:0/4215638174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d64003970 0x7f5d64003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d777fe640 1 -- 172.21.15.116:0/4215638174 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d7806d910 msgr2=0x7f5d64005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d777fe640 1 --2- 172.21.15.116:0/4215638174 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d7806d910 0x7f5d64005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d777fe640 1 -- 172.21.15.116:0/4215638174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d640047c0 msgr2=0x7f5d64004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d6ffff640 1 --2- 172.21.15.116:0/4215638174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d640047c0 0x7f5d64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d777fe640 1 --2- 172.21.15.116:0/4215638174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d640047c0 0x7f5d64004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d777fe640 1 -- 172.21.15.116:0/4215638174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d64003520 con 0x7f5d64003970 2023-11-28T17:10:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.669+0000 7f5d6ffff640 1 --2- 172.21.15.116:0/4215638174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d640047c0 0x7f5d64004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:00.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.671+0000 7f5d777fe640 1 --2- 172.21.15.116:0/4215638174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d64003970 0x7f5d64003d90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800c110 tx=0x7f5d640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3290f7f8cd02271 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:00.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.671+0000 7f5d76ffd640 1 -- 172.21.15.116:0/4215638174 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d68005ad0 con 0x7f5d64003970 2023-11-28T17:10:00.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.671+0000 7f5d76ffd640 1 -- 172.21.15.116:0/4215638174 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d68005c50 con 0x7f5d64003970 2023-11-28T17:10:00.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.671+0000 7f5d76ffd640 1 -- 172.21.15.116:0/4215638174 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6802c810 con 0x7f5d64003970 2023-11-28T17:10:00.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.671+0000 7f5d7cebb640 1 -- 172.21.15.116:0/4215638174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d64003970 msgr2=0x7f5d64003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:00.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.671+0000 7f5d7cebb640 1 --2- 172.21.15.116:0/4215638174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d64003970 0x7f5d64003d90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800c110 tx=0x7f5d640057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.671+0000 7f5d77fff640 1 -- 172.21.15.116:0/4215638174 reap_dead start 2023-11-28T17:10:00.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.671+0000 7f5d7cebb640 1 -- 172.21.15.116:0/4215638174 shutdown_connections 2023-11-28T17:10:00.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.671+0000 7f5d7cebb640 1 -- 172.21.15.116:0/4215638174 >> 172.21.15.116:0/4215638174 conn(0x7f5d780b0950 msgr2=0x7f5d780b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:00.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.672+0000 7f5d7cebb640 1 -- 172.21.15.116:0/4215638174 shutdown_connections 2023-11-28T17:10:00.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.672+0000 7f5d7cebb640 1 -- 172.21.15.116:0/4215638174 wait complete. 2023-11-28T17:10:00.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.673+0000 7f5d7cebb640 1 Processor -- start 2023-11-28T17:10:00.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.674+0000 7f5d7cebb640 1 -- start start 2023-11-28T17:10:00.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.674+0000 7f5d7cebb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d64003970 0x7f5d7815cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.674+0000 7f5d7cebb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d7813e820 0x7f5d78152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.674+0000 7f5d7cebb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d7814f080 0x7f5d7814f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.674+0000 7f5d7cebb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7815bbc0 con 0x7f5d64003970 2023-11-28T17:10:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.674+0000 7f5d7cebb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7815bea0 con 0x7f5d7814f080 2023-11-28T17:10:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.674+0000 7f5d7cebb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7815c180 con 0x7f5d7813e820 2023-11-28T17:10:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.675+0000 7f5d777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d64003970 0x7f5d7815cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.675+0000 7f5d777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d64003970 0x7f5d7815cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60058/0 (socket says 172.21.15.116:60058) 2023-11-28T17:10:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.675+0000 7f5d777fe640 1 -- 172.21.15.116:0/3555562232 learned_addr learned my addr 172.21.15.116:0/3555562232 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.675+0000 7f5d777fe640 1 -- 172.21.15.116:0/3555562232 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d7813e820 msgr2=0x7f5d78152a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:00.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.675+0000 7f5d777fe640 1 --2- 172.21.15.116:0/3555562232 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d7813e820 0x7f5d78152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.675+0000 7f5d777fe640 1 -- 172.21.15.116:0/3555562232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d7814f080 msgr2=0x7f5d7814f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.675+0000 7f5d777fe640 1 --2- 172.21.15.116:0/3555562232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d7814f080 0x7f5d7814f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.675+0000 7f5d777fe640 1 -- 172.21.15.116:0/3555562232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7815c460 con 0x7f5d64003970 2023-11-28T17:10:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.675+0000 7f5d777fe640 1 --2- 172.21.15.116:0/3555562232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d64003970 0x7f5d7815cc60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5d6802c410 tx=0x7f5d68001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.676+0000 7f5d74ff9640 1 -- 172.21.15.116:0/3555562232 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d68039070 con 0x7f5d64003970 2023-11-28T17:10:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.676+0000 7f5d74ff9640 1 -- 172.21.15.116:0/3555562232 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6802cc20 con 0x7f5d64003970 2023-11-28T17:10:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.676+0000 7f5d74ff9640 1 -- 172.21.15.116:0/3555562232 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6803c040 con 0x7f5d64003970 2023-11-28T17:10:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.676+0000 7f5d7cebb640 1 -- 172.21.15.116:0/3555562232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7815c740 con 0x7f5d64003970 2023-11-28T17:10:00.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.676+0000 7f5d7cebb640 1 -- 172.21.15.116:0/3555562232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7814fa40 con 0x7f5d64003970 2023-11-28T17:10:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.677+0000 7f5d7cebb640 1 -- 172.21.15.116:0/3555562232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7815c900 con 0x7f5d64003970 2023-11-28T17:10:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.678+0000 7f5d74ff9640 1 -- 172.21.15.116:0/3555562232 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f5d68032070 con 0x7f5d64003970 2023-11-28T17:10:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.679+0000 7f5d74ff9640 1 --2- 172.21.15.116:0/3555562232 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5d5806b2a0 0x7f5d5806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.679+0000 7f5d74ff9640 1 -- 172.21.15.116:0/3555562232 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5d680aaed0 con 0x7f5d64003970 2023-11-28T17:10:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.679+0000 7f5d6ffff640 1 --2- 172.21.15.116:0/3555562232 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5d5806b2a0 0x7f5d5806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:00.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.680+0000 7f5d6ffff640 1 --2- 172.21.15.116:0/3555562232 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5d5806b2a0 0x7f5d5806d760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5d780ae1b0 tx=0x7f5d60008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:00.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.684+0000 7f5d74ff9640 1 -- 172.21.15.116:0/3555562232 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d68075d20 con 0x7f5d64003970 2023-11-28T17:10:00.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.859+0000 7f5d7cebb640 1 -- 172.21.15.116:0/3555562232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5d780708f0 con 0x7f5d64003970 2023-11-28T17:10:00.861 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.860+0000 7f5d74ff9640 1 -- 172.21.15.116:0/3555562232 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5d68007070 con 0x7f5d64003970 2023-11-28T17:10:00.862 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:10:00.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.863+0000 7f5d7cebb640 1 -- 172.21.15.116:0/3555562232 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5d5806b2a0 msgr2=0x7f5d5806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.863+0000 7f5d7cebb640 1 --2- 172.21.15.116:0/3555562232 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5d5806b2a0 0x7f5d5806d760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5d780ae1b0 tx=0x7f5d60008040 comp rx=0 tx=0).stop 2023-11-28T17:10:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.863+0000 7f5d7cebb640 1 -- 172.21.15.116:0/3555562232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d64003970 msgr2=0x7f5d7815cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.863+0000 7f5d7cebb640 1 --2- 172.21.15.116:0/3555562232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d64003970 0x7f5d7815cc60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5d6802c410 tx=0x7f5d68001930 comp rx=0 tx=0).stop 2023-11-28T17:10:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.863+0000 7f5d7cebb640 1 -- 172.21.15.116:0/3555562232 shutdown_connections 2023-11-28T17:10:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.863+0000 7f5d7cebb640 1 --2- 172.21.15.116:0/3555562232 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5d5806b2a0 0x7f5d5806d760 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.863+0000 7f5d7cebb640 1 --2- 172.21.15.116:0/3555562232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d7814f080 0x7f5d7814f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.865 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.863+0000 7f5d7cebb640 1 --2- 172.21.15.116:0/3555562232 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d7813e820 0x7f5d78152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.863+0000 7f5d7cebb640 1 --2- 172.21.15.116:0/3555562232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d64003970 0x7f5d7815cc60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.863+0000 7f5d7cebb640 1 -- 172.21.15.116:0/3555562232 >> 172.21.15.116:0/3555562232 conn(0x7f5d780b0950 msgr2=0x7f5d780ac770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.863+0000 7f5d7cebb640 1 -- 172.21.15.116:0/3555562232 shutdown_connections 2023-11-28T17:10:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:00.863+0000 7f5d7cebb640 1 -- 172.21.15.116:0/3555562232 wait complete. 2023-11-28T17:10:00.882 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:10:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.095+0000 7f3044c46640 1 Processor -- start 2023-11-28T17:10:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.095+0000 7f3044c46640 1 -- start start 2023-11-28T17:10:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.097+0000 7f3044c46640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f304015b0c0 0x7f304015ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.097+0000 7f3044c46640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30400ce940 0x7f30400cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.097+0000 7f3044c46640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3040071b40 0x7f3040071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.097+0000 7f3044c46640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3040072510 con 0x7f304015b0c0 2023-11-28T17:10:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.097+0000 7f3044c46640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30400cd320 con 0x7f3040071b40 2023-11-28T17:10:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.097+0000 7f3044c46640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30400ced80 con 0x7f30400ce940 2023-11-28T17:10:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.097+0000 7f303e59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f304015b0c0 0x7f304015ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.097+0000 7f303e59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f304015b0c0 0x7f304015ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60072/0 (socket says 172.21.15.116:60072) 2023-11-28T17:10:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.097+0000 7f303e59c640 1 -- 172.21.15.116:0/2671141019 learned_addr learned my addr 172.21.15.116:0/2671141019 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.098+0000 7f303ed9d640 1 --2- 172.21.15.116:0/2671141019 >> v2:172.21.15.156:3300/0 conn(0x7f3040071b40 0x7f3040071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:01.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.098+0000 7f3037fff640 1 --2- 172.21.15.116:0/2671141019 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30400ce940 0x7f30400cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:01.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.098+0000 7f303e59c640 1 -- 172.21.15.116:0/2671141019 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30400ce940 msgr2=0x7f30400cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:01.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.098+0000 7f303e59c640 1 --2- 172.21.15.116:0/2671141019 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30400ce940 0x7f30400cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.098+0000 7f303e59c640 1 -- 172.21.15.116:0/2671141019 >> v2:172.21.15.156:3300/0 conn(0x7f3040071b40 msgr2=0x7f3040071fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:01.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.098+0000 7f303e59c640 1 --2- 172.21.15.116:0/2671141019 >> v2:172.21.15.156:3300/0 conn(0x7f3040071b40 0x7f3040071fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.098+0000 7f303e59c640 1 -- 172.21.15.116:0/2671141019 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30400cbee0 con 0x7f304015b0c0 2023-11-28T17:10:01.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.098+0000 7f303ed9d640 1 --2- 172.21.15.116:0/2671141019 >> v2:172.21.15.156:3300/0 conn(0x7f3040071b40 0x7f3040071fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:10:01.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.099+0000 7f303e59c640 1 --2- 172.21.15.116:0/2671141019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f304015b0c0 0x7f304015ed10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f30400b4fc0 tx=0x7f303002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8699aadb8adc0b8f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:01.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.100+0000 7f303dd9b640 1 -- 172.21.15.116:0/2671141019 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30300317e0 con 0x7f304015b0c0 2023-11-28T17:10:01.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.100+0000 7f303dd9b640 1 -- 172.21.15.116:0/2671141019 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3030031940 con 0x7f304015b0c0 2023-11-28T17:10:01.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.100+0000 7f303dd9b640 1 -- 172.21.15.116:0/2671141019 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030031cd0 con 0x7f304015b0c0 2023-11-28T17:10:01.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.100+0000 7f3044c46640 1 -- 172.21.15.116:0/2671141019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f304015b0c0 msgr2=0x7f304015ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:01.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.100+0000 7f3044c46640 1 --2- 172.21.15.116:0/2671141019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f304015b0c0 0x7f304015ed10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f30400b4fc0 tx=0x7f303002b880 comp rx=0 tx=0).stop 2023-11-28T17:10:01.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.100+0000 7f3044c46640 1 -- 172.21.15.116:0/2671141019 shutdown_connections 2023-11-28T17:10:01.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.100+0000 7f3044c46640 1 --2- 172.21.15.116:0/2671141019 >> v2:172.21.15.156:3300/0 conn(0x7f3040071b40 0x7f3040071fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.100+0000 7f3044c46640 1 --2- 172.21.15.116:0/2671141019 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f30400ce940 0x7f30400cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.100+0000 7f3044c46640 1 --2- 172.21.15.116:0/2671141019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f304015b0c0 0x7f304015ed10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.100+0000 7f3044c46640 1 -- 172.21.15.116:0/2671141019 >> 172.21.15.116:0/2671141019 conn(0x7f30400b0cf0 msgr2=0x7f30400b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.100+0000 7f3044c46640 1 -- 172.21.15.116:0/2671141019 shutdown_connections 2023-11-28T17:10:01.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.100+0000 7f3044c46640 1 -- 172.21.15.116:0/2671141019 wait complete. 2023-11-28T17:10:01.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.102+0000 7f3044c46640 1 Processor -- start 2023-11-28T17:10:01.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.102+0000 7f3044c46640 1 -- start start 2023-11-28T17:10:01.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.103+0000 7f3044c46640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040071b40 0x7f30401e9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:01.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.103+0000 7f3044c46640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30400ce940 0x7f30401e3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:01.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.103+0000 7f3044c46640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f304015b0c0 0x7f30401563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:01.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.103+0000 7f3044c46640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30401e54c0 con 0x7f3040071b40 2023-11-28T17:10:01.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.103+0000 7f3044c46640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30401434d0 con 0x7f30400ce940 2023-11-28T17:10:01.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.103+0000 7f3044c46640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30401e4840 con 0x7f304015b0c0 2023-11-28T17:10:01.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.104+0000 7f303e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040071b40 0x7f30401e9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:01.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.104+0000 7f303e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040071b40 0x7f30401e9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60088/0 (socket says 172.21.15.116:60088) 2023-11-28T17:10:01.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.104+0000 7f303e59c640 1 -- 172.21.15.116:0/2196979733 learned_addr learned my addr 172.21.15.116:0/2196979733 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:01.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.104+0000 7f3037fff640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30400ce940 0x7f30401e3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:01.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.104+0000 7f303ed9d640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f304015b0c0 0x7f30401563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:01.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.104+0000 7f303e59c640 1 -- 172.21.15.116:0/2196979733 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f304015b0c0 msgr2=0x7f30401563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:01.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.104+0000 7f303e59c640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f304015b0c0 0x7f30401563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.104+0000 7f303e59c640 1 -- 172.21.15.116:0/2196979733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30400ce940 msgr2=0x7f30401e3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:01.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.104+0000 7f303e59c640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30400ce940 0x7f30401e3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.104+0000 7f303e59c640 1 -- 172.21.15.116:0/2196979733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30300094d0 con 0x7f3040071b40 2023-11-28T17:10:01.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.104+0000 7f303ed9d640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f304015b0c0 0x7f30401563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:01.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.104+0000 7f303e59c640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040071b40 0x7f30401e9170 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f303002b8c0 tx=0x7f30300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:01.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.105+0000 7f3036ffd640 1 -- 172.21.15.116:0/2196979733 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030032880 con 0x7f3040071b40 2023-11-28T17:10:01.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.105+0000 7f3036ffd640 1 -- 172.21.15.116:0/2196979733 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30300329e0 con 0x7f3040071b40 2023-11-28T17:10:01.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.105+0000 7f3036ffd640 1 -- 172.21.15.116:0/2196979733 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030032d70 con 0x7f3040071b40 2023-11-28T17:10:01.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.105+0000 7f3044c46640 1 -- 172.21.15.116:0/2196979733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30401e4b20 con 0x7f3040071b40 2023-11-28T17:10:01.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.105+0000 7f3044c46640 1 -- 172.21.15.116:0/2196979733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30401e50e0 con 0x7f3040071b40 2023-11-28T17:10:01.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.106+0000 7f3044c46640 1 -- 172.21.15.116:0/2196979733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3040069360 con 0x7f3040071b40 2023-11-28T17:10:01.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.107+0000 7f3036ffd640 1 -- 172.21.15.116:0/2196979733 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f30300453f0 con 0x7f3040071b40 2023-11-28T17:10:01.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.108+0000 7f3036ffd640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f302006b2a0 0x7f302006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:01.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.108+0000 7f3036ffd640 1 -- 172.21.15.116:0/2196979733 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f30300abf10 con 0x7f3040071b40 2023-11-28T17:10:01.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.108+0000 7f3037fff640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f302006b2a0 0x7f302006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:01.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.108+0000 7f3037fff640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f302006b2a0 0x7f302006d760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3040149e30 tx=0x7f3028008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:01.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.112+0000 7f3036ffd640 1 -- 172.21.15.116:0/2196979733 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3030076eb0 con 0x7f3040071b40 2023-11-28T17:10:01.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.304+0000 7f3044c46640 1 -- 172.21.15.116:0/2196979733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f30400b1990 con 0x7f3040071b40 2023-11-28T17:10:01.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.305+0000 7f3036ffd640 1 -- 172.21.15.116:0/2196979733 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f303007a4d0 con 0x7f3040071b40 2023-11-28T17:10:01.306 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:10:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.307+0000 7f3044c46640 1 -- 172.21.15.116:0/2196979733 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f302006b2a0 msgr2=0x7f302006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.307+0000 7f3044c46640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f302006b2a0 0x7f302006d760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3040149e30 tx=0x7f3028008040 comp rx=0 tx=0).stop 2023-11-28T17:10:01.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.308+0000 7f3044c46640 1 -- 172.21.15.116:0/2196979733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040071b40 msgr2=0x7f30401e9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.308+0000 7f3044c46640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040071b40 0x7f30401e9170 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f303002b8c0 tx=0x7f30300046c0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.308+0000 7f3044c46640 1 -- 172.21.15.116:0/2196979733 shutdown_connections 2023-11-28T17:10:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.308+0000 7f3044c46640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f302006b2a0 0x7f302006d760 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.308+0000 7f3044c46640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f304015b0c0 0x7f30401563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.308+0000 7f3044c46640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30400ce940 0x7f30401e3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.308+0000 7f3044c46640 1 --2- 172.21.15.116:0/2196979733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040071b40 0x7f30401e9170 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.308+0000 7f3044c46640 1 -- 172.21.15.116:0/2196979733 >> 172.21.15.116:0/2196979733 conn(0x7f30400b0cf0 msgr2=0x7f304015e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.308+0000 7f3044c46640 1 -- 172.21.15.116:0/2196979733 shutdown_connections 2023-11-28T17:10:01.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.308+0000 7f3044c46640 1 -- 172.21.15.116:0/2196979733 wait complete. 2023-11-28T17:10:01.329 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:10:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.542+0000 7f2f67547640 1 Processor -- start 2023-11-28T17:10:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.542+0000 7f2f67547640 1 -- start start 2023-11-28T17:10:01.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.544+0000 7f2f67547640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f600cdf90 0x7f2f600ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:01.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.544+0000 7f2f67547640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2f600ceb90 0x7f2f600cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.544+0000 7f2f67547640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f60070560 con 0x7f2f6015ae20 2023-11-28T17:10:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.544+0000 7f2f67547640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f6006d3e0 con 0x7f2f600cdf90 2023-11-28T17:10:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.544+0000 7f2f67547640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f600cf540 con 0x7f2f600ceb90 2023-11-28T17:10:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.544+0000 7f2f65d44640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f600cdf90 0x7f2f600ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.544+0000 7f2f65d44640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2f600cdf90 0x7f2f600ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60100/0 (socket says 172.21.15.116:60100) 2023-11-28T17:10:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.544+0000 7f2f65d44640 1 -- 172.21.15.116:0/2451854505 learned_addr learned my addr 172.21.15.116:0/2451854505 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.544+0000 7f2f65d44640 1 -- 172.21.15.116:0/2451854505 >> v2:172.21.15.156:3300/0 conn(0x7f2f600ceb90 msgr2=0x7f2f600cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:01.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.544+0000 7f2f65d44640 1 --2- 172.21.15.116:0/2451854505 >> v2:172.21.15.156:3300/0 conn(0x7f2f600ceb90 0x7f2f600cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.544+0000 7f2f65d44640 1 -- 172.21.15.116:0/2451854505 >> v1:172.21.15.156:6789/0 conn(0x7f2f6015ae20 legacy=0x7f2f6006ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:01.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.544+0000 7f2f65d44640 1 -- 172.21.15.116:0/2451854505 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f60070840 con 0x7f2f600cdf90 2023-11-28T17:10:01.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.545+0000 7f2f65d44640 1 --2- 172.21.15.116:0/2451854505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f600cdf90 0x7f2f600ce420 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009870 tx=0x7f2f5002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441bb2185c250f65 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:01.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.546+0000 7f2f65543640 1 -- 172.21.15.116:0/2451854505 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f500317d0 con 0x7f2f600cdf90 2023-11-28T17:10:01.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.546+0000 7f2f65543640 1 -- 172.21.15.116:0/2451854505 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f50031930 con 0x7f2f600cdf90 2023-11-28T17:10:01.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.546+0000 7f2f65543640 1 -- 172.21.15.116:0/2451854505 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50031cc0 con 0x7f2f600cdf90 2023-11-28T17:10:01.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.546+0000 7f2f67547640 1 -- 172.21.15.116:0/2451854505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f600cdf90 msgr2=0x7f2f600ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:01.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.547+0000 7f2f67547640 1 --2- 172.21.15.116:0/2451854505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f600cdf90 0x7f2f600ce420 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009870 tx=0x7f2f5002b870 comp rx=0 tx=0).stop 2023-11-28T17:10:01.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.547+0000 7f2f67547640 1 -- 172.21.15.116:0/2451854505 shutdown_connections 2023-11-28T17:10:01.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.547+0000 7f2f67547640 1 --2- 172.21.15.116:0/2451854505 >> v2:172.21.15.156:3300/0 conn(0x7f2f600ceb90 0x7f2f600cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.547+0000 7f2f67547640 1 --2- 172.21.15.116:0/2451854505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f600cdf90 0x7f2f600ce420 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.547+0000 7f2f67547640 1 -- 172.21.15.116:0/2451854505 >> 172.21.15.116:0/2451854505 conn(0x7f2f600b0950 msgr2=0x7f2f600b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:01.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.547+0000 7f2f67547640 1 -- 172.21.15.116:0/2451854505 shutdown_connections 2023-11-28T17:10:01.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.547+0000 7f2f67547640 1 -- 172.21.15.116:0/2451854505 wait complete. 2023-11-28T17:10:01.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.549+0000 7f2f67547640 1 Processor -- start 2023-11-28T17:10:01.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.549+0000 7f2f67547640 1 -- start start 2023-11-28T17:10:01.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.550+0000 7f2f67547640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f600cdf90 0x7f2f6009c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:01.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.550+0000 7f2f67547640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f600ceb90 0x7f2f6014c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:01.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.550+0000 7f2f67547640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f6015ae20 0x7f2f600b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:01.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.550+0000 7f2f67547640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6014cc30 con 0x7f2f600cdf90 2023-11-28T17:10:01.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.550+0000 7f2f67547640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f600b2470 con 0x7f2f600ceb90 2023-11-28T17:10:01.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.550+0000 7f2f67547640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f600b2750 con 0x7f2f6015ae20 2023-11-28T17:10:01.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.550+0000 7f2f66545640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f600cdf90 0x7f2f6009c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:01.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.551+0000 7f2f66545640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f600cdf90 0x7f2f6009c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60108/0 (socket says 172.21.15.116:60108) 2023-11-28T17:10:01.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.551+0000 7f2f66545640 1 -- 172.21.15.116:0/2011400728 learned_addr learned my addr 172.21.15.116:0/2011400728 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:01.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.551+0000 7f2f66d46640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f6015ae20 0x7f2f600b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:01.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.551+0000 7f2f65d44640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f600ceb90 0x7f2f6014c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:01.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.551+0000 7f2f66545640 1 -- 172.21.15.116:0/2011400728 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f6015ae20 msgr2=0x7f2f600b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:01.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.551+0000 7f2f66545640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f6015ae20 0x7f2f600b3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.551+0000 7f2f66545640 1 -- 172.21.15.116:0/2011400728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f600ceb90 msgr2=0x7f2f6014c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:01.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.551+0000 7f2f66545640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f600ceb90 0x7f2f6014c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.551+0000 7f2f66545640 1 -- 172.21.15.116:0/2011400728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f500094d0 con 0x7f2f600cdf90 2023-11-28T17:10:01.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.551+0000 7f2f65d44640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f600ceb90 0x7f2f6014c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:01.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.551+0000 7f2f66545640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f600cdf90 0x7f2f6009c3c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c00c410 tx=0x7f2f5c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:01.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.551+0000 7f2f56ffd640 1 -- 172.21.15.116:0/2011400728 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c0035f0 con 0x7f2f600cdf90 2023-11-28T17:10:01.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.551+0000 7f2f56ffd640 1 -- 172.21.15.116:0/2011400728 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c003750 con 0x7f2f600cdf90 2023-11-28T17:10:01.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.552+0000 7f2f67547640 1 -- 172.21.15.116:0/2011400728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f600b2a30 con 0x7f2f600cdf90 2023-11-28T17:10:01.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.552+0000 7f2f56ffd640 1 -- 172.21.15.116:0/2011400728 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c003b40 con 0x7f2f600cdf90 2023-11-28T17:10:01.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.552+0000 7f2f67547640 1 -- 172.21.15.116:0/2011400728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f600b2ff0 con 0x7f2f600cdf90 2023-11-28T17:10:01.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.553+0000 7f2f67547640 1 -- 172.21.15.116:0/2011400728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6006c640 con 0x7f2f600cdf90 2023-11-28T17:10:01.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.554+0000 7f2f56ffd640 1 -- 172.21.15.116:0/2011400728 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2f5c003ca0 con 0x7f2f600cdf90 2023-11-28T17:10:01.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.555+0000 7f2f56ffd640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2f4406b2a0 0x7f2f4406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:01.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.555+0000 7f2f56ffd640 1 -- 172.21.15.116:0/2011400728 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2f5c088d70 con 0x7f2f600cdf90 2023-11-28T17:10:01.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.555+0000 7f2f65d44640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2f4406b2a0 0x7f2f4406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:01.557 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.556+0000 7f2f65d44640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2f4406b2a0 0x7f2f4406d760 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2f50036000 tx=0x7f2f5003d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:01.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.559+0000 7f2f56ffd640 1 -- 172.21.15.116:0/2011400728 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f5c053c50 con 0x7f2f600cdf90 2023-11-28T17:10:01.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.762+0000 7f2f67547640 1 -- 172.21.15.116:0/2011400728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2f6009d0b0 con 0x7f2f600cdf90 2023-11-28T17:10:01.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.763+0000 7f2f56ffd640 1 -- 172.21.15.116:0/2011400728 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2f5c057270 con 0x7f2f600cdf90 2023-11-28T17:10:01.765 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-28T17:10:01.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.766+0000 7f2f67547640 1 -- 172.21.15.116:0/2011400728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2f4406b2a0 msgr2=0x7f2f4406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:01.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.766+0000 7f2f67547640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2f4406b2a0 0x7f2f4406d760 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2f50036000 tx=0x7f2f5003d040 comp rx=0 tx=0).stop 2023-11-28T17:10:01.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.766+0000 7f2f67547640 1 -- 172.21.15.116:0/2011400728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f600cdf90 msgr2=0x7f2f6009c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:01.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.766+0000 7f2f67547640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f600cdf90 0x7f2f6009c3c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c00c410 tx=0x7f2f5c00c8d0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.766+0000 7f2f67547640 1 -- 172.21.15.116:0/2011400728 shutdown_connections 2023-11-28T17:10:01.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.766+0000 7f2f67547640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2f4406b2a0 0x7f2f4406d760 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.766+0000 7f2f67547640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f6015ae20 0x7f2f600b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.766+0000 7f2f67547640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f600ceb90 0x7f2f6014c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.766+0000 7f2f67547640 1 --2- 172.21.15.116:0/2011400728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f600cdf90 0x7f2f6009c3c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:01.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.766+0000 7f2f67547640 1 -- 172.21.15.116:0/2011400728 >> 172.21.15.116:0/2011400728 conn(0x7f2f600b0950 msgr2=0x7f2f600b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:01.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.766+0000 7f2f67547640 1 -- 172.21.15.116:0/2011400728 shutdown_connections 2023-11-28T17:10:01.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:01.767+0000 7f2f67547640 1 -- 172.21.15.116:0/2011400728 wait complete. 2023-11-28T17:10:01.787 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2023-11-28T17:10:01.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:01.787 DEBUG:teuthology.orchestra.run.smithi156:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2023-11-28T17:10:01.896 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:10:01.896 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:10:01.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:01.897 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-11-28T17:10:01.944 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T17:10:01.944 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:10:02.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.161+0000 7f4900cd0640 1 Processor -- start 2023-11-28T17:10:02.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.161+0000 7f4900cd0640 1 -- start start 2023-11-28T17:10:02.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.163+0000 7f4900cd0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f48fc0b2f70 0x7f48fc1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:02.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.163+0000 7f4900cd0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f48fc161b30 0x7f48fc06a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:02.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.163+0000 7f4900cd0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48fc0b34a0 con 0x7f48fc0cf800 2023-11-28T17:10:02.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.163+0000 7f4900cd0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48fc0b5480 con 0x7f48fc0b2f70 2023-11-28T17:10:02.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.163+0000 7f4900cd0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc0b3780 con 0x7f48fc161b30 2023-11-28T17:10:02.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.163+0000 7f48f9d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f48fc0b2f70 0x7f48fc1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:02.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.164+0000 7f48f9d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f48fc0b2f70 0x7f48fc1615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60116/0 (socket says 172.21.15.116:60116) 2023-11-28T17:10:02.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.164+0000 7f48f9d9b640 1 -- 172.21.15.116:0/2792915437 learned_addr learned my addr 172.21.15.116:0/2792915437 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:02.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.164+0000 7f48fad9d640 1 --2- 172.21.15.116:0/2792915437 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f48fc161b30 0x7f48fc06a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:02.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.164+0000 7f48f9d9b640 1 -- 172.21.15.116:0/2792915437 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f48fc161b30 msgr2=0x7f48fc06a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:02.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.164+0000 7f48f9d9b640 1 --2- 172.21.15.116:0/2792915437 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f48fc161b30 0x7f48fc06a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.164+0000 7f48f9d9b640 1 -- 172.21.15.116:0/2792915437 >> v1:172.21.15.156:6789/0 conn(0x7f48fc0cf800 legacy=0x7f48fc15e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:02.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.164+0000 7f48f9d9b640 1 -- 172.21.15.116:0/2792915437 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48fc0b3a60 con 0x7f48fc0b2f70 2023-11-28T17:10:02.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.165+0000 7f48f9d9b640 1 --2- 172.21.15.116:0/2792915437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48fc0b2f70 0x7f48fc1615f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f48f0009870 tx=0x7f48f002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd10a3a12a8dc04f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:02.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.165+0000 7f48f959a640 1 -- 172.21.15.116:0/2792915437 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0031640 con 0x7f48fc0b2f70 2023-11-28T17:10:02.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.165+0000 7f48f959a640 1 -- 172.21.15.116:0/2792915437 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f00317a0 con 0x7f48fc0b2f70 2023-11-28T17:10:02.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.165+0000 7f48f959a640 1 -- 172.21.15.116:0/2792915437 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0031b30 con 0x7f48fc0b2f70 2023-11-28T17:10:02.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.166+0000 7f4900cd0640 1 -- 172.21.15.116:0/2792915437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48fc0b2f70 msgr2=0x7f48fc1615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:02.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.166+0000 7f4900cd0640 1 --2- 172.21.15.116:0/2792915437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48fc0b2f70 0x7f48fc1615f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f48f0009870 tx=0x7f48f002b8f0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.166+0000 7f4900cd0640 1 -- 172.21.15.116:0/2792915437 shutdown_connections 2023-11-28T17:10:02.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.166+0000 7f4900cd0640 1 --2- 172.21.15.116:0/2792915437 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f48fc161b30 0x7f48fc06a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.166+0000 7f4900cd0640 1 --2- 172.21.15.116:0/2792915437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48fc0b2f70 0x7f48fc1615f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.166+0000 7f4900cd0640 1 -- 172.21.15.116:0/2792915437 >> 172.21.15.116:0/2792915437 conn(0x7f48fc0cfec0 msgr2=0x7f48fc0d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:02.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.166+0000 7f4900cd0640 1 -- 172.21.15.116:0/2792915437 shutdown_connections 2023-11-28T17:10:02.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.166+0000 7f4900cd0640 1 -- 172.21.15.116:0/2792915437 wait complete. 2023-11-28T17:10:02.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.168+0000 7f4900cd0640 1 Processor -- start 2023-11-28T17:10:02.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.168+0000 7f4900cd0640 1 -- start start 2023-11-28T17:10:02.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.169+0000 7f4900cd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48fc0b2f70 0x7f48fc0ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:02.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.169+0000 7f4900cd0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f48fc0cf800 0x7f48fc0ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:02.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.169+0000 7f4900cd0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48fc161b30 0x7f48fc0acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:02.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.169+0000 7f4900cd0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc0adc40 con 0x7f48fc0b2f70 2023-11-28T17:10:02.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.169+0000 7f4900cd0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc14ebf0 con 0x7f48fc161b30 2023-11-28T17:10:02.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.169+0000 7f4900cd0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48fc0aeef0 con 0x7f48fc0cf800 2023-11-28T17:10:02.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.169+0000 7f48fa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48fc0b2f70 0x7f48fc0ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:02.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.169+0000 7f48fa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48fc0b2f70 0x7f48fc0ae9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60132/0 (socket says 172.21.15.116:60132) 2023-11-28T17:10:02.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.169+0000 7f48fa59c640 1 -- 172.21.15.116:0/2275028102 learned_addr learned my addr 172.21.15.116:0/2275028102 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:02.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.170+0000 7f48fa59c640 1 -- 172.21.15.116:0/2275028102 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f48fc0cf800 msgr2=0x7f48fc0ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:02.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.170+0000 7f48fa59c640 1 --2- 172.21.15.116:0/2275028102 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f48fc0cf800 0x7f48fc0ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.170+0000 7f48fa59c640 1 -- 172.21.15.116:0/2275028102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48fc161b30 msgr2=0x7f48fc0acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:02.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.170+0000 7f48fa59c640 1 --2- 172.21.15.116:0/2275028102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48fc161b30 0x7f48fc0acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.170+0000 7f48fa59c640 1 -- 172.21.15.116:0/2275028102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f00094d0 con 0x7f48fc0b2f70 2023-11-28T17:10:02.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.170+0000 7f48fa59c640 1 --2- 172.21.15.116:0/2275028102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48fc0b2f70 0x7f48fc0ae9b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f48ec00c430 tx=0x7f48ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:02.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.170+0000 7f48eaffd640 1 -- 172.21.15.116:0/2275028102 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec011790 con 0x7f48fc0b2f70 2023-11-28T17:10:02.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.170+0000 7f48eaffd640 1 -- 172.21.15.116:0/2275028102 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ec0118f0 con 0x7f48fc0b2f70 2023-11-28T17:10:02.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.170+0000 7f4900cd0640 1 -- 172.21.15.116:0/2275028102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48fc0af170 con 0x7f48fc0b2f70 2023-11-28T17:10:02.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.170+0000 7f48eaffd640 1 -- 172.21.15.116:0/2275028102 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec011ce0 con 0x7f48fc0b2f70 2023-11-28T17:10:02.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.170+0000 7f4900cd0640 1 -- 172.21.15.116:0/2275028102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48fc14e880 con 0x7f48fc0b2f70 2023-11-28T17:10:02.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.171+0000 7f4900cd0640 1 -- 172.21.15.116:0/2275028102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48fc14e6e0 con 0x7f48fc0b2f70 2023-11-28T17:10:02.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.172+0000 7f48eaffd640 1 -- 172.21.15.116:0/2275028102 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f48ec01c460 con 0x7f48fc0b2f70 2023-11-28T17:10:02.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.173+0000 7f48eaffd640 1 --2- 172.21.15.116:0/2275028102 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f48dc06b2a0 0x7f48dc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:02.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.173+0000 7f48eaffd640 1 -- 172.21.15.116:0/2275028102 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f48ec089bf0 con 0x7f48fc0b2f70 2023-11-28T17:10:02.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.173+0000 7f48f9d9b640 1 --2- 172.21.15.116:0/2275028102 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f48dc06b2a0 0x7f48dc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:02.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.174+0000 7f48f9d9b640 1 --2- 172.21.15.116:0/2275028102 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f48dc06b2a0 0x7f48dc06d760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f48f0036000 tx=0x7f48f0030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:02.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.177+0000 7f48eaffd640 1 -- 172.21.15.116:0/2275028102 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f48fc14e6e0 con 0x7f48fc0b2f70 2023-11-28T17:10:02.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.402+0000 7f4900cd0640 1 -- 172.21.15.116:0/2275028102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48fc14e6e0 con 0x7f48fc0b2f70 2023-11-28T17:10:02.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.404+0000 7f48eaffd640 1 -- 172.21.15.116:0/2275028102 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f48fc14e6e0 con 0x7f48fc0b2f70 2023-11-28T17:10:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:02.406 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3138968863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":3138968863},{"type":"v1","addr":"172.21.15.156:6835","nonce":3138968863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/733981412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":733981412},{"type":"v1","addr":"172.21.15.116:6836","nonce":733981412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2751219705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2751219705},{"type":"v1","addr":"172.21.15.116:6837","nonce":2751219705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:09:00.673159+0000","modified":"2023-11-28T17:09:03.771842+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6834/3268354971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3268354971},{"type":"v1","addr":"172.21.15.156:6834","nonce":3268354971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T17:10:02.406 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2023-11-28T17:10:02.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.406+0000 7f4900cd0640 1 -- 172.21.15.116:0/2275028102 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f48dc06b2a0 msgr2=0x7f48dc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:02.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.406+0000 7f4900cd0640 1 --2- 172.21.15.116:0/2275028102 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f48dc06b2a0 0x7f48dc06d760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f48f0036000 tx=0x7f48f0030000 comp rx=0 tx=0).stop 2023-11-28T17:10:02.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.406+0000 7f4900cd0640 1 -- 172.21.15.116:0/2275028102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48fc0b2f70 msgr2=0x7f48fc0ae9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:02.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.406+0000 7f4900cd0640 1 --2- 172.21.15.116:0/2275028102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48fc0b2f70 0x7f48fc0ae9b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f48ec00c430 tx=0x7f48ec00c8f0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.406+0000 7f4900cd0640 1 -- 172.21.15.116:0/2275028102 shutdown_connections 2023-11-28T17:10:02.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.406+0000 7f4900cd0640 1 --2- 172.21.15.116:0/2275028102 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f48dc06b2a0 0x7f48dc06d760 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.406+0000 7f4900cd0640 1 --2- 172.21.15.116:0/2275028102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48fc161b30 0x7f48fc0acdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.406+0000 7f4900cd0640 1 --2- 172.21.15.116:0/2275028102 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f48fc0cf800 0x7f48fc0ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.406+0000 7f4900cd0640 1 --2- 172.21.15.116:0/2275028102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f48fc0b2f70 0x7f48fc0ae9b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.406+0000 7f4900cd0640 1 -- 172.21.15.116:0/2275028102 >> 172.21.15.116:0/2275028102 conn(0x7f48fc0cfec0 msgr2=0x7f48fc0b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:02.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.406+0000 7f4900cd0640 1 -- 172.21.15.116:0/2275028102 shutdown_connections 2023-11-28T17:10:02.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.407+0000 7f4900cd0640 1 -- 172.21.15.116:0/2275028102 wait complete. 2023-11-28T17:10:02.427 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:10:02.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.640+0000 7f526213e640 1 Processor -- start 2023-11-28T17:10:02.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.640+0000 7f526213e640 1 -- start start 2023-11-28T17:10:02.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.642+0000 7f526213e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f525c144430 0x7f525c15f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:02.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.642+0000 7f526213e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f525c06c250 0x7f525c06c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:02.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.642+0000 7f526213e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c070bd0 con 0x7f525c0703f0 2023-11-28T17:10:02.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.642+0000 7f526213e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f525c144810 con 0x7f525c06c250 2023-11-28T17:10:02.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.642+0000 7f526213e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525c06cc20 con 0x7f525c144430 2023-11-28T17:10:02.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.642+0000 7f525b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f525c144430 0x7f525c15f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:02.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.643+0000 7f525b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f525c144430 0x7f525c15f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:42802/0 (socket says 172.21.15.116:42802) 2023-11-28T17:10:02.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.643+0000 7f525b7fe640 1 -- 172.21.15.116:0/3555139220 learned_addr learned my addr 172.21.15.116:0/3555139220 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:02.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.643+0000 7f525bfff640 1 --2- 172.21.15.116:0/3555139220 >> v2:172.21.15.156:3300/0 conn(0x7f525c06c250 0x7f525c06c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:02.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.644+0000 7f525a7fc640 1 -- 172.21.15.116:0/3555139220 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573874400 0 0) 0x7f525c070bd0 con 0x7f525c0703f0 2023-11-28T17:10:02.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.644+0000 7f525a7fc640 1 -- 172.21.15.116:0/3555139220 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5244003520 con 0x7f525c0703f0 2023-11-28T17:10:02.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.644+0000 7f525b7fe640 1 -- 172.21.15.116:0/3555139220 >> v2:172.21.15.156:3300/0 conn(0x7f525c06c250 msgr2=0x7f525c06c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:02.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.644+0000 7f525b7fe640 1 --2- 172.21.15.116:0/3555139220 >> v2:172.21.15.156:3300/0 conn(0x7f525c06c250 0x7f525c06c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.644+0000 7f525b7fe640 1 -- 172.21.15.116:0/3555139220 >> v1:172.21.15.156:6789/0 conn(0x7f525c0703f0 legacy=0x7f525c071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:02.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.644+0000 7f525b7fe640 1 -- 172.21.15.116:0/3555139220 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c070eb0 con 0x7f525c144430 2023-11-28T17:10:02.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.645+0000 7f525b7fe640 1 --2- 172.21.15.116:0/3555139220 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f525c144430 0x7f525c15f7b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f524c009870 tx=0x7f524c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ccbcdf55d4eeb20 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:02.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.646+0000 7f525a7fc640 1 -- 172.21.15.116:0/3555139220 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524c030840 con 0x7f525c144430 2023-11-28T17:10:02.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.646+0000 7f525a7fc640 1 -- 172.21.15.116:0/3555139220 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f524c0309a0 con 0x7f525c144430 2023-11-28T17:10:02.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.646+0000 7f525a7fc640 1 -- 172.21.15.116:0/3555139220 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524c030d30 con 0x7f525c144430 2023-11-28T17:10:02.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.647+0000 7f526213e640 1 -- 172.21.15.116:0/3555139220 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f525c144430 msgr2=0x7f525c15f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:02.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.647+0000 7f526213e640 1 --2- 172.21.15.116:0/3555139220 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f525c144430 0x7f525c15f7b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f524c009870 tx=0x7f524c02b980 comp rx=0 tx=0).stop 2023-11-28T17:10:02.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.647+0000 7f526213e640 1 -- 172.21.15.116:0/3555139220 shutdown_connections 2023-11-28T17:10:02.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.647+0000 7f526213e640 1 --2- 172.21.15.116:0/3555139220 >> v2:172.21.15.156:3300/0 conn(0x7f525c06c250 0x7f525c06c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.647+0000 7f526213e640 1 --2- 172.21.15.116:0/3555139220 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f525c144430 0x7f525c15f7b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.647+0000 7f526213e640 1 -- 172.21.15.116:0/3555139220 >> 172.21.15.116:0/3555139220 conn(0x7f525c06a4c0 msgr2=0x7f525c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:02.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.647+0000 7f526213e640 1 -- 172.21.15.116:0/3555139220 shutdown_connections 2023-11-28T17:10:02.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.647+0000 7f526213e640 1 -- 172.21.15.116:0/3555139220 wait complete. 2023-11-28T17:10:02.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.649+0000 7f526213e640 1 Processor -- start 2023-11-28T17:10:02.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.649+0000 7f526213e640 1 -- start start 2023-11-28T17:10:02.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.650+0000 7f526213e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f525c06c250 0x7f525c152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:02.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.650+0000 7f526213e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f525c0703f0 0x7f525c1f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:02.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.650+0000 7f526213e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f525c144430 0x7f525c143290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:02.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.650+0000 7f526213e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c13e4e0 con 0x7f525c0703f0 2023-11-28T17:10:02.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.650+0000 7f526213e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c13e7c0 con 0x7f525c06c250 2023-11-28T17:10:02.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.650+0000 7f526213e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525c13eaa0 con 0x7f525c144430 2023-11-28T17:10:02.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.650+0000 7f525affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f525c0703f0 0x7f525c1f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:02.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.650+0000 7f525affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f525c0703f0 0x7f525c1f38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60146/0 (socket says 172.21.15.116:60146) 2023-11-28T17:10:02.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.650+0000 7f525affd640 1 -- 172.21.15.116:0/2716756809 learned_addr learned my addr 172.21.15.116:0/2716756809 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:02.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.651+0000 7f525bfff640 1 --2- 172.21.15.116:0/2716756809 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f525c144430 0x7f525c143290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:02.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.651+0000 7f525b7fe640 1 --2- 172.21.15.116:0/2716756809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f525c06c250 0x7f525c152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:02.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.651+0000 7f525affd640 1 -- 172.21.15.116:0/2716756809 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f525c144430 msgr2=0x7f525c143290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:02.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.651+0000 7f525affd640 1 --2- 172.21.15.116:0/2716756809 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f525c144430 0x7f525c143290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.651+0000 7f525affd640 1 -- 172.21.15.116:0/2716756809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f525c06c250 msgr2=0x7f525c152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:02.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.651+0000 7f525affd640 1 --2- 172.21.15.116:0/2716756809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f525c06c250 0x7f525c152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.651+0000 7f525affd640 1 -- 172.21.15.116:0/2716756809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f524c0094d0 con 0x7f525c0703f0 2023-11-28T17:10:02.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.651+0000 7f525affd640 1 --2- 172.21.15.116:0/2716756809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f525c0703f0 0x7f525c1f38e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f525000c490 tx=0x7f525000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:02.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.651+0000 7f5243fff640 1 -- 172.21.15.116:0/2716756809 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52500052d0 con 0x7f525c0703f0 2023-11-28T17:10:02.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.651+0000 7f5243fff640 1 -- 172.21.15.116:0/2716756809 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5250005430 con 0x7f525c0703f0 2023-11-28T17:10:02.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.651+0000 7f526213e640 1 -- 172.21.15.116:0/2716756809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f525c13ed80 con 0x7f525c0703f0 2023-11-28T17:10:02.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.651+0000 7f5243fff640 1 -- 172.21.15.116:0/2716756809 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250012870 con 0x7f525c0703f0 2023-11-28T17:10:02.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.652+0000 7f526213e640 1 -- 172.21.15.116:0/2716756809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f525c1f3e20 con 0x7f525c0703f0 2023-11-28T17:10:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.653+0000 7f526213e640 1 -- 172.21.15.116:0/2716756809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525c0693d0 con 0x7f525c0703f0 2023-11-28T17:10:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.653+0000 7f5243fff640 1 -- 172.21.15.116:0/2716756809 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f52500129d0 con 0x7f525c0703f0 2023-11-28T17:10:02.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.654+0000 7f5243fff640 1 --2- 172.21.15.116:0/2716756809 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f524806b2a0 0x7f524806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.654+0000 7f5243fff640 1 -- 172.21.15.116:0/2716756809 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f5250089800 con 0x7f525c0703f0 2023-11-28T17:10:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.655+0000 7f525b7fe640 1 --2- 172.21.15.116:0/2716756809 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f524806b2a0 0x7f524806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.655+0000 7f525b7fe640 1 --2- 172.21.15.116:0/2716756809 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f524806b2a0 0x7f524806d760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f524c036000 tx=0x7f524c008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.659+0000 7f5243fff640 1 -- 172.21.15.116:0/2716756809 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52500545b0 con 0x7f525c0703f0 2023-11-28T17:10:02.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.845+0000 7f526213e640 1 -- 172.21.15.116:0/2716756809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f525c06d100 con 0x7f525c0703f0 2023-11-28T17:10:02.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.848+0000 7f5243fff640 1 -- 172.21.15.116:0/2716756809 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f5250057bd0 con 0x7f525c0703f0 2023-11-28T17:10:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:02.850 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:43.049465+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:10:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.851+0000 7f526213e640 1 -- 172.21.15.116:0/2716756809 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f524806b2a0 msgr2=0x7f524806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.851+0000 7f526213e640 1 --2- 172.21.15.116:0/2716756809 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f524806b2a0 0x7f524806d760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f524c036000 tx=0x7f524c008030 comp rx=0 tx=0).stop 2023-11-28T17:10:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.851+0000 7f526213e640 1 -- 172.21.15.116:0/2716756809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f525c0703f0 msgr2=0x7f525c1f38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.851+0000 7f526213e640 1 --2- 172.21.15.116:0/2716756809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f525c0703f0 0x7f525c1f38e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f525000c490 tx=0x7f525000c950 comp rx=0 tx=0).stop 2023-11-28T17:10:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.851+0000 7f526213e640 1 -- 172.21.15.116:0/2716756809 shutdown_connections 2023-11-28T17:10:02.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.851+0000 7f526213e640 1 --2- 172.21.15.116:0/2716756809 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f524806b2a0 0x7f524806d760 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.851+0000 7f526213e640 1 --2- 172.21.15.116:0/2716756809 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f525c144430 0x7f525c143290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.851+0000 7f526213e640 1 --2- 172.21.15.116:0/2716756809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f525c0703f0 0x7f525c1f38e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.851+0000 7f526213e640 1 --2- 172.21.15.116:0/2716756809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f525c06c250 0x7f525c152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.851+0000 7f526213e640 1 -- 172.21.15.116:0/2716756809 >> 172.21.15.116:0/2716756809 conn(0x7f525c06a4c0 msgr2=0x7f525c0ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:02.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.851+0000 7f526213e640 1 -- 172.21.15.116:0/2716756809 shutdown_connections 2023-11-28T17:10:02.854 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:02.851+0000 7f526213e640 1 -- 172.21.15.116:0/2716756809 wait complete. 2023-11-28T17:10:02.871 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:08:59.158048+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:08:59.691319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:10:02.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:10:03.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.076+0000 7ff1daf0f640 1 Processor -- start 2023-11-28T17:10:03.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.076+0000 7ff1daf0f640 1 -- start start 2023-11-28T17:10:03.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.077+0000 7ff1daf0f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff1d4072250 0x7ff1d415ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1daf0f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d40cd320 con 0x7ff1d415b0c0 2023-11-28T17:10:03.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1daf0f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d40ced90 con 0x7ff1d40ce940 2023-11-28T17:10:03.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1daf0f640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1d40cc710 con 0x7ff1d4072250 2023-11-28T17:10:03.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1d8cab640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff1d415b0c0 0x7ff1d4071b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57384/0 (socket says 172.21.15.116:57384) 2023-11-28T17:10:03.079 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1d8cab640 1 -- 172.21.15.116:0/2139952920 learned_addr learned my addr 172.21.15.116:0/2139952920 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1d37fe640 1 -- 172.21.15.116:0/2139952920 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669781776 0 0) 0x7ff1d40cd320 con 0x7ff1d415b0c0 2023-11-28T17:10:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1d37fe640 1 -- 172.21.15.116:0/2139952920 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c4003520 con 0x7ff1d415b0c0 2023-11-28T17:10:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1d37fe640 1 -- 172.21.15.116:0/2139952920 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4025579561 0 0) 0x7ff1c4003520 con 0x7ff1d415b0c0 2023-11-28T17:10:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1d37fe640 1 -- 172.21.15.116:0/2139952920 >> v2:172.21.15.156:3300/0 conn(0x7ff1d4072250 msgr2=0x7ff1d415ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1d37fe640 1 --2- 172.21.15.116:0/2139952920 >> v2:172.21.15.156:3300/0 conn(0x7ff1d4072250 0x7ff1d415ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1d37fe640 1 -- 172.21.15.116:0/2139952920 >> v1:172.21.15.156:6789/0 conn(0x7ff1d40ce940 legacy=0x7ff1d415b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1d37fe640 1 -- 172.21.15.116:0/2139952920 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d40cc9f0 con 0x7ff1d415b0c0 2023-11-28T17:10:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1d37fe640 1 -- 172.21.15.116:0/2139952920 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7ff1c8001670 con 0x7ff1d415b0c0 2023-11-28T17:10:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1d37fe640 1 -- 172.21.15.116:0/2139952920 >> v1:172.21.15.116:6789/0 conn(0x7ff1d415b0c0 legacy=0x7ff1d4071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.078+0000 7ff1d37fe640 1 --2- 172.21.15.116:0/2139952920 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1c40039c0 0x7ff1c4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d37fe640 1 --2- 172.21.15.116:0/2139952920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1c40047c0 0x7ff1c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d37fe640 1 --2- 172.21.15.116:0/2139952920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1c4005330 0x7ff1c400b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d37fe640 1 -- 172.21.15.116:0/2139952920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d40cc710 con 0x7ff1c40047c0 2023-11-28T17:10:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d37fe640 1 -- 172.21.15.116:0/2139952920 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c400c280 con 0x7ff1c4005330 2023-11-28T17:10:03.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d37fe640 1 -- 172.21.15.116:0/2139952920 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c400c440 con 0x7ff1c40039c0 2023-11-28T17:10:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d3fff640 1 --2- 172.21.15.116:0/2139952920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1c40047c0 0x7ff1c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d8cab640 1 --2- 172.21.15.116:0/2139952920 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1c40039c0 0x7ff1c4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d3fff640 1 -- 172.21.15.116:0/2139952920 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1c40039c0 msgr2=0x7ff1c4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d3fff640 1 --2- 172.21.15.116:0/2139952920 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1c40039c0 0x7ff1c4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d3fff640 1 -- 172.21.15.116:0/2139952920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1c4005330 msgr2=0x7ff1c400b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:03.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d94ac640 1 --2- 172.21.15.116:0/2139952920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1c4005330 0x7ff1c400b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:03.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d3fff640 1 --2- 172.21.15.116:0/2139952920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1c4005330 0x7ff1c400b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d3fff640 1 -- 172.21.15.116:0/2139952920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c4003520 con 0x7ff1c40047c0 2023-11-28T17:10:03.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d94ac640 1 --2- 172.21.15.116:0/2139952920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1c4005330 0x7ff1c400b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:03.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.079+0000 7ff1d94ac640 1 -- 172.21.15.116:0/2139952920 reap_dead start 2023-11-28T17:10:03.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.081+0000 7ff1d3fff640 1 --2- 172.21.15.116:0/2139952920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1c40047c0 0x7ff1c4004c40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0009860 tx=0x7ff1c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9157641fbe302f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:03.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.081+0000 7ff1d37fe640 1 -- 172.21.15.116:0/2139952920 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0032600 con 0x7ff1c40047c0 2023-11-28T17:10:03.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.081+0000 7ff1d37fe640 1 -- 172.21.15.116:0/2139952920 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c002e050 con 0x7ff1c40047c0 2023-11-28T17:10:03.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.081+0000 7ff1d37fe640 1 -- 172.21.15.116:0/2139952920 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c00329c0 con 0x7ff1c40047c0 2023-11-28T17:10:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.082+0000 7ff1daf0f640 1 -- 172.21.15.116:0/2139952920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1c40047c0 msgr2=0x7ff1c4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.082+0000 7ff1daf0f640 1 --2- 172.21.15.116:0/2139952920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1c40047c0 0x7ff1c4004c40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0009860 tx=0x7ff1c40057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.082+0000 7ff1daf0f640 1 -- 172.21.15.116:0/2139952920 shutdown_connections 2023-11-28T17:10:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.082+0000 7ff1daf0f640 1 --2- 172.21.15.116:0/2139952920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1c40047c0 0x7ff1c4004c40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.082+0000 7ff1daf0f640 1 -- 172.21.15.116:0/2139952920 >> 172.21.15.116:0/2139952920 conn(0x7ff1d40b0cf0 msgr2=0x7ff1d40b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.082+0000 7ff1daf0f640 1 -- 172.21.15.116:0/2139952920 shutdown_connections 2023-11-28T17:10:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.082+0000 7ff1daf0f640 1 -- 172.21.15.116:0/2139952920 wait complete. 2023-11-28T17:10:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.083+0000 7ff1daf0f640 1 Processor -- start 2023-11-28T17:10:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.084+0000 7ff1daf0f640 1 -- start start 2023-11-28T17:10:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1daf0f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1c40047c0 0x7ff1d4153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1daf0f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1d40ad080 0x7ff1d41586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1daf0f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1d40ad4e0 0x7ff1d4158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1daf0f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d41f3f60 con 0x7ff1c40047c0 2023-11-28T17:10:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1daf0f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d41543d0 con 0x7ff1d40ad080 2023-11-28T17:10:03.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1daf0f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d41f3c80 con 0x7ff1d40ad4e0 2023-11-28T17:10:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1d8cab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1c40047c0 0x7ff1d4153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1d8cab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1c40047c0 0x7ff1d4153e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60154/0 (socket says 172.21.15.116:60154) 2023-11-28T17:10:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1d8cab640 1 -- 172.21.15.116:0/712934935 learned_addr learned my addr 172.21.15.116:0/712934935 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1d8cab640 1 -- 172.21.15.116:0/712934935 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1d40ad4e0 msgr2=0x7ff1d4158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1d8cab640 1 --2- 172.21.15.116:0/712934935 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1d40ad4e0 0x7ff1d4158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1d8cab640 1 -- 172.21.15.116:0/712934935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1d40ad080 msgr2=0x7ff1d41586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1d8cab640 1 --2- 172.21.15.116:0/712934935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1d40ad080 0x7ff1d41586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1d8cab640 1 -- 172.21.15.116:0/712934935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d415b620 con 0x7ff1c40047c0 2023-11-28T17:10:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.085+0000 7ff1d8cab640 1 --2- 172.21.15.116:0/712934935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1c40047c0 0x7ff1d4153e90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800e8c0 tx=0x7ff1c800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.086+0000 7ff1d17fa640 1 -- 172.21.15.116:0/712934935 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c800c870 con 0x7ff1c40047c0 2023-11-28T17:10:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.086+0000 7ff1d17fa640 1 -- 172.21.15.116:0/712934935 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c800c9f0 con 0x7ff1c40047c0 2023-11-28T17:10:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.086+0000 7ff1d17fa640 1 -- 172.21.15.116:0/712934935 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c800cda0 con 0x7ff1c40047c0 2023-11-28T17:10:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.086+0000 7ff1daf0f640 1 -- 172.21.15.116:0/712934935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d41533b0 con 0x7ff1c40047c0 2023-11-28T17:10:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.086+0000 7ff1daf0f640 1 -- 172.21.15.116:0/712934935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d4153970 con 0x7ff1c40047c0 2023-11-28T17:10:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.087+0000 7ff1daf0f640 1 -- 172.21.15.116:0/712934935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d415ed10 con 0x7ff1c40047c0 2023-11-28T17:10:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.088+0000 7ff1d17fa640 1 -- 172.21.15.116:0/712934935 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff1c801d460 con 0x7ff1c40047c0 2023-11-28T17:10:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.089+0000 7ff1d17fa640 1 --2- 172.21.15.116:0/712934935 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff1b806b2d0 0x7ff1b806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.089+0000 7ff1d17fa640 1 -- 172.21.15.116:0/712934935 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff1c8089c50 con 0x7ff1c40047c0 2023-11-28T17:10:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.089+0000 7ff1d3fff640 1 --2- 172.21.15.116:0/712934935 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff1b806b2d0 0x7ff1b806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.089+0000 7ff1d3fff640 1 --2- 172.21.15.116:0/712934935 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff1b806b2d0 0x7ff1b806d790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0034000 tx=0x7ff1c002f000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.093+0000 7ff1d17fa640 1 -- 172.21.15.116:0/712934935 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1c8054a90 con 0x7ff1c40047c0 2023-11-28T17:10:03.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.273+0000 7ff1daf0f640 1 -- 172.21.15.116:0/712934935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff1d4069360 con 0x7ff1c40047c0 2023-11-28T17:10:03.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.274+0000 7ff1d17fa640 1 -- 172.21.15.116:0/712934935 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff1c80580b0 con 0x7ff1c40047c0 2023-11-28T17:10:03.275 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:10:03.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.276+0000 7ff1daf0f640 1 -- 172.21.15.116:0/712934935 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff1b806b2d0 msgr2=0x7ff1b806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.276+0000 7ff1daf0f640 1 --2- 172.21.15.116:0/712934935 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff1b806b2d0 0x7ff1b806d790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0034000 tx=0x7ff1c002f000 comp rx=0 tx=0).stop 2023-11-28T17:10:03.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.276+0000 7ff1daf0f640 1 -- 172.21.15.116:0/712934935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1c40047c0 msgr2=0x7ff1d4153e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.276+0000 7ff1daf0f640 1 --2- 172.21.15.116:0/712934935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1c40047c0 0x7ff1d4153e90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800e8c0 tx=0x7ff1c800ed80 comp rx=0 tx=0).stop 2023-11-28T17:10:03.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.277+0000 7ff1daf0f640 1 -- 172.21.15.116:0/712934935 shutdown_connections 2023-11-28T17:10:03.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.277+0000 7ff1daf0f640 1 --2- 172.21.15.116:0/712934935 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff1b806b2d0 0x7ff1b806d790 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.277+0000 7ff1daf0f640 1 --2- 172.21.15.116:0/712934935 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1d40ad4e0 0x7ff1d4158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.277+0000 7ff1daf0f640 1 --2- 172.21.15.116:0/712934935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1d40ad080 0x7ff1d41586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.277+0000 7ff1daf0f640 1 --2- 172.21.15.116:0/712934935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1c40047c0 0x7ff1d4153e90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.277+0000 7ff1daf0f640 1 -- 172.21.15.116:0/712934935 >> 172.21.15.116:0/712934935 conn(0x7ff1d40b0cf0 msgr2=0x7ff1d4071910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:03.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.277+0000 7ff1daf0f640 1 -- 172.21.15.116:0/712934935 shutdown_connections 2023-11-28T17:10:03.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.277+0000 7ff1daf0f640 1 -- 172.21.15.116:0/712934935 wait complete. 2023-11-28T17:10:03.296 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:10:03.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.507+0000 7fa8fcb67640 1 Processor -- start 2023-11-28T17:10:03.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.507+0000 7fa8fcb67640 1 -- start start 2023-11-28T17:10:03.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.508+0000 7fa8fcb67640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa8f815d480 0x7fa8f80d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.509+0000 7fa8fcb67640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f815d860 con 0x7fa8f80b1570 2023-11-28T17:10:03.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.509+0000 7fa8fcb67640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f80b00d0 con 0x7fa8f80ce6a0 2023-11-28T17:10:03.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.509+0000 7fa8fcb67640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80d0400 con 0x7fa8f815d480 2023-11-28T17:10:03.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.509+0000 7fa8f7fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa8f80b1570 0x7fa8f80cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57396/0 (socket says 172.21.15.116:57396) 2023-11-28T17:10:03.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.509+0000 7fa8f7fff640 1 -- 172.21.15.116:0/2097182964 learned_addr learned my addr 172.21.15.116:0/2097182964 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:03.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.509+0000 7fa8f77fe640 1 --2- 172.21.15.116:0/2097182964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa8f815d480 0x7fa8f80d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:03.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.509+0000 7fa8f6ffd640 1 -- 172.21.15.116:0/2097182964 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994945224 0 0) 0x7fa8f0001490 con 0x7fa8f80b1570 2023-11-28T17:10:03.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f6ffd640 1 -- 172.21.15.116:0/2097182964 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e4003520 con 0x7fa8f80b1570 2023-11-28T17:10:03.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f6ffd640 1 -- 172.21.15.116:0/2097182964 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 611864099 0 0) 0x7fa8e4003520 con 0x7fa8f80b1570 2023-11-28T17:10:03.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f6ffd640 1 -- 172.21.15.116:0/2097182964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa8f815d480 msgr2=0x7fa8f80d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:03.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f6ffd640 1 --2- 172.21.15.116:0/2097182964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa8f815d480 0x7fa8f80d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f6ffd640 1 -- 172.21.15.116:0/2097182964 >> v1:172.21.15.156:6789/0 conn(0x7fa8f80ce6a0 legacy=0x7fa8f80cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f6ffd640 1 -- 172.21.15.116:0/2097182964 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f80cebd0 con 0x7fa8f80b1570 2023-11-28T17:10:03.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f6ffd640 1 -- 172.21.15.116:0/2097182964 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fa8f0001b80 con 0x7fa8f80b1570 2023-11-28T17:10:03.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f6ffd640 1 -- 172.21.15.116:0/2097182964 >> v1:172.21.15.116:6789/0 conn(0x7fa8f80b1570 legacy=0x7fa8f80cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f6ffd640 1 --2- 172.21.15.116:0/2097182964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8e40039a0 0x7fa8e4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f6ffd640 1 --2- 172.21.15.116:0/2097182964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa8f815d480 0x7fa8e40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f6ffd640 1 --2- 172.21.15.116:0/2097182964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8e4004d00 0x7fa8e4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f6ffd640 1 -- 172.21.15.116:0/2097182964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80d0400 con 0x7fa8e40039a0 2023-11-28T17:10:03.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f6ffd640 1 -- 172.21.15.116:0/2097182964 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e400bab0 con 0x7fa8e4004d00 2023-11-28T17:10:03.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f6ffd640 1 -- 172.21.15.116:0/2097182964 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e400bc70 con 0x7fa8f815d480 2023-11-28T17:10:03.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.510+0000 7fa8f77fe640 1 --2- 172.21.15.116:0/2097182964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8e40039a0 0x7fa8e4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:03.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.511+0000 7fa8f77fe640 1 -- 172.21.15.116:0/2097182964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa8f815d480 msgr2=0x7fa8e40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:03.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.511+0000 7fa8f77fe640 1 --2- 172.21.15.116:0/2097182964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa8f815d480 0x7fa8e40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.511+0000 7fa8f77fe640 1 -- 172.21.15.116:0/2097182964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8e4004d00 msgr2=0x7fa8e4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:03.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.511+0000 7fa8f77fe640 1 --2- 172.21.15.116:0/2097182964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8e4004d00 0x7fa8e4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.511+0000 7fa8f77fe640 1 -- 172.21.15.116:0/2097182964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e4003520 con 0x7fa8e40039a0 2023-11-28T17:10:03.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.512+0000 7fa8f77fe640 1 --2- 172.21.15.116:0/2097182964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8e40039a0 0x7fa8e4003dc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8009ac0 tx=0x7fa8e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e517acfb56b511 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:03.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.512+0000 7fa8f6ffd640 1 -- 172.21.15.116:0/2097182964 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e802c630 con 0x7fa8e40039a0 2023-11-28T17:10:03.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.512+0000 7fa8f6ffd640 1 -- 172.21.15.116:0/2097182964 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e802c790 con 0x7fa8e40039a0 2023-11-28T17:10:03.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.512+0000 7fa8f6ffd640 1 -- 172.21.15.116:0/2097182964 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e802cb20 con 0x7fa8e40039a0 2023-11-28T17:10:03.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.512+0000 7fa8fcb67640 1 -- 172.21.15.116:0/2097182964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8e40039a0 msgr2=0x7fa8e4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.512+0000 7fa8fcb67640 1 --2- 172.21.15.116:0/2097182964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8e40039a0 0x7fa8e4003dc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8009ac0 tx=0x7fa8e40057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.512+0000 7fa8f7fff640 1 -- 172.21.15.116:0/2097182964 reap_dead start 2023-11-28T17:10:03.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.513+0000 7fa8fcb67640 1 -- 172.21.15.116:0/2097182964 shutdown_connections 2023-11-28T17:10:03.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.513+0000 7fa8fcb67640 1 -- 172.21.15.116:0/2097182964 >> 172.21.15.116:0/2097182964 conn(0x7fa8f809c2f0 msgr2=0x7fa8f80d2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:03.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.513+0000 7fa8fcb67640 1 -- 172.21.15.116:0/2097182964 shutdown_connections 2023-11-28T17:10:03.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.513+0000 7fa8fcb67640 1 -- 172.21.15.116:0/2097182964 wait complete. 2023-11-28T17:10:03.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.514+0000 7fa8fcb67640 1 Processor -- start 2023-11-28T17:10:03.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.515+0000 7fa8fcb67640 1 -- start start 2023-11-28T17:10:03.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8fcb67640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa8e40039a0 0x7fa8f80c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8fcb67640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f80b5d60 0x7fa8f80b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8fcb67640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8f80b83a0 0x7fa8f80b8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8fcb67640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f81f3250 con 0x7fa8f80b5d60 2023-11-28T17:10:03.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8fcb67640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80b6720 con 0x7fa8f80b83a0 2023-11-28T17:10:03.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8fcb67640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80c7ea0 con 0x7fa8e40039a0 2023-11-28T17:10:03.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8eeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f80b5d60 0x7fa8f80b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:03.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8eeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f80b5d60 0x7fa8f80b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60160/0 (socket says 172.21.15.116:60160) 2023-11-28T17:10:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8eeffd640 1 -- 172.21.15.116:0/4087727028 learned_addr learned my addr 172.21.15.116:0/4087727028 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8f77fe640 1 --2- 172.21.15.116:0/4087727028 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa8e40039a0 0x7fa8f80c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8f7fff640 1 --2- 172.21.15.116:0/4087727028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8f80b83a0 0x7fa8f80b8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8eeffd640 1 -- 172.21.15.116:0/4087727028 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa8e40039a0 msgr2=0x7fa8f80c7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8eeffd640 1 --2- 172.21.15.116:0/4087727028 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa8e40039a0 0x7fa8f80c7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8eeffd640 1 -- 172.21.15.116:0/4087727028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8f80b83a0 msgr2=0x7fa8f80b8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8eeffd640 1 --2- 172.21.15.116:0/4087727028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8f80b83a0 0x7fa8f80b8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.516+0000 7fa8eeffd640 1 -- 172.21.15.116:0/4087727028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f80c80d0 con 0x7fa8f80b5d60 2023-11-28T17:10:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.517+0000 7fa8eeffd640 1 --2- 172.21.15.116:0/4087727028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f80b5d60 0x7fa8f80b61e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000c500 tx=0x7fa8e000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.517+0000 7fa8f4ff9640 1 -- 172.21.15.116:0/4087727028 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00052d0 con 0x7fa8f80b5d60 2023-11-28T17:10:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.517+0000 7fa8f4ff9640 1 -- 172.21.15.116:0/4087727028 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e0005430 con 0x7fa8f80b5d60 2023-11-28T17:10:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.517+0000 7fa8f4ff9640 1 -- 172.21.15.116:0/4087727028 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0012850 con 0x7fa8f80b5d60 2023-11-28T17:10:03.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.517+0000 7fa8fcb67640 1 -- 172.21.15.116:0/4087727028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f80c83b0 con 0x7fa8f80b5d60 2023-11-28T17:10:03.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.517+0000 7fa8fcb67640 1 -- 172.21.15.116:0/4087727028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f80c8970 con 0x7fa8f80b5d60 2023-11-28T17:10:03.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.518+0000 7fa8fcb67640 1 -- 172.21.15.116:0/4087727028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f808c770 con 0x7fa8f80b5d60 2023-11-28T17:10:03.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.519+0000 7fa8f4ff9640 1 -- 172.21.15.116:0/4087727028 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa8e00129b0 con 0x7fa8f80b5d60 2023-11-28T17:10:03.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.520+0000 7fa8f4ff9640 1 --2- 172.21.15.116:0/4087727028 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa8d806b2a0 0x7fa8d806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.521+0000 7fa8f4ff9640 1 -- 172.21.15.116:0/4087727028 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa8e0089720 con 0x7fa8f80b5d60 2023-11-28T17:10:03.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.521+0000 7fa8f77fe640 1 --2- 172.21.15.116:0/4087727028 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa8d806b2a0 0x7fa8d806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:03.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.521+0000 7fa8f77fe640 1 --2- 172.21.15.116:0/4087727028 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa8d806b2a0 0x7fa8d806d760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8036000 tx=0x7fa8e8030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:03.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.525+0000 7fa8f4ff9640 1 -- 172.21.15.116:0/4087727028 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa8e0054600 con 0x7fa8f80b5d60 2023-11-28T17:10:03.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.708+0000 7fa8fcb67640 1 -- 172.21.15.116:0/4087727028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa8f804e960 con 0x7fa8f80b5d60 2023-11-28T17:10:03.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.709+0000 7fa8f4ff9640 1 -- 172.21.15.116:0/4087727028 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa8e0057c20 con 0x7fa8f80b5d60 2023-11-28T17:10:03.711 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:10:03.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.712+0000 7fa8fcb67640 1 -- 172.21.15.116:0/4087727028 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa8d806b2a0 msgr2=0x7fa8d806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.712+0000 7fa8fcb67640 1 --2- 172.21.15.116:0/4087727028 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa8d806b2a0 0x7fa8d806d760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8036000 tx=0x7fa8e8030000 comp rx=0 tx=0).stop 2023-11-28T17:10:03.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.712+0000 7fa8fcb67640 1 -- 172.21.15.116:0/4087727028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f80b5d60 msgr2=0x7fa8f80b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.713 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.712+0000 7fa8fcb67640 1 --2- 172.21.15.116:0/4087727028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f80b5d60 0x7fa8f80b61e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000c500 tx=0x7fa8e000c9c0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.712+0000 7fa8fcb67640 1 -- 172.21.15.116:0/4087727028 shutdown_connections 2023-11-28T17:10:03.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.712+0000 7fa8fcb67640 1 --2- 172.21.15.116:0/4087727028 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa8d806b2a0 0x7fa8d806d760 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.712+0000 7fa8fcb67640 1 --2- 172.21.15.116:0/4087727028 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8f80b83a0 0x7fa8f80b8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.712+0000 7fa8fcb67640 1 --2- 172.21.15.116:0/4087727028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8f80b5d60 0x7fa8f80b61e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.712+0000 7fa8fcb67640 1 --2- 172.21.15.116:0/4087727028 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa8e40039a0 0x7fa8f80c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.712+0000 7fa8fcb67640 1 -- 172.21.15.116:0/4087727028 >> 172.21.15.116:0/4087727028 conn(0x7fa8f809c2f0 msgr2=0x7fa8f815ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:03.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.712+0000 7fa8fcb67640 1 -- 172.21.15.116:0/4087727028 shutdown_connections 2023-11-28T17:10:03.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.712+0000 7fa8fcb67640 1 -- 172.21.15.116:0/4087727028 wait complete. 2023-11-28T17:10:03.733 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:10:03.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.948+0000 7f43da383640 1 Processor -- start 2023-11-28T17:10:03.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.948+0000 7f43da383640 1 -- start start 2023-11-28T17:10:03.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.950+0000 7f43da383640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43d413cee0 0x7f43d413d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.950+0000 7f43da383640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d413d2c0 con 0x7f43d4072430 2023-11-28T17:10:03.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.950+0000 7f43da383640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d41400e0 con 0x7f43d413f640 2023-11-28T17:10:03.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.950+0000 7f43da383640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d41403c0 con 0x7f43d413cee0 2023-11-28T17:10:03.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.950+0000 7f43d3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43d413cee0 0x7f43d413d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:03.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.950+0000 7f43d3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43d413cee0 0x7f43d413d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60172/0 (socket says 172.21.15.116:60172) 2023-11-28T17:10:03.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.950+0000 7f43d3fff640 1 -- 172.21.15.116:0/1386887782 learned_addr learned my addr 172.21.15.116:0/1386887782 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:03.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.950+0000 7f43d8920640 1 --1- 172.21.15.116:0/1386887782 >> v1:172.21.15.116:6789/0 conn(0x7f43d4072430 0x7f43d40ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57408/0 (socket says 172.21.15.116:57408) 2023-11-28T17:10:03.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.951+0000 7f43d3fff640 1 -- 172.21.15.116:0/1386887782 >> v1:172.21.15.156:6789/0 conn(0x7f43d413f640 legacy=0x7f43d4071d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:03.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.951+0000 7f43d3fff640 1 -- 172.21.15.116:0/1386887782 >> v1:172.21.15.116:6789/0 conn(0x7f43d4072430 legacy=0x7f43d40ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.951+0000 7f43d37fe640 1 -- 172.21.15.116:0/1386887782 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746664576 0 0) 0x7f43d413d2c0 con 0x7f43d4072430 2023-11-28T17:10:03.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.951+0000 7f43d3fff640 1 -- 172.21.15.116:0/1386887782 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d40cdb50 con 0x7f43d413cee0 2023-11-28T17:10:03.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.952+0000 7f43d3fff640 1 --2- 172.21.15.116:0/1386887782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d413cee0 0x7f43d413d600 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009e50 tx=0x7f43c402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c09f586938620f9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:03.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.952+0000 7f43d37fe640 1 -- 172.21.15.116:0/1386887782 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c402c9f0 con 0x7f43d413cee0 2023-11-28T17:10:03.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.952+0000 7f43d37fe640 1 -- 172.21.15.116:0/1386887782 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c402cb50 con 0x7f43d413cee0 2023-11-28T17:10:03.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.952+0000 7f43d37fe640 1 -- 172.21.15.116:0/1386887782 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c40343f0 con 0x7f43d413cee0 2023-11-28T17:10:03.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.953+0000 7f43da383640 1 -- 172.21.15.116:0/1386887782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d413cee0 msgr2=0x7f43d413d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.953+0000 7f43da383640 1 --2- 172.21.15.116:0/1386887782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d413cee0 0x7f43d413d600 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009e50 tx=0x7f43c402be50 comp rx=0 tx=0).stop 2023-11-28T17:10:03.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.953+0000 7f43da383640 1 -- 172.21.15.116:0/1386887782 shutdown_connections 2023-11-28T17:10:03.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.953+0000 7f43da383640 1 --2- 172.21.15.116:0/1386887782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d413cee0 0x7f43d413d600 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.953+0000 7f43da383640 1 -- 172.21.15.116:0/1386887782 >> 172.21.15.116:0/1386887782 conn(0x7f43d40b0cf0 msgr2=0x7f43d40b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:03.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.953+0000 7f43da383640 1 -- 172.21.15.116:0/1386887782 shutdown_connections 2023-11-28T17:10:03.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.953+0000 7f43da383640 1 -- 172.21.15.116:0/1386887782 wait complete. 2023-11-28T17:10:03.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.955+0000 7f43da383640 1 Processor -- start 2023-11-28T17:10:03.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.955+0000 7f43da383640 1 -- start start 2023-11-28T17:10:03.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.956+0000 7f43da383640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d4072430 0x7f43d4154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.956+0000 7f43da383640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43d413cee0 0x7f43d41f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.956+0000 7f43da383640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43d413f640 0x7f43d413b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.956+0000 7f43da383640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d4153840 con 0x7f43d4072430 2023-11-28T17:10:03.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.956+0000 7f43da383640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d41517e0 con 0x7f43d413f640 2023-11-28T17:10:03.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.956+0000 7f43da383640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d41592c0 con 0x7f43d413cee0 2023-11-28T17:10:03.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.956+0000 7f43d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d4072430 0x7f43d4154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:03.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.956+0000 7f43d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d4072430 0x7f43d4154250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60176/0 (socket says 172.21.15.116:60176) 2023-11-28T17:10:03.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.956+0000 7f43d3fff640 1 -- 172.21.15.116:0/3887829708 learned_addr learned my addr 172.21.15.116:0/3887829708 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:03.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.956+0000 7f43d3fff640 1 -- 172.21.15.116:0/3887829708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43d413cee0 msgr2=0x7f43d41f2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.956+0000 7f43d3fff640 1 --2- 172.21.15.116:0/3887829708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43d413cee0 0x7f43d41f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.956+0000 7f43d3fff640 1 -- 172.21.15.116:0/3887829708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43d413f640 msgr2=0x7f43d413b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:03.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.956+0000 7f43d3fff640 1 --2- 172.21.15.116:0/3887829708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43d413f640 0x7f43d413b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:03.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.956+0000 7f43d3fff640 1 -- 172.21.15.116:0/3887829708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c4009b30 con 0x7f43d4072430 2023-11-28T17:10:03.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.957+0000 7f43d3fff640 1 --2- 172.21.15.116:0/3887829708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d4072430 0x7f43d4154250 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009e50 tx=0x7f43c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:03.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.957+0000 7f43d17fa640 1 -- 172.21.15.116:0/3887829708 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c403e450 con 0x7f43d4072430 2023-11-28T17:10:03.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.957+0000 7f43d17fa640 1 -- 172.21.15.116:0/3887829708 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c403e5d0 con 0x7f43d4072430 2023-11-28T17:10:03.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.957+0000 7f43d17fa640 1 -- 172.21.15.116:0/3887829708 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c403e980 con 0x7f43d4072430 2023-11-28T17:10:03.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.957+0000 7f43da383640 1 -- 172.21.15.116:0/3887829708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d41595a0 con 0x7f43d4072430 2023-11-28T17:10:03.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.957+0000 7f43da383640 1 -- 172.21.15.116:0/3887829708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d4153d30 con 0x7f43d4072430 2023-11-28T17:10:03.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.958+0000 7f43da383640 1 -- 172.21.15.116:0/3887829708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d4153f50 con 0x7f43d4072430 2023-11-28T17:10:03.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.959+0000 7f43d17fa640 1 -- 172.21.15.116:0/3887829708 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f43c403eae0 con 0x7f43d4072430 2023-11-28T17:10:03.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.960+0000 7f43d17fa640 1 --2- 172.21.15.116:0/3887829708 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f43b406b2a0 0x7f43b406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:03.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.960+0000 7f43d17fa640 1 -- 172.21.15.116:0/3887829708 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f43c40acbd0 con 0x7f43d4072430 2023-11-28T17:10:03.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.960+0000 7f43cb7fe640 1 --2- 172.21.15.116:0/3887829708 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f43b406b2a0 0x7f43b406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:03.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.961+0000 7f43cb7fe640 1 --2- 172.21.15.116:0/3887829708 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f43b406b2a0 0x7f43b406d760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f43d41f6360 tx=0x7f43bc0079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:03.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:03.964+0000 7f43d17fa640 1 -- 172.21.15.116:0/3887829708 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f43c4077aa0 con 0x7f43d4072430 2023-11-28T17:10:04.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.157+0000 7f43da383640 1 -- 172.21.15.116:0/3887829708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f43d4159730 con 0x7f43d4072430 2023-11-28T17:10:04.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.158+0000 7f43d17fa640 1 -- 172.21.15.116:0/3887829708 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f43c40430a0 con 0x7f43d4072430 2023-11-28T17:10:04.159 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:10:04.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.160+0000 7f43da383640 1 -- 172.21.15.116:0/3887829708 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f43b406b2a0 msgr2=0x7f43b406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:04.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.160+0000 7f43da383640 1 --2- 172.21.15.116:0/3887829708 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f43b406b2a0 0x7f43b406d760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f43d41f6360 tx=0x7f43bc0079b0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.161+0000 7f43da383640 1 -- 172.21.15.116:0/3887829708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d4072430 msgr2=0x7f43d4154250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:04.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.161+0000 7f43da383640 1 --2- 172.21.15.116:0/3887829708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d4072430 0x7f43d4154250 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009e50 tx=0x7f43c4004a40 comp rx=0 tx=0).stop 2023-11-28T17:10:04.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.161+0000 7f43da383640 1 -- 172.21.15.116:0/3887829708 shutdown_connections 2023-11-28T17:10:04.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.161+0000 7f43da383640 1 --2- 172.21.15.116:0/3887829708 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f43b406b2a0 0x7f43b406d760 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.161+0000 7f43da383640 1 --2- 172.21.15.116:0/3887829708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f43d413f640 0x7f43d413b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.161+0000 7f43da383640 1 --2- 172.21.15.116:0/3887829708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f43d413cee0 0x7f43d41f2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.161+0000 7f43da383640 1 --2- 172.21.15.116:0/3887829708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d4072430 0x7f43d4154250 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.161+0000 7f43da383640 1 -- 172.21.15.116:0/3887829708 >> 172.21.15.116:0/3887829708 conn(0x7f43d40b0cf0 msgr2=0x7f43d40aeea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:04.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.161+0000 7f43da383640 1 -- 172.21.15.116:0/3887829708 shutdown_connections 2023-11-28T17:10:04.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.161+0000 7f43da383640 1 -- 172.21.15.116:0/3887829708 wait complete. 2023-11-28T17:10:04.183 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:10:04.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.395+0000 7ff6cad1d640 1 Processor -- start 2023-11-28T17:10:04.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.396+0000 7ff6cad1d640 1 -- start start 2023-11-28T17:10:04.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.397+0000 7ff6cad1d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff6c406c960 0x7ff6c4167e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:04.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.397+0000 7ff6cad1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c41405a0 con 0x7ff6c41401c0 2023-11-28T17:10:04.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.397+0000 7ff6cad1d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c4140880 con 0x7ff6c4071710 2023-11-28T17:10:04.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.397+0000 7ff6cad1d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6c415db00 con 0x7ff6c406c960 2023-11-28T17:10:04.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.398+0000 7ff6c8ab9640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff6c41401c0 0x7ff6c4071eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57410/0 (socket says 172.21.15.116:57410) 2023-11-28T17:10:04.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.398+0000 7ff6c8ab9640 1 -- 172.21.15.116:0/3670021545 learned_addr learned my addr 172.21.15.116:0/3670021545 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.398+0000 7ff6c37fe640 1 -- 172.21.15.116:0/3670021545 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571675334 0 0) 0x7ff6c41405a0 con 0x7ff6c41401c0 2023-11-28T17:10:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.398+0000 7ff6c37fe640 1 -- 172.21.15.116:0/3670021545 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6b4003520 con 0x7ff6c41401c0 2023-11-28T17:10:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.398+0000 7ff6c92ba640 1 --2- 172.21.15.116:0/3670021545 >> v2:172.21.15.156:3300/0 conn(0x7ff6c406c960 0x7ff6c4167e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.398+0000 7ff6c37fe640 1 -- 172.21.15.116:0/3670021545 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1536737441 0 0) 0x7ff6b4003520 con 0x7ff6c41401c0 2023-11-28T17:10:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.398+0000 7ff6c37fe640 1 -- 172.21.15.116:0/3670021545 >> v2:172.21.15.156:3300/0 conn(0x7ff6c406c960 msgr2=0x7ff6c4167e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.398+0000 7ff6c37fe640 1 --2- 172.21.15.116:0/3670021545 >> v2:172.21.15.156:3300/0 conn(0x7ff6c406c960 0x7ff6c4167e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.398+0000 7ff6c37fe640 1 -- 172.21.15.116:0/3670021545 >> v1:172.21.15.156:6789/0 conn(0x7ff6c4071710 legacy=0x7ff6c406c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.398+0000 7ff6c37fe640 1 -- 172.21.15.116:0/3670021545 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c415dde0 con 0x7ff6c41401c0 2023-11-28T17:10:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.398+0000 7ff6c37fe640 1 -- 172.21.15.116:0/3670021545 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7ff6b8001670 con 0x7ff6c41401c0 2023-11-28T17:10:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.398+0000 7ff6c37fe640 1 -- 172.21.15.116:0/3670021545 >> v1:172.21.15.116:6789/0 conn(0x7ff6c41401c0 legacy=0x7ff6c4071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:04.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.398+0000 7ff6c37fe640 1 --2- 172.21.15.116:0/3670021545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6b40039c0 0x7ff6b4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:04.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.399+0000 7ff6c37fe640 1 --2- 172.21.15.116:0/3670021545 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6b40047c0 0x7ff6b4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:04.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.399+0000 7ff6c37fe640 1 --2- 172.21.15.116:0/3670021545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6b40051d0 0x7ff6b400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:04.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.399+0000 7ff6c37fe640 1 -- 172.21.15.116:0/3670021545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c415db00 con 0x7ff6b40051d0 2023-11-28T17:10:04.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.399+0000 7ff6c37fe640 1 -- 172.21.15.116:0/3670021545 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b400c100 con 0x7ff6b40039c0 2023-11-28T17:10:04.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.399+0000 7ff6c37fe640 1 -- 172.21.15.116:0/3670021545 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b400c2c0 con 0x7ff6b40047c0 2023-11-28T17:10:04.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.399+0000 7ff6c92ba640 1 --2- 172.21.15.116:0/3670021545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6b40051d0 0x7ff6b400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:04.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.399+0000 7ff6c8ab9640 1 --2- 172.21.15.116:0/3670021545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6b40039c0 0x7ff6b4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:04.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.399+0000 7ff6c92ba640 1 -- 172.21.15.116:0/3670021545 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6b40047c0 msgr2=0x7ff6b4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:04.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.399+0000 7ff6c92ba640 1 --2- 172.21.15.116:0/3670021545 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6b40047c0 0x7ff6b4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.399+0000 7ff6c92ba640 1 -- 172.21.15.116:0/3670021545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6b40039c0 msgr2=0x7ff6b4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:04.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.399+0000 7ff6c92ba640 1 --2- 172.21.15.116:0/3670021545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6b40039c0 0x7ff6b4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.399+0000 7ff6c92ba640 1 -- 172.21.15.116:0/3670021545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b4003520 con 0x7ff6b40051d0 2023-11-28T17:10:04.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.399+0000 7ff6c8ab9640 1 --2- 172.21.15.116:0/3670021545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6b40039c0 0x7ff6b4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:04.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.400+0000 7ff6c92ba640 1 -- 172.21.15.116:0/3670021545 reap_dead start 2023-11-28T17:10:04.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.400+0000 7ff6c92ba640 1 --2- 172.21.15.116:0/3670021545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6b40051d0 0x7ff6b400b980 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc007330 tx=0x7ff6b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58186f4077ff8b1b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:04.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.400+0000 7ff6c37fe640 1 -- 172.21.15.116:0/3670021545 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc02ccd0 con 0x7ff6b40051d0 2023-11-28T17:10:04.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.401+0000 7ff6c37fe640 1 -- 172.21.15.116:0/3670021545 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6bc02ce30 con 0x7ff6b40051d0 2023-11-28T17:10:04.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.401+0000 7ff6c37fe640 1 -- 172.21.15.116:0/3670021545 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc003bd0 con 0x7ff6b40051d0 2023-11-28T17:10:04.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.401+0000 7ff6cad1d640 1 -- 172.21.15.116:0/3670021545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6b40051d0 msgr2=0x7ff6b400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:04.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.401+0000 7ff6cad1d640 1 --2- 172.21.15.116:0/3670021545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6b40051d0 0x7ff6b400b980 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc007330 tx=0x7ff6b40057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.401+0000 7ff6cad1d640 1 -- 172.21.15.116:0/3670021545 shutdown_connections 2023-11-28T17:10:04.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.401+0000 7ff6cad1d640 1 --2- 172.21.15.116:0/3670021545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6b40051d0 0x7ff6b400b980 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.401+0000 7ff6cad1d640 1 -- 172.21.15.116:0/3670021545 >> 172.21.15.116:0/3670021545 conn(0x7ff6c406a4c0 msgr2=0x7ff6c406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:04.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.401+0000 7ff6cad1d640 1 -- 172.21.15.116:0/3670021545 shutdown_connections 2023-11-28T17:10:04.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.401+0000 7ff6cad1d640 1 -- 172.21.15.116:0/3670021545 wait complete. 2023-11-28T17:10:04.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.403+0000 7ff6cad1d640 1 Processor -- start 2023-11-28T17:10:04.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.403+0000 7ff6cad1d640 1 -- start start 2023-11-28T17:10:04.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.404+0000 7ff6cad1d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6b40051d0 0x7ff6c41387e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:04.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.404+0000 7ff6cad1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6c413e0e0 0x7ff6c4155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:04.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.404+0000 7ff6cad1d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6c415b3f0 0x7ff6c40b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:04.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.404+0000 7ff6cad1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c415b870 con 0x7ff6c413e0e0 2023-11-28T17:10:04.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.404+0000 7ff6cad1d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c415ff30 con 0x7ff6c415b3f0 2023-11-28T17:10:04.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.404+0000 7ff6cad1d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c415e950 con 0x7ff6b40051d0 2023-11-28T17:10:04.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.404+0000 7ff6c3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6c413e0e0 0x7ff6c4155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:04.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.404+0000 7ff6c3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6c413e0e0 0x7ff6c4155430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60206/0 (socket says 172.21.15.116:60206) 2023-11-28T17:10:04.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.404+0000 7ff6c3fff640 1 -- 172.21.15.116:0/815150941 learned_addr learned my addr 172.21.15.116:0/815150941 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:04.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.405+0000 7ff6c3fff640 1 -- 172.21.15.116:0/815150941 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6b40051d0 msgr2=0x7ff6c41387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:04.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.405+0000 7ff6c3fff640 1 --2- 172.21.15.116:0/815150941 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6b40051d0 0x7ff6c41387e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.405+0000 7ff6c3fff640 1 -- 172.21.15.116:0/815150941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6c415b3f0 msgr2=0x7ff6c40b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:04.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.405+0000 7ff6c92ba640 1 --2- 172.21.15.116:0/815150941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6c415b3f0 0x7ff6c40b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:04.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.405+0000 7ff6c3fff640 1 --2- 172.21.15.116:0/815150941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6c415b3f0 0x7ff6c40b0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.405+0000 7ff6c3fff640 1 -- 172.21.15.116:0/815150941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c4160180 con 0x7ff6c413e0e0 2023-11-28T17:10:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.405+0000 7ff6c3fff640 1 --2- 172.21.15.116:0/815150941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6c413e0e0 0x7ff6c4155430 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000c4b0 tx=0x7ff6b00036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.405+0000 7ff6c17fa640 1 -- 172.21.15.116:0/815150941 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0011790 con 0x7ff6c413e0e0 2023-11-28T17:10:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.405+0000 7ff6c17fa640 1 -- 172.21.15.116:0/815150941 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b00118f0 con 0x7ff6c413e0e0 2023-11-28T17:10:04.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.405+0000 7ff6c17fa640 1 -- 172.21.15.116:0/815150941 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0011ca0 con 0x7ff6c413e0e0 2023-11-28T17:10:04.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.405+0000 7ff6cad1d640 1 -- 172.21.15.116:0/815150941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c4160460 con 0x7ff6c413e0e0 2023-11-28T17:10:04.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.405+0000 7ff6cad1d640 1 -- 172.21.15.116:0/815150941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c4160a20 con 0x7ff6c413e0e0 2023-11-28T17:10:04.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.406+0000 7ff6cad1d640 1 -- 172.21.15.116:0/815150941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c40693d0 con 0x7ff6c413e0e0 2023-11-28T17:10:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.407+0000 7ff6c17fa640 1 -- 172.21.15.116:0/815150941 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff6b001c460 con 0x7ff6c413e0e0 2023-11-28T17:10:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.408+0000 7ff6c17fa640 1 --2- 172.21.15.116:0/815150941 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff6a806b2d0 0x7ff6a806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.408+0000 7ff6c17fa640 1 -- 172.21.15.116:0/815150941 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff6b0089ba0 con 0x7ff6c413e0e0 2023-11-28T17:10:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.409+0000 7ff6c8ab9640 1 --2- 172.21.15.116:0/815150941 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff6a806b2d0 0x7ff6a806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.409+0000 7ff6c8ab9640 1 --2- 172.21.15.116:0/815150941 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff6a806b2d0 0x7ff6a806d790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff6c41f65b0 tx=0x7ff6b8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.413+0000 7ff6c17fa640 1 -- 172.21.15.116:0/815150941 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff6b0054950 con 0x7ff6c413e0e0 2023-11-28T17:10:04.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.641+0000 7ff6cad1d640 1 -- 172.21.15.116:0/815150941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6c415ea90 con 0x7ff6c413e0e0 2023-11-28T17:10:04.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.642+0000 7ff6c17fa640 1 -- 172.21.15.116:0/815150941 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7ff6b0057f70 con 0x7ff6c413e0e0 2023-11-28T17:10:04.643 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:04.644 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3138968863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":3138968863},{"type":"v1","addr":"172.21.15.156:6835","nonce":3138968863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/733981412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":733981412},{"type":"v1","addr":"172.21.15.116:6836","nonce":733981412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2751219705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2751219705},{"type":"v1","addr":"172.21.15.116:6837","nonce":2751219705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:09:00.673159+0000","modified":"2023-11-28T17:09:03.771842+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6834/3268354971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3268354971},{"type":"v1","addr":"172.21.15.156:6834","nonce":3268354971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T17:10:04.644 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2023-11-28T17:10:04.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.644+0000 7ff6cad1d640 1 -- 172.21.15.116:0/815150941 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff6a806b2d0 msgr2=0x7ff6a806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:04.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.644+0000 7ff6cad1d640 1 --2- 172.21.15.116:0/815150941 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff6a806b2d0 0x7ff6a806d790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff6c41f65b0 tx=0x7ff6b8009040 comp rx=0 tx=0).stop 2023-11-28T17:10:04.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.644+0000 7ff6cad1d640 1 -- 172.21.15.116:0/815150941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6c413e0e0 msgr2=0x7ff6c4155430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:04.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.644+0000 7ff6cad1d640 1 --2- 172.21.15.116:0/815150941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6c413e0e0 0x7ff6c4155430 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000c4b0 tx=0x7ff6b00036f0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.644+0000 7ff6cad1d640 1 -- 172.21.15.116:0/815150941 shutdown_connections 2023-11-28T17:10:04.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.644+0000 7ff6cad1d640 1 --2- 172.21.15.116:0/815150941 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff6a806b2d0 0x7ff6a806d790 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.644+0000 7ff6cad1d640 1 --2- 172.21.15.116:0/815150941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6c415b3f0 0x7ff6c40b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.644+0000 7ff6cad1d640 1 --2- 172.21.15.116:0/815150941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6c413e0e0 0x7ff6c4155430 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.644+0000 7ff6cad1d640 1 --2- 172.21.15.116:0/815150941 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6b40051d0 0x7ff6c41387e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.644+0000 7ff6cad1d640 1 -- 172.21.15.116:0/815150941 >> 172.21.15.116:0/815150941 conn(0x7ff6c406a4c0 msgr2=0x7ff6c40ce210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:04.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.644+0000 7ff6cad1d640 1 -- 172.21.15.116:0/815150941 shutdown_connections 2023-11-28T17:10:04.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.645+0000 7ff6cad1d640 1 -- 172.21.15.116:0/815150941 wait complete. 2023-11-28T17:10:04.663 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:10:04.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.862+0000 7ff2edb4c640 1 Processor -- start 2023-11-28T17:10:04.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.863+0000 7ff2edb4c640 1 -- start start 2023-11-28T17:10:04.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.864+0000 7ff2edb4c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff2e815f7b0 0x7ff2e8071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:04.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.864+0000 7ff2edb4c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff2e80721b0 0x7ff2e806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:04.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.864+0000 7ff2edb4c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e815fc10 con 0x7ff2e8144430 2023-11-28T17:10:04.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.864+0000 7ff2edb4c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e8144810 con 0x7ff2e815f7b0 2023-11-28T17:10:04.866 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.864+0000 7ff2edb4c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e8070bd0 con 0x7ff2e80721b0 2023-11-28T17:10:04.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.865+0000 7ff2e6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff2e815f7b0 0x7ff2e8071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:04.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.865+0000 7ff2e6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff2e815f7b0 0x7ff2e8071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60214/0 (socket says 172.21.15.116:60214) 2023-11-28T17:10:04.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.865+0000 7ff2e6ffd640 1 -- 172.21.15.116:0/2445918881 learned_addr learned my addr 172.21.15.116:0/2445918881 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:04.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.865+0000 7ff2e6ffd640 1 -- 172.21.15.116:0/2445918881 >> v2:172.21.15.156:3300/0 conn(0x7ff2e80721b0 msgr2=0x7ff2e806c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:04.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.865+0000 7ff2e6ffd640 1 --2- 172.21.15.116:0/2445918881 >> v2:172.21.15.156:3300/0 conn(0x7ff2e80721b0 0x7ff2e806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.865+0000 7ff2e6ffd640 1 -- 172.21.15.116:0/2445918881 >> v1:172.21.15.156:6789/0 conn(0x7ff2e8144430 legacy=0x7ff2e80703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:04.867 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.865+0000 7ff2e6ffd640 1 -- 172.21.15.116:0/2445918881 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e8070eb0 con 0x7ff2e815f7b0 2023-11-28T17:10:04.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.866+0000 7ff2e6ffd640 1 --2- 172.21.15.116:0/2445918881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e815f7b0 0x7ff2e8071c70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8067310 tx=0x7ff2d002b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7af1f3e359ecd2c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:04.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.866+0000 7ff2e67fc640 1 -- 172.21.15.116:0/2445918881 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d0031890 con 0x7ff2e815f7b0 2023-11-28T17:10:04.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.866+0000 7ff2e67fc640 1 -- 172.21.15.116:0/2445918881 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d00319f0 con 0x7ff2e815f7b0 2023-11-28T17:10:04.868 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.867+0000 7ff2e67fc640 1 -- 172.21.15.116:0/2445918881 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d0031d80 con 0x7ff2e815f7b0 2023-11-28T17:10:04.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.867+0000 7ff2edb4c640 1 -- 172.21.15.116:0/2445918881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e815f7b0 msgr2=0x7ff2e8071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:04.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.867+0000 7ff2edb4c640 1 --2- 172.21.15.116:0/2445918881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e815f7b0 0x7ff2e8071c70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8067310 tx=0x7ff2d002b930 comp rx=0 tx=0).stop 2023-11-28T17:10:04.869 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.867+0000 7ff2edb4c640 1 -- 172.21.15.116:0/2445918881 shutdown_connections 2023-11-28T17:10:04.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.867+0000 7ff2edb4c640 1 --2- 172.21.15.116:0/2445918881 >> v2:172.21.15.156:3300/0 conn(0x7ff2e80721b0 0x7ff2e806c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.867+0000 7ff2edb4c640 1 --2- 172.21.15.116:0/2445918881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e815f7b0 0x7ff2e8071c70 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.867+0000 7ff2edb4c640 1 -- 172.21.15.116:0/2445918881 >> 172.21.15.116:0/2445918881 conn(0x7ff2e806a4c0 msgr2=0x7ff2e806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:04.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.867+0000 7ff2edb4c640 1 -- 172.21.15.116:0/2445918881 shutdown_connections 2023-11-28T17:10:04.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.867+0000 7ff2edb4c640 1 -- 172.21.15.116:0/2445918881 wait complete. 2023-11-28T17:10:04.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.869+0000 7ff2edb4c640 1 Processor -- start 2023-11-28T17:10:04.872 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.869+0000 7ff2edb4c640 1 -- start start 2023-11-28T17:10:04.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.870+0000 7ff2edb4c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2e80721b0 0x7ff2e80a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:04.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.870+0000 7ff2edb4c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e8144430 0x7ff2e81412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:04.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.870+0000 7ff2edb4c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff2e815f7b0 0x7ff2e8140740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:04.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.870+0000 7ff2edb4c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e81e8340 con 0x7ff2e8144430 2023-11-28T17:10:04.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.870+0000 7ff2edb4c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e81e7730 con 0x7ff2e80721b0 2023-11-28T17:10:04.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.870+0000 7ff2edb4c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e81e7a10 con 0x7ff2e815f7b0 2023-11-28T17:10:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.870+0000 7ff2e6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e8144430 0x7ff2e81412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.870+0000 7ff2e6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e8144430 0x7ff2e81412f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60228/0 (socket says 172.21.15.116:60228) 2023-11-28T17:10:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.870+0000 7ff2e6ffd640 1 -- 172.21.15.116:0/1207842791 learned_addr learned my addr 172.21.15.116:0/1207842791 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.871+0000 7ff2e6ffd640 1 -- 172.21.15.116:0/1207842791 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff2e815f7b0 msgr2=0x7ff2e8140740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.871+0000 7ff2e6ffd640 1 --2- 172.21.15.116:0/1207842791 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff2e815f7b0 0x7ff2e8140740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.871+0000 7ff2e6ffd640 1 -- 172.21.15.116:0/1207842791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2e80721b0 msgr2=0x7ff2e80a9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.871+0000 7ff2e6ffd640 1 --2- 172.21.15.116:0/1207842791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2e80721b0 0x7ff2e80a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.871+0000 7ff2e6ffd640 1 -- 172.21.15.116:0/1207842791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d00094d0 con 0x7ff2e8144430 2023-11-28T17:10:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.871+0000 7ff2e6ffd640 1 --2- 172.21.15.116:0/1207842791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e8144430 0x7ff2e81412f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff2d00090b0 tx=0x7ff2d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:04.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.871+0000 7ff2dffff640 1 -- 172.21.15.116:0/1207842791 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d0031820 con 0x7ff2e8144430 2023-11-28T17:10:04.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.871+0000 7ff2dffff640 1 -- 172.21.15.116:0/1207842791 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d00443f0 con 0x7ff2e8144430 2023-11-28T17:10:04.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.871+0000 7ff2edb4c640 1 -- 172.21.15.116:0/1207842791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e81e7cf0 con 0x7ff2e8144430 2023-11-28T17:10:04.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.871+0000 7ff2dffff640 1 -- 172.21.15.116:0/1207842791 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d00447a0 con 0x7ff2e8144430 2023-11-28T17:10:04.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.871+0000 7ff2edb4c640 1 -- 172.21.15.116:0/1207842791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e8141e70 con 0x7ff2e8144430 2023-11-28T17:10:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.872+0000 7ff2edb4c640 1 -- 172.21.15.116:0/1207842791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e80693d0 con 0x7ff2e8144430 2023-11-28T17:10:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.873+0000 7ff2dffff640 1 -- 172.21.15.116:0/1207842791 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff2d004d450 con 0x7ff2e8144430 2023-11-28T17:10:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.874+0000 7ff2dffff640 1 --2- 172.21.15.116:0/1207842791 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff2c806b2a0 0x7ff2c806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.874+0000 7ff2dffff640 1 -- 172.21.15.116:0/1207842791 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff2d00ac450 con 0x7ff2e8144430 2023-11-28T17:10:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.874+0000 7ff2e77fe640 1 --2- 172.21.15.116:0/1207842791 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff2c806b2a0 0x7ff2c806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:04.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.875+0000 7ff2e77fe640 1 --2- 172.21.15.116:0/1207842791 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff2c806b2a0 0x7ff2c806d760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8001c60 tx=0x7ff2d8007d10 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:04.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:04.878+0000 7ff2dffff640 1 -- 172.21.15.116:0/1207842791 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff2d0077290 con 0x7ff2e8144430 2023-11-28T17:10:05.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.064+0000 7ff2edb4c640 1 -- 172.21.15.116:0/1207842791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2e806d100 con 0x7ff2e8144430 2023-11-28T17:10:05.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.066+0000 7ff2dffff640 1 -- 172.21.15.116:0/1207842791 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16570 (secure 0 0 0) 0x7ff2d007a8b0 con 0x7ff2e8144430 2023-11-28T17:10:05.071 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:05.071 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:43.049465+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:10:05.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.068+0000 7ff2edb4c640 1 -- 172.21.15.116:0/1207842791 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff2c806b2a0 msgr2=0x7ff2c806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:05.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.068+0000 7ff2edb4c640 1 --2- 172.21.15.116:0/1207842791 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff2c806b2a0 0x7ff2c806d760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8001c60 tx=0x7ff2d8007d10 comp rx=0 tx=0).stop 2023-11-28T17:10:05.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.068+0000 7ff2edb4c640 1 -- 172.21.15.116:0/1207842791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e8144430 msgr2=0x7ff2e81412f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:05.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.068+0000 7ff2edb4c640 1 --2- 172.21.15.116:0/1207842791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e8144430 0x7ff2e81412f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff2d00090b0 tx=0x7ff2d00046c0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.069+0000 7ff2edb4c640 1 -- 172.21.15.116:0/1207842791 shutdown_connections 2023-11-28T17:10:05.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.069+0000 7ff2edb4c640 1 --2- 172.21.15.116:0/1207842791 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff2c806b2a0 0x7ff2c806d760 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.069+0000 7ff2edb4c640 1 --2- 172.21.15.116:0/1207842791 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff2e815f7b0 0x7ff2e8140740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.069+0000 7ff2edb4c640 1 --2- 172.21.15.116:0/1207842791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e8144430 0x7ff2e81412f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.069+0000 7ff2edb4c640 1 --2- 172.21.15.116:0/1207842791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2e80721b0 0x7ff2e80a9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.069+0000 7ff2edb4c640 1 -- 172.21.15.116:0/1207842791 >> 172.21.15.116:0/1207842791 conn(0x7ff2e806a4c0 msgr2=0x7ff2e80703f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:05.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.069+0000 7ff2edb4c640 1 -- 172.21.15.116:0/1207842791 shutdown_connections 2023-11-28T17:10:05.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.069+0000 7ff2edb4c640 1 -- 172.21.15.116:0/1207842791 wait complete. 2023-11-28T17:10:05.088 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T17:10:05.088 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:10:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.290+0000 7fa14cc56640 1 Processor -- start 2023-11-28T17:10:05.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.291+0000 7fa14cc56640 1 -- start start 2023-11-28T17:10:05.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.292+0000 7fa14cc56640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa1481401c0 0x7fa148071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:05.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.292+0000 7fa14cc56640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa148071710 0x7fa14806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:05.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.292+0000 7fa14cc56640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1481405a0 con 0x7fa14806c790 2023-11-28T17:10:05.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.292+0000 7fa14cc56640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa14806cc00 con 0x7fa1481401c0 2023-11-28T17:10:05.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.292+0000 7fa14cc56640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa148140880 con 0x7fa148071710 2023-11-28T17:10:05.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.293+0000 7fa14659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa1481401c0 0x7fa148071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:05.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.293+0000 7fa14659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa1481401c0 0x7fa148071eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60236/0 (socket says 172.21.15.116:60236) 2023-11-28T17:10:05.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.293+0000 7fa14659c640 1 -- 172.21.15.116:0/1754449708 learned_addr learned my addr 172.21.15.116:0/1754449708 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.293+0000 7fa13ffff640 1 --2- 172.21.15.116:0/1754449708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa148071710 0x7fa14806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.293+0000 7fa14659c640 1 -- 172.21.15.116:0/1754449708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa148071710 msgr2=0x7fa14806c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.293+0000 7fa14659c640 1 --2- 172.21.15.116:0/1754449708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa148071710 0x7fa14806c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.293+0000 7fa14659c640 1 -- 172.21.15.116:0/1754449708 >> v1:172.21.15.156:6789/0 conn(0x7fa14806c790 legacy=0x7fa1480703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.293+0000 7fa14659c640 1 -- 172.21.15.116:0/1754449708 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14815db00 con 0x7fa1481401c0 2023-11-28T17:10:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.293+0000 7fa13ffff640 1 --2- 172.21.15.116:0/1754449708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa148071710 0x7fa14806c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:05.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.295+0000 7fa14659c640 1 --2- 172.21.15.116:0/1754449708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1481401c0 0x7fa148071eb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa138009870 tx=0x7fa13802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28286bdfcae33998 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:05.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.295+0000 7fa145d9b640 1 -- 172.21.15.116:0/1754449708 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa138030840 con 0x7fa1481401c0 2023-11-28T17:10:05.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.295+0000 7fa145d9b640 1 -- 172.21.15.116:0/1754449708 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1380309a0 con 0x7fa1481401c0 2023-11-28T17:10:05.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.295+0000 7fa145d9b640 1 -- 172.21.15.116:0/1754449708 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa138030d30 con 0x7fa1481401c0 2023-11-28T17:10:05.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.296+0000 7fa14cc56640 1 -- 172.21.15.116:0/1754449708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1481401c0 msgr2=0x7fa148071eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:05.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.296+0000 7fa14cc56640 1 --2- 172.21.15.116:0/1754449708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1481401c0 0x7fa148071eb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa138009870 tx=0x7fa13802b980 comp rx=0 tx=0).stop 2023-11-28T17:10:05.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.296+0000 7fa14cc56640 1 -- 172.21.15.116:0/1754449708 shutdown_connections 2023-11-28T17:10:05.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.296+0000 7fa14cc56640 1 --2- 172.21.15.116:0/1754449708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa148071710 0x7fa14806c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.296+0000 7fa14cc56640 1 --2- 172.21.15.116:0/1754449708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1481401c0 0x7fa148071eb0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.296+0000 7fa14cc56640 1 -- 172.21.15.116:0/1754449708 >> 172.21.15.116:0/1754449708 conn(0x7fa14806a4c0 msgr2=0x7fa14806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:05.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.296+0000 7fa14cc56640 1 -- 172.21.15.116:0/1754449708 shutdown_connections 2023-11-28T17:10:05.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.296+0000 7fa14cc56640 1 -- 172.21.15.116:0/1754449708 wait complete. 2023-11-28T17:10:05.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.297+0000 7fa14cc56640 1 Processor -- start 2023-11-28T17:10:05.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.298+0000 7fa14cc56640 1 -- start start 2023-11-28T17:10:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.298+0000 7fa14cc56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa14806c790 0x7fa1481306f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.298+0000 7fa14cc56640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa148071710 0x7fa14815ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.299+0000 7fa14cc56640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1481401c0 0x7fa148141f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.299+0000 7fa14cc56640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa148131950 con 0x7fa14806c790 2023-11-28T17:10:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.299+0000 7fa14cc56640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa148130cd0 con 0x7fa148071710 2023-11-28T17:10:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.299+0000 7fa14cc56640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa148130fb0 con 0x7fa1481401c0 2023-11-28T17:10:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.299+0000 7fa14659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa14806c790 0x7fa1481306f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.299+0000 7fa14659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa14806c790 0x7fa1481306f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60246/0 (socket says 172.21.15.116:60246) 2023-11-28T17:10:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.299+0000 7fa14659c640 1 -- 172.21.15.116:0/1238375275 learned_addr learned my addr 172.21.15.116:0/1238375275 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.299+0000 7fa14659c640 1 -- 172.21.15.116:0/1238375275 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1481401c0 msgr2=0x7fa148141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.299+0000 7fa14659c640 1 --2- 172.21.15.116:0/1238375275 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1481401c0 0x7fa148141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.299+0000 7fa14659c640 1 -- 172.21.15.116:0/1238375275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa148071710 msgr2=0x7fa14815ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.299+0000 7fa14659c640 1 --2- 172.21.15.116:0/1238375275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa148071710 0x7fa14815ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.299+0000 7fa14659c640 1 -- 172.21.15.116:0/1238375275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1380094d0 con 0x7fa14806c790 2023-11-28T17:10:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.299+0000 7fa14659c640 1 --2- 172.21.15.116:0/1238375275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa14806c790 0x7fa1481306f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa138000c00 tx=0x7fa138004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.300+0000 7fa13effd640 1 -- 172.21.15.116:0/1238375275 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa138030810 con 0x7fa14806c790 2023-11-28T17:10:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.300+0000 7fa13effd640 1 -- 172.21.15.116:0/1238375275 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa138043400 con 0x7fa14806c790 2023-11-28T17:10:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.300+0000 7fa14cc56640 1 -- 172.21.15.116:0/1238375275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa148131290 con 0x7fa14806c790 2023-11-28T17:10:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.300+0000 7fa13effd640 1 -- 172.21.15.116:0/1238375275 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1380437d0 con 0x7fa14806c790 2023-11-28T17:10:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.300+0000 7fa14cc56640 1 -- 172.21.15.116:0/1238375275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14815b160 con 0x7fa14806c790 2023-11-28T17:10:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.301+0000 7fa14cc56640 1 -- 172.21.15.116:0/1238375275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa148131a90 con 0x7fa14806c790 2023-11-28T17:10:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.302+0000 7fa13effd640 1 -- 172.21.15.116:0/1238375275 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa138043930 con 0x7fa14806c790 2023-11-28T17:10:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.303+0000 7fa13effd640 1 --2- 172.21.15.116:0/1238375275 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa12806b2a0 0x7fa12806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.303+0000 7fa13effd640 1 -- 172.21.15.116:0/1238375275 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa13803e070 con 0x7fa14806c790 2023-11-28T17:10:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.303+0000 7fa13ffff640 1 --2- 172.21.15.116:0/1238375275 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa12806b2a0 0x7fa12806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.304+0000 7fa13ffff640 1 --2- 172.21.15.116:0/1238375275 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa12806b2a0 0x7fa12806d760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa130006fd0 tx=0x7fa130008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.308+0000 7fa13effd640 1 -- 172.21.15.116:0/1238375275 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa138077db0 con 0x7fa14806c790 2023-11-28T17:10:05.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.520+0000 7fa14cc56640 1 -- 172.21.15.116:0/1238375275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1480ae840 con 0x7fa14806c790 2023-11-28T17:10:05.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.521+0000 7fa13effd640 1 -- 172.21.15.116:0/1238375275 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7fa138034090 con 0x7fa14806c790 2023-11-28T17:10:05.522 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:05.523 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3138968863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":3138968863},{"type":"v1","addr":"172.21.15.156:6835","nonce":3138968863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/733981412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":733981412},{"type":"v1","addr":"172.21.15.116:6836","nonce":733981412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2751219705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2751219705},{"type":"v1","addr":"172.21.15.116:6837","nonce":2751219705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:09:00.673159+0000","modified":"2023-11-28T17:09:03.771842+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6834/3268354971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3268354971},{"type":"v1","addr":"172.21.15.156:6834","nonce":3268354971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T17:10:05.523 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2023-11-28T17:10:05.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.524+0000 7fa14cc56640 1 -- 172.21.15.116:0/1238375275 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa12806b2a0 msgr2=0x7fa12806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:05.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.524+0000 7fa14cc56640 1 --2- 172.21.15.116:0/1238375275 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa12806b2a0 0x7fa12806d760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa130006fd0 tx=0x7fa130008040 comp rx=0 tx=0).stop 2023-11-28T17:10:05.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.524+0000 7fa14cc56640 1 -- 172.21.15.116:0/1238375275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa14806c790 msgr2=0x7fa1481306f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:05.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.524+0000 7fa14cc56640 1 --2- 172.21.15.116:0/1238375275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa14806c790 0x7fa1481306f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa138000c00 tx=0x7fa138004730 comp rx=0 tx=0).stop 2023-11-28T17:10:05.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.524+0000 7fa14cc56640 1 -- 172.21.15.116:0/1238375275 shutdown_connections 2023-11-28T17:10:05.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.524+0000 7fa14cc56640 1 --2- 172.21.15.116:0/1238375275 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa12806b2a0 0x7fa12806d760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.524+0000 7fa14cc56640 1 --2- 172.21.15.116:0/1238375275 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa1481401c0 0x7fa148141f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.524+0000 7fa14cc56640 1 --2- 172.21.15.116:0/1238375275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa148071710 0x7fa14815ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.524+0000 7fa14cc56640 1 --2- 172.21.15.116:0/1238375275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa14806c790 0x7fa1481306f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.524+0000 7fa14cc56640 1 -- 172.21.15.116:0/1238375275 >> 172.21.15.116:0/1238375275 conn(0x7fa14806a4c0 msgr2=0x7fa1480ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:05.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.524+0000 7fa14cc56640 1 -- 172.21.15.116:0/1238375275 shutdown_connections 2023-11-28T17:10:05.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.524+0000 7fa14cc56640 1 -- 172.21.15.116:0/1238375275 wait complete. 2023-11-28T17:10:05.542 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:10:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.750+0000 7f3ee2378640 1 Processor -- start 2023-11-28T17:10:05.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.750+0000 7f3ee2378640 1 -- start start 2023-11-28T17:10:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.751+0000 7f3ee2378640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3edc06c960 0x7f3edc167e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.751+0000 7f3ee2378640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3edc1405a0 con 0x7f3edc071710 2023-11-28T17:10:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.751+0000 7f3ee2378640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3edc140880 con 0x7f3edc1401c0 2023-11-28T17:10:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.751+0000 7f3ee2378640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3edc15db00 con 0x7f3edc06c960 2023-11-28T17:10:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.751+0000 7f3ee0915640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3edc06c960 0x7f3edc167e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:05.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.752+0000 7f3ee0915640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3edc06c960 0x7f3edc167e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60250/0 (socket says 172.21.15.116:60250) 2023-11-28T17:10:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.752+0000 7f3ee0915640 1 -- 172.21.15.116:0/4213438671 learned_addr learned my addr 172.21.15.116:0/4213438671 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:05.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.752+0000 7f3ed3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3edc071710 0x7f3edc06c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57414/0 (socket says 172.21.15.116:57414) 2023-11-28T17:10:05.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.752+0000 7f3ee0915640 1 -- 172.21.15.116:0/4213438671 >> v1:172.21.15.156:6789/0 conn(0x7f3edc1401c0 legacy=0x7f3edc071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:05.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.752+0000 7f3ee0915640 1 -- 172.21.15.116:0/4213438671 >> v1:172.21.15.116:6789/0 conn(0x7f3edc071710 legacy=0x7f3edc06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:05.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.752+0000 7f3ee0915640 1 -- 172.21.15.116:0/4213438671 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc15dde0 con 0x7f3edc06c960 2023-11-28T17:10:05.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.752+0000 7f3edb7fe640 1 -- 172.21.15.116:0/4213438671 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169177401 0 0) 0x7f3edc1405a0 con 0x7f3edc071710 2023-11-28T17:10:05.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.753+0000 7f3ee0915640 1 --2- 172.21.15.116:0/4213438671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3edc06c960 0x7f3edc167e20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4009e50 tx=0x7f3ed402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee5e4451245fb20 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:05.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.753+0000 7f3edb7fe640 1 -- 172.21.15.116:0/4213438671 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed402cbb0 con 0x7f3edc06c960 2023-11-28T17:10:05.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.753+0000 7f3edb7fe640 1 -- 172.21.15.116:0/4213438671 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed402cd10 con 0x7f3edc06c960 2023-11-28T17:10:05.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.753+0000 7f3edb7fe640 1 -- 172.21.15.116:0/4213438671 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4034610 con 0x7f3edc06c960 2023-11-28T17:10:05.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.754+0000 7f3ee2378640 1 -- 172.21.15.116:0/4213438671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3edc06c960 msgr2=0x7f3edc167e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:05.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.754+0000 7f3ee2378640 1 --2- 172.21.15.116:0/4213438671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3edc06c960 0x7f3edc167e20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4009e50 tx=0x7f3ed402c010 comp rx=0 tx=0).stop 2023-11-28T17:10:05.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.754+0000 7f3ee2378640 1 -- 172.21.15.116:0/4213438671 shutdown_connections 2023-11-28T17:10:05.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.754+0000 7f3ee2378640 1 --2- 172.21.15.116:0/4213438671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3edc06c960 0x7f3edc167e20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.754+0000 7f3ee2378640 1 -- 172.21.15.116:0/4213438671 >> 172.21.15.116:0/4213438671 conn(0x7f3edc06a4c0 msgr2=0x7f3edc06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.754+0000 7f3ee2378640 1 -- 172.21.15.116:0/4213438671 shutdown_connections 2023-11-28T17:10:05.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.754+0000 7f3ee2378640 1 -- 172.21.15.116:0/4213438671 wait complete. 2023-11-28T17:10:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.756+0000 7f3ee2378640 1 Processor -- start 2023-11-28T17:10:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.756+0000 7f3ee2378640 1 -- start start 2023-11-28T17:10:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.757+0000 7f3ee2378640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3edc06c960 0x7f3edc15a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.757+0000 7f3ee2378640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3edc071710 0x7f3edc13d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:05.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.757+0000 7f3ee2378640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3edc1401c0 0x7f3edc141f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.757+0000 7f3ee2378640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc15aea0 con 0x7f3edc071710 2023-11-28T17:10:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.757+0000 7f3ee2378640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc141a70 con 0x7f3edc1401c0 2023-11-28T17:10:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.757+0000 7f3ee2378640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc140e60 con 0x7f3edc06c960 2023-11-28T17:10:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.757+0000 7f3ed3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3edc071710 0x7f3edc13d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.757+0000 7f3ed3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3edc071710 0x7f3edc13d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60264/0 (socket says 172.21.15.116:60264) 2023-11-28T17:10:05.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.757+0000 7f3ed3fff640 1 -- 172.21.15.116:0/3184082040 learned_addr learned my addr 172.21.15.116:0/3184082040 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.757+0000 7f3ed3fff640 1 -- 172.21.15.116:0/3184082040 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3edc06c960 msgr2=0x7f3edc15a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.758+0000 7f3ed3fff640 1 --2- 172.21.15.116:0/3184082040 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3edc06c960 0x7f3edc15a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.758+0000 7f3ed3fff640 1 -- 172.21.15.116:0/3184082040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3edc1401c0 msgr2=0x7f3edc141f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.758+0000 7f3ed3fff640 1 --2- 172.21.15.116:0/3184082040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3edc1401c0 0x7f3edc141f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.758+0000 7f3ed3fff640 1 -- 172.21.15.116:0/3184082040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed4009b30 con 0x7f3edc071710 2023-11-28T17:10:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.758+0000 7f3ed3fff640 1 --2- 172.21.15.116:0/3184082040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3edc071710 0x7f3edc13d910 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400c4d0 tx=0x7f3ec400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.758+0000 7f3ed97fa640 1 -- 172.21.15.116:0/3184082040 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec40074e0 con 0x7f3edc071710 2023-11-28T17:10:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.758+0000 7f3ed97fa640 1 -- 172.21.15.116:0/3184082040 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec4007640 con 0x7f3edc071710 2023-11-28T17:10:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.758+0000 7f3ed97fa640 1 -- 172.21.15.116:0/3184082040 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4004a20 con 0x7f3edc071710 2023-11-28T17:10:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.758+0000 7f3ee2378640 1 -- 172.21.15.116:0/3184082040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc141140 con 0x7f3edc071710 2023-11-28T17:10:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.758+0000 7f3ee2378640 1 -- 172.21.15.116:0/3184082040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3edc141700 con 0x7f3edc071710 2023-11-28T17:10:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.759+0000 7f3ee2378640 1 -- 172.21.15.116:0/3184082040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3edc0693d0 con 0x7f3edc071710 2023-11-28T17:10:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.760+0000 7f3ed97fa640 1 -- 172.21.15.116:0/3184082040 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3ec4004b80 con 0x7f3edc071710 2023-11-28T17:10:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.761+0000 7f3ed97fa640 1 --2- 172.21.15.116:0/3184082040 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3ebc06b2a0 0x7f3ebc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.761+0000 7f3ed97fa640 1 -- 172.21.15.116:0/3184082040 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3ec4088f00 con 0x7f3edc071710 2023-11-28T17:10:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.762+0000 7f3edbfff640 1 --2- 172.21.15.116:0/3184082040 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3ebc06b2a0 0x7f3ebc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:05.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.762+0000 7f3edbfff640 1 --2- 172.21.15.116:0/3184082040 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3ebc06b2a0 0x7f3ebc06d760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc000f60 tx=0x7f3ecc0079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.766+0000 7f3ed97fa640 1 -- 172.21.15.116:0/3184082040 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ec4053cb0 con 0x7f3edc071710 2023-11-28T17:10:05.995 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.994+0000 7f3ee2378640 1 -- 172.21.15.116:0/3184082040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3edc0ae5a0 con 0x7f3edc071710 2023-11-28T17:10:05.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.995+0000 7f3ed97fa640 1 -- 172.21.15.116:0/3184082040 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f3ec4018070 con 0x7f3edc071710 2023-11-28T17:10:05.997 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:05.997 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/3138968863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":3138968863},{"type":"v1","addr":"172.21.15.156:6835","nonce":3138968863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6836/733981412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":733981412},{"type":"v1","addr":"172.21.15.116:6836","nonce":733981412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2751219705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6835","nonce":2751219705},{"type":"v1","addr":"172.21.15.116:6837","nonce":2751219705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:09:00.673159+0000","modified":"2023-11-28T17:09:03.771842+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6834/3268354971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3268354971},{"type":"v1","addr":"172.21.15.156:6834","nonce":3268354971}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-28T17:10:05.997 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2023-11-28T17:10:05.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.998+0000 7f3ee2378640 1 -- 172.21.15.116:0/3184082040 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3ebc06b2a0 msgr2=0x7f3ebc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.998+0000 7f3ee2378640 1 --2- 172.21.15.116:0/3184082040 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3ebc06b2a0 0x7f3ebc06d760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc000f60 tx=0x7f3ecc0079b0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.998+0000 7f3ee2378640 1 -- 172.21.15.116:0/3184082040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3edc071710 msgr2=0x7f3edc13d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.998+0000 7f3ee2378640 1 --2- 172.21.15.116:0/3184082040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3edc071710 0x7f3edc13d910 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400c4d0 tx=0x7f3ec400c990 comp rx=0 tx=0).stop 2023-11-28T17:10:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.998+0000 7f3ee2378640 1 -- 172.21.15.116:0/3184082040 shutdown_connections 2023-11-28T17:10:06.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.998+0000 7f3ee2378640 1 --2- 172.21.15.116:0/3184082040 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3ebc06b2a0 0x7f3ebc06d760 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.998+0000 7f3ee2378640 1 --2- 172.21.15.116:0/3184082040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3edc1401c0 0x7f3edc141f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.998+0000 7f3ee2378640 1 --2- 172.21.15.116:0/3184082040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3edc071710 0x7f3edc13d910 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.998+0000 7f3ee2378640 1 --2- 172.21.15.116:0/3184082040 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3edc06c960 0x7f3edc15a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.998+0000 7f3ee2378640 1 -- 172.21.15.116:0/3184082040 >> 172.21.15.116:0/3184082040 conn(0x7f3edc06a4c0 msgr2=0x7f3edc0acd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:06.001 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.998+0000 7f3ee2378640 1 -- 172.21.15.116:0/3184082040 shutdown_connections 2023-11-28T17:10:06.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:05.998+0000 7f3ee2378640 1 -- 172.21.15.116:0/3184082040 wait complete. 2023-11-28T17:10:06.019 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:10:06.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.230+0000 7fd037cc9640 1 Processor -- start 2023-11-28T17:10:06.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.231+0000 7fd037cc9640 1 -- start start 2023-11-28T17:10:06.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.232+0000 7fd037cc9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03006c250 0x7fd03006c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.232+0000 7fd037cc9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03015fc10 con 0x7fd03015f7b0 2023-11-28T17:10:06.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.232+0000 7fd037cc9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd030070bd0 con 0x7fd030144430 2023-11-28T17:10:06.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.232+0000 7fd037cc9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd030144810 con 0x7fd03006c250 2023-11-28T17:10:06.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.232+0000 7fd035264640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd03015f7b0 0x7fd030071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57420/0 (socket says 172.21.15.116:57420) 2023-11-28T17:10:06.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.232+0000 7fd035264640 1 -- 172.21.15.116:0/2190501234 learned_addr learned my addr 172.21.15.116:0/2190501234 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:06.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd036266640 1 --2- 172.21.15.116:0/2190501234 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03006c250 0x7fd03006c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:06.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 -- 172.21.15.116:0/2190501234 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055534598 0 0) 0x7fd03015fc10 con 0x7fd03015f7b0 2023-11-28T17:10:06.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 -- 172.21.15.116:0/2190501234 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd01c003520 con 0x7fd03015f7b0 2023-11-28T17:10:06.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 -- 172.21.15.116:0/2190501234 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2885563474 0 0) 0x7fd01c003520 con 0x7fd03015f7b0 2023-11-28T17:10:06.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 -- 172.21.15.116:0/2190501234 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03006c250 msgr2=0x7fd03006c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:06.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 --2- 172.21.15.116:0/2190501234 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03006c250 0x7fd03006c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 -- 172.21.15.116:0/2190501234 >> v1:172.21.15.156:6789/0 conn(0x7fd030144430 legacy=0x7fd0300703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 -- 172.21.15.116:0/2190501234 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd030070eb0 con 0x7fd03015f7b0 2023-11-28T17:10:06.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 -- 172.21.15.116:0/2190501234 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fd024001670 con 0x7fd03015f7b0 2023-11-28T17:10:06.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 -- 172.21.15.116:0/2190501234 >> v1:172.21.15.116:6789/0 conn(0x7fd03015f7b0 legacy=0x7fd030071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 --2- 172.21.15.116:0/2190501234 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03006c250 0x7fd01c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 --2- 172.21.15.116:0/2190501234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd01c003f20 0x7fd01c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 --2- 172.21.15.116:0/2190501234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd01c004d00 0x7fd01c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 -- 172.21.15.116:0/2190501234 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd030144810 con 0x7fd01c004d00 2023-11-28T17:10:06.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 -- 172.21.15.116:0/2190501234 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c00ba60 con 0x7fd01c003f20 2023-11-28T17:10:06.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.233+0000 7fd034a63640 1 -- 172.21.15.116:0/2190501234 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c00bc20 con 0x7fd03006c250 2023-11-28T17:10:06.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.234+0000 7fd036266640 1 --2- 172.21.15.116:0/2190501234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd01c004d00 0x7fd01c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:06.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.234+0000 7fd036266640 1 -- 172.21.15.116:0/2190501234 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03006c250 msgr2=0x7fd01c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:06.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.234+0000 7fd036266640 1 --2- 172.21.15.116:0/2190501234 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd03006c250 0x7fd01c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.234+0000 7fd036266640 1 -- 172.21.15.116:0/2190501234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd01c003f20 msgr2=0x7fd01c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:06.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.234+0000 7fd036266640 1 --2- 172.21.15.116:0/2190501234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd01c003f20 0x7fd01c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.234+0000 7fd036266640 1 -- 172.21.15.116:0/2190501234 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c003520 con 0x7fd01c004d00 2023-11-28T17:10:06.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.235+0000 7fd036266640 1 --2- 172.21.15.116:0/2190501234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd01c004d00 0x7fd01c005180 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd028009fc0 tx=0x7fd01c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779baaeebb797905 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:06.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.235+0000 7fd034a63640 1 -- 172.21.15.116:0/2190501234 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02802cc40 con 0x7fd01c004d00 2023-11-28T17:10:06.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.235+0000 7fd034a63640 1 -- 172.21.15.116:0/2190501234 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02800f040 con 0x7fd01c004d00 2023-11-28T17:10:06.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.235+0000 7fd034a63640 1 -- 172.21.15.116:0/2190501234 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd028034440 con 0x7fd01c004d00 2023-11-28T17:10:06.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.236+0000 7fd037cc9640 1 -- 172.21.15.116:0/2190501234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd01c004d00 msgr2=0x7fd01c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.236+0000 7fd037cc9640 1 --2- 172.21.15.116:0/2190501234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd01c004d00 0x7fd01c005180 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd028009fc0 tx=0x7fd01c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.236+0000 7fd036266640 1 -- 172.21.15.116:0/2190501234 reap_dead start 2023-11-28T17:10:06.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.236+0000 7fd037cc9640 1 -- 172.21.15.116:0/2190501234 shutdown_connections 2023-11-28T17:10:06.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.236+0000 7fd037cc9640 1 -- 172.21.15.116:0/2190501234 >> 172.21.15.116:0/2190501234 conn(0x7fd03006a4c0 msgr2=0x7fd03006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:06.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.236+0000 7fd037cc9640 1 -- 172.21.15.116:0/2190501234 shutdown_connections 2023-11-28T17:10:06.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.236+0000 7fd037cc9640 1 -- 172.21.15.116:0/2190501234 wait complete. 2023-11-28T17:10:06.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.238+0000 7fd037cc9640 1 Processor -- start 2023-11-28T17:10:06.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.238+0000 7fd037cc9640 1 -- start start 2023-11-28T17:10:06.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd037cc9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd01c004d00 0x7fd0301f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd037cc9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd0300c9a90 0x7fd0301f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd037cc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0301f39f0 0x7fd030142940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd037cc9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0301394c0 con 0x7fd0301f39f0 2023-11-28T17:10:06.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd037cc9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0301397f0 con 0x7fd01c004d00 2023-11-28T17:10:06.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd037cc9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd030139a90 con 0x7fd0300c9a90 2023-11-28T17:10:06.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd036266640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0301f39f0 0x7fd030142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:06.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd036266640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0301f39f0 0x7fd030142940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60278/0 (socket says 172.21.15.116:60278) 2023-11-28T17:10:06.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd036266640 1 -- 172.21.15.116:0/3045930486 learned_addr learned my addr 172.21.15.116:0/3045930486 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd035264640 1 --2- 172.21.15.116:0/3045930486 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd0300c9a90 0x7fd0301f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd036266640 1 -- 172.21.15.116:0/3045930486 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd0300c9a90 msgr2=0x7fd0301f34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd036266640 1 --2- 172.21.15.116:0/3045930486 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd0300c9a90 0x7fd0301f34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd036266640 1 -- 172.21.15.116:0/3045930486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd01c004d00 msgr2=0x7fd0301f3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd036266640 1 --2- 172.21.15.116:0/3045930486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd01c004d00 0x7fd0301f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.239+0000 7fd036266640 1 -- 172.21.15.116:0/3045930486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd030139d70 con 0x7fd0301f39f0 2023-11-28T17:10:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.240+0000 7fd036266640 1 --2- 172.21.15.116:0/3045930486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0301f39f0 0x7fd030142940 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd028002230 tx=0x7fd028009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.240+0000 7fd0227fc640 1 -- 172.21.15.116:0/3045930486 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd028034cd0 con 0x7fd0301f39f0 2023-11-28T17:10:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.240+0000 7fd0227fc640 1 -- 172.21.15.116:0/3045930486 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd028034e30 con 0x7fd0301f39f0 2023-11-28T17:10:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.240+0000 7fd0227fc640 1 -- 172.21.15.116:0/3045930486 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd028003bb0 con 0x7fd0301f39f0 2023-11-28T17:10:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.240+0000 7fd037cc9640 1 -- 172.21.15.116:0/3045930486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03013a050 con 0x7fd0301f39f0 2023-11-28T17:10:06.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.240+0000 7fd037cc9640 1 -- 172.21.15.116:0/3045930486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0300c9f10 con 0x7fd0301f39f0 2023-11-28T17:10:06.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.241+0000 7fd037cc9640 1 -- 172.21.15.116:0/3045930486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd03013a570 con 0x7fd0301f39f0 2023-11-28T17:10:06.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.242+0000 7fd0227fc640 1 -- 172.21.15.116:0/3045930486 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fd028003d10 con 0x7fd0301f39f0 2023-11-28T17:10:06.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.243+0000 7fd0227fc640 1 --2- 172.21.15.116:0/3045930486 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd01406b2d0 0x7fd01406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.243+0000 7fd0227fc640 1 -- 172.21.15.116:0/3045930486 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd0280ab2e0 con 0x7fd0301f39f0 2023-11-28T17:10:06.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.244+0000 7fd035a65640 1 --2- 172.21.15.116:0/3045930486 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd01406b2d0 0x7fd01406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:06.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.244+0000 7fd035a65640 1 --2- 172.21.15.116:0/3045930486 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd01406b2d0 0x7fd01406d790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd02c001590 tx=0x7fd02c007990 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:06.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.248+0000 7fd0227fc640 1 -- 172.21.15.116:0/3045930486 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd03013a570 con 0x7fd0301f39f0 2023-11-28T17:10:06.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.439+0000 7fd037cc9640 1 -- 172.21.15.116:0/3045930486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd03013a570 con 0x7fd0301f39f0 2023-11-28T17:10:06.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.441+0000 7fd0227fc640 1 -- 172.21.15.116:0/3045930486 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16570 (secure 0 0 0) 0x7fd03013a570 con 0x7fd0301f39f0 2023-11-28T17:10:06.443 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:06.443 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:09:43.049465+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:08:59.158048+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-28T17:08:59.691319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.39","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:10:06.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.444+0000 7fd037cc9640 1 -- 172.21.15.116:0/3045930486 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd01406b2d0 msgr2=0x7fd01406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.444+0000 7fd037cc9640 1 --2- 172.21.15.116:0/3045930486 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd01406b2d0 0x7fd01406d790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd02c001590 tx=0x7fd02c007990 comp rx=0 tx=0).stop 2023-11-28T17:10:06.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.444+0000 7fd037cc9640 1 -- 172.21.15.116:0/3045930486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0301f39f0 msgr2=0x7fd030142940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.444+0000 7fd037cc9640 1 --2- 172.21.15.116:0/3045930486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0301f39f0 0x7fd030142940 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd028002230 tx=0x7fd028009750 comp rx=0 tx=0).stop 2023-11-28T17:10:06.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.444+0000 7fd037cc9640 1 -- 172.21.15.116:0/3045930486 shutdown_connections 2023-11-28T17:10:06.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.444+0000 7fd037cc9640 1 --2- 172.21.15.116:0/3045930486 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd01406b2d0 0x7fd01406d790 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.444+0000 7fd037cc9640 1 --2- 172.21.15.116:0/3045930486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0301f39f0 0x7fd030142940 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.444+0000 7fd037cc9640 1 --2- 172.21.15.116:0/3045930486 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd0300c9a90 0x7fd0301f34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.444+0000 7fd037cc9640 1 --2- 172.21.15.116:0/3045930486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd01c004d00 0x7fd0301f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.444+0000 7fd037cc9640 1 -- 172.21.15.116:0/3045930486 >> 172.21.15.116:0/3045930486 conn(0x7fd03006a4c0 msgr2=0x7fd03015cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:06.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.444+0000 7fd037cc9640 1 -- 172.21.15.116:0/3045930486 shutdown_connections 2023-11-28T17:10:06.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.445+0000 7fd037cc9640 1 -- 172.21.15.116:0/3045930486 wait complete. 2023-11-28T17:10:06.465 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T17:10:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.681+0000 7f01251d0640 1 Processor -- start 2023-11-28T17:10:06.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.681+0000 7f01251d0640 1 -- start start 2023-11-28T17:10:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.682+0000 7f01251d0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0120159f30 0x7f01200701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.682+0000 7f01251d0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01200cf2a0 0x7f01200ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.682+0000 7f01251d0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01200cd8e0 con 0x7f012015ae90 2023-11-28T17:10:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.682+0000 7f01251d0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0120070730 con 0x7f0120159f30 2023-11-28T17:10:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.682+0000 7f01251d0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0120070ee0 con 0x7f01200cf2a0 2023-11-28T17:10:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f012015ae90 0x7f01200ceb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57430/0 (socket says 172.21.15.116:57430) 2023-11-28T17:10:06.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011f7fe640 1 -- 172.21.15.116:0/1677615387 learned_addr learned my addr 172.21.15.116:0/1677615387 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f01249cf640 1 --2- 172.21.15.116:0/1677615387 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01200cf2a0 0x7f01200ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011effd640 1 -- 172.21.15.116:0/1677615387 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988941259 0 0) 0x7f01200cd8e0 con 0x7f012015ae90 2023-11-28T17:10:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011effd640 1 -- 172.21.15.116:0/1677615387 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f010c003520 con 0x7f012015ae90 2023-11-28T17:10:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011ffff640 1 --2- 172.21.15.116:0/1677615387 >> v2:172.21.15.156:3300/0 conn(0x7f0120159f30 0x7f01200701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:06.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011effd640 1 -- 172.21.15.116:0/1677615387 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1421008902 0 0) 0x7f010c003520 con 0x7f012015ae90 2023-11-28T17:10:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011effd640 1 -- 172.21.15.116:0/1677615387 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01200cf2a0 msgr2=0x7f01200ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011effd640 1 --2- 172.21.15.116:0/1677615387 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01200cf2a0 0x7f01200ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011effd640 1 -- 172.21.15.116:0/1677615387 >> v2:172.21.15.156:3300/0 conn(0x7f0120159f30 msgr2=0x7f01200701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011effd640 1 --2- 172.21.15.116:0/1677615387 >> v2:172.21.15.156:3300/0 conn(0x7f0120159f30 0x7f01200701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011effd640 1 -- 172.21.15.116:0/1677615387 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012015b2f0 con 0x7f012015ae90 2023-11-28T17:10:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011effd640 1 -- 172.21.15.116:0/1677615387 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f0108001670 con 0x7f012015ae90 2023-11-28T17:10:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011effd640 1 -- 172.21.15.116:0/1677615387 >> v1:172.21.15.116:6789/0 conn(0x7f012015ae90 legacy=0x7f01200ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011effd640 1 --2- 172.21.15.116:0/1677615387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c003970 0x7f010c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.683+0000 7f011effd640 1 --2- 172.21.15.116:0/1677615387 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01200cf2a0 0x7f010c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.684+0000 7f011effd640 1 --2- 172.21.15.116:0/1677615387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f010c004d00 0x7f010c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.684+0000 7f011effd640 1 -- 172.21.15.116:0/1677615387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0120070ee0 con 0x7f010c003970 2023-11-28T17:10:06.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.684+0000 7f011effd640 1 -- 172.21.15.116:0/1677615387 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010c00ba30 con 0x7f010c004d00 2023-11-28T17:10:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.684+0000 7f011effd640 1 -- 172.21.15.116:0/1677615387 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010c00bb70 con 0x7f01200cf2a0 2023-11-28T17:10:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.684+0000 7f011ffff640 1 --2- 172.21.15.116:0/1677615387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c003970 0x7f010c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.684+0000 7f011f7fe640 1 --2- 172.21.15.116:0/1677615387 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01200cf2a0 0x7f010c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.684+0000 7f011ffff640 1 -- 172.21.15.116:0/1677615387 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01200cf2a0 msgr2=0x7f010c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.684+0000 7f011ffff640 1 --2- 172.21.15.116:0/1677615387 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f01200cf2a0 0x7f010c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.684+0000 7f011ffff640 1 -- 172.21.15.116:0/1677615387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f010c004d00 msgr2=0x7f010c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.684+0000 7f011ffff640 1 --2- 172.21.15.116:0/1677615387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f010c004d00 0x7f010c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.684+0000 7f011ffff640 1 -- 172.21.15.116:0/1677615387 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010c003520 con 0x7f010c003970 2023-11-28T17:10:06.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.685+0000 7f011ffff640 1 --2- 172.21.15.116:0/1677615387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c003970 0x7f010c003d90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f011000ba60 tx=0x7f010c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269c0a700272ff47 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.685+0000 7f011effd640 1 -- 172.21.15.116:0/1677615387 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0110032680 con 0x7f010c003970 2023-11-28T17:10:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.685+0000 7f011effd640 1 -- 172.21.15.116:0/1677615387 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01100327e0 con 0x7f010c003970 2023-11-28T17:10:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.685+0000 7f011effd640 1 -- 172.21.15.116:0/1677615387 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0110032b70 con 0x7f010c003970 2023-11-28T17:10:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.686+0000 7f01251d0640 1 -- 172.21.15.116:0/1677615387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c003970 msgr2=0x7f010c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.686+0000 7f01251d0640 1 --2- 172.21.15.116:0/1677615387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c003970 0x7f010c003d90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f011000ba60 tx=0x7f010c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.686+0000 7f01249cf640 1 -- 172.21.15.116:0/1677615387 reap_dead start 2023-11-28T17:10:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.686+0000 7f01251d0640 1 -- 172.21.15.116:0/1677615387 shutdown_connections 2023-11-28T17:10:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.686+0000 7f01251d0640 1 -- 172.21.15.116:0/1677615387 >> 172.21.15.116:0/1677615387 conn(0x7f01200b0950 msgr2=0x7f01200b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.686+0000 7f01251d0640 1 -- 172.21.15.116:0/1677615387 shutdown_connections 2023-11-28T17:10:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.686+0000 7f01251d0640 1 -- 172.21.15.116:0/1677615387 wait complete. 2023-11-28T17:10:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.688+0000 7f01251d0640 1 Processor -- start 2023-11-28T17:10:06.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.688+0000 7f01251d0640 1 -- start start 2023-11-28T17:10:06.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.689+0000 7f01251d0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f010c003970 0x7f012009c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.689+0000 7f01251d0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f012013db90 0x7f012013c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.689+0000 7f01251d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012013c570 0x7f012013e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.689+0000 7f01251d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012013caf0 con 0x7f012013c570 2023-11-28T17:10:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.689+0000 7f01251d0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012013cdd0 con 0x7f012013db90 2023-11-28T17:10:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.689+0000 7f01251d0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012013d0b0 con 0x7f010c003970 2023-11-28T17:10:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.689+0000 7f01249cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012013c570 0x7f012013e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.689+0000 7f01249cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012013c570 0x7f012013e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60302/0 (socket says 172.21.15.116:60302) 2023-11-28T17:10:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.689+0000 7f01249cf640 1 -- 172.21.15.116:0/1507150872 learned_addr learned my addr 172.21.15.116:0/1507150872 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.690+0000 7f011ffff640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f010c003970 0x7f012009c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.690+0000 7f011f7fe640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f012013db90 0x7f012013c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:06.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.690+0000 7f01249cf640 1 -- 172.21.15.116:0/1507150872 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f010c003970 msgr2=0x7f012009c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.690+0000 7f01249cf640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f010c003970 0x7f012009c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.690+0000 7f01249cf640 1 -- 172.21.15.116:0/1507150872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f012013db90 msgr2=0x7f012013c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.690+0000 7f01249cf640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f012013db90 0x7f012013c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.690+0000 7f01249cf640 1 -- 172.21.15.116:0/1507150872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012013d390 con 0x7f012013c570 2023-11-28T17:10:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.690+0000 7f011ffff640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f010c003970 0x7f012009c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.690+0000 7f01249cf640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012013c570 0x7f012013e090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f012006e700 tx=0x7f011800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.690+0000 7f011cff9640 1 -- 172.21.15.116:0/1507150872 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118018070 con 0x7f012013c570 2023-11-28T17:10:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.690+0000 7f011cff9640 1 -- 172.21.15.116:0/1507150872 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011800eb90 con 0x7f012013c570 2023-11-28T17:10:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.690+0000 7f01251d0640 1 -- 172.21.15.116:0/1507150872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0120152960 con 0x7f012013c570 2023-11-28T17:10:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.690+0000 7f011cff9640 1 -- 172.21.15.116:0/1507150872 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118003990 con 0x7f012013c570 2023-11-28T17:10:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.690+0000 7f01251d0640 1 -- 172.21.15.116:0/1507150872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f012013e7b0 con 0x7f012013c570 2023-11-28T17:10:06.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.692+0000 7f01251d0640 1 -- 172.21.15.116:0/1507150872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012013d520 con 0x7f012013c570 2023-11-28T17:10:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.692+0000 7f011cff9640 1 -- 172.21.15.116:0/1507150872 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0118014420 con 0x7f012013c570 2023-11-28T17:10:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.693+0000 7f011cff9640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f010006b2d0 0x7f010006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:06.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.693+0000 7f011cff9640 1 -- 172.21.15.116:0/1507150872 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f01180899c0 con 0x7f012013c570 2023-11-28T17:10:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.693+0000 7f011ffff640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f010006b2d0 0x7f010006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:06.696 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.694+0000 7f011ffff640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f010006b2d0 0x7f010006d790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f011000ba30 tx=0x7f011003d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:06.699 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.698+0000 7f011cff9640 1 -- 172.21.15.116:0/1507150872 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01180548a0 con 0x7f012013c570 2023-11-28T17:10:06.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.909+0000 7f01251d0640 1 -- 172.21.15.116:0/1507150872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f012009d0b0 con 0x7f012013c570 2023-11-28T17:10:06.914 INFO:tasks.ceph.mon.a.smithi116.stderr:2023-11-28T17:10:06.910+0000 7f2312418640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T17:10:06.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.914+0000 7f011cff9640 1 -- 172.21.15.116:0/1507150872 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0118057ec0 con 0x7f012013c570 2023-11-28T17:10:06.915 INFO:teuthology.orchestra.run.smithi116.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T17:10:06.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.917+0000 7f01251d0640 1 -- 172.21.15.116:0/1507150872 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f010006b2d0 msgr2=0x7f010006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.917+0000 7f01251d0640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f010006b2d0 0x7f010006d790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f011000ba30 tx=0x7f011003d040 comp rx=0 tx=0).stop 2023-11-28T17:10:06.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.917+0000 7f01251d0640 1 -- 172.21.15.116:0/1507150872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012013c570 msgr2=0x7f012013e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:06.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.917+0000 7f01251d0640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012013c570 0x7f012013e090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f012006e700 tx=0x7f011800e4b0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.917+0000 7f01251d0640 1 -- 172.21.15.116:0/1507150872 shutdown_connections 2023-11-28T17:10:06.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.917+0000 7f01251d0640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f010006b2d0 0x7f010006d790 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.917+0000 7f01251d0640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f012013c570 0x7f012013e090 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.917+0000 7f01251d0640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f012013db90 0x7f012013c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.917+0000 7f01251d0640 1 --2- 172.21.15.116:0/1507150872 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f010c003970 0x7f012009c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:06.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.917+0000 7f01251d0640 1 -- 172.21.15.116:0/1507150872 >> 172.21.15.116:0/1507150872 conn(0x7f01200b0950 msgr2=0x7f012015a310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:06.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.918+0000 7f01251d0640 1 -- 172.21.15.116:0/1507150872 shutdown_connections 2023-11-28T17:10:06.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:06.918+0000 7f01251d0640 1 -- 172.21.15.116:0/1507150872 wait complete. 2023-11-28T17:10:06.938 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T17:10:06.942 INFO:tasks.ceph.mds.b.smithi156.stderr: -3525> 2023-11-28T17:09:07.201+0000 7f29f4a64640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T17:10:07.023 INFO:tasks.ceph.mds.b.smithi156.stdout:starting mds.b at 2023-11-28T17:10:07.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.148+0000 7fad3dac9640 1 Processor -- start 2023-11-28T17:10:07.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.148+0000 7fad3dac9640 1 -- start start 2023-11-28T17:10:07.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.149+0000 7fad3dac9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad38146540 0x7fad38160e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:07.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.149+0000 7fad3dac9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad380b5110 con 0x7fad38146a90 2023-11-28T17:10:07.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.149+0000 7fad3dac9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad380b2190 con 0x7fad38145ee0 2023-11-28T17:10:07.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.149+0000 7fad3dac9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad38161dc0 con 0x7fad38146540 2023-11-28T17:10:07.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.150+0000 7fad37fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fad38146a90 0x7fad38071030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57440/0 (socket says 172.21.15.116:57440) 2023-11-28T17:10:07.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.150+0000 7fad37fff640 1 -- 172.21.15.116:0/620760731 learned_addr learned my addr 172.21.15.116:0/620760731 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:07.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.150+0000 7fad36ffd640 1 -- 172.21.15.116:0/620760731 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654084920 0 0) 0x7fad380b5110 con 0x7fad38146a90 2023-11-28T17:10:07.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.150+0000 7fad36ffd640 1 -- 172.21.15.116:0/620760731 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad24003520 con 0x7fad38146a90 2023-11-28T17:10:07.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.150+0000 7fad2ffff640 1 --2- 172.21.15.116:0/620760731 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad38146540 0x7fad38160e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:07.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad36ffd640 1 -- 172.21.15.116:0/620760731 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1013135930 0 0) 0x7fad24003520 con 0x7fad38146a90 2023-11-28T17:10:07.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad36ffd640 1 -- 172.21.15.116:0/620760731 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad38146540 msgr2=0x7fad38160e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:07.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad36ffd640 1 --2- 172.21.15.116:0/620760731 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad38146540 0x7fad38160e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:07.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad36ffd640 1 -- 172.21.15.116:0/620760731 >> v1:172.21.15.156:6789/0 conn(0x7fad38145ee0 legacy=0x7fad380ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:07.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad36ffd640 1 -- 172.21.15.116:0/620760731 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad38161ae0 con 0x7fad38146a90 2023-11-28T17:10:07.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad36ffd640 1 -- 172.21.15.116:0/620760731 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fad300019f0 con 0x7fad38146a90 2023-11-28T17:10:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad36ffd640 1 -- 172.21.15.116:0/620760731 >> v1:172.21.15.116:6789/0 conn(0x7fad38146a90 legacy=0x7fad38071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad36ffd640 1 --2- 172.21.15.116:0/620760731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad240039a0 0x7fad24003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad36ffd640 1 --2- 172.21.15.116:0/620760731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad240047c0 0x7fad24004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad36ffd640 1 --2- 172.21.15.116:0/620760731 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad38146540 0x7fad24005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad36ffd640 1 -- 172.21.15.116:0/620760731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad38161dc0 con 0x7fad240039a0 2023-11-28T17:10:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad36ffd640 1 -- 172.21.15.116:0/620760731 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2400bb10 con 0x7fad240047c0 2023-11-28T17:10:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad36ffd640 1 -- 172.21.15.116:0/620760731 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2400bcd0 con 0x7fad38146540 2023-11-28T17:10:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad377fe640 1 --2- 172.21.15.116:0/620760731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad240039a0 0x7fad24003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.151+0000 7fad377fe640 1 -- 172.21.15.116:0/620760731 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad38146540 msgr2=0x7fad24005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.152+0000 7fad377fe640 1 --2- 172.21.15.116:0/620760731 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad38146540 0x7fad24005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.152+0000 7fad377fe640 1 -- 172.21.15.116:0/620760731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad240047c0 msgr2=0x7fad24004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:07.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.152+0000 7fad377fe640 1 --2- 172.21.15.116:0/620760731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad240047c0 0x7fad24004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:07.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.152+0000 7fad377fe640 1 -- 172.21.15.116:0/620760731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad24003520 con 0x7fad240039a0 2023-11-28T17:10:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.153+0000 7fad377fe640 1 --2- 172.21.15.116:0/620760731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad240039a0 0x7fad24003dc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fad28001750 tx=0x7fad240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a09f31a9647087cc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.153+0000 7fad36ffd640 1 -- 172.21.15.116:0/620760731 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad280052d0 con 0x7fad240039a0 2023-11-28T17:10:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.153+0000 7fad36ffd640 1 -- 172.21.15.116:0/620760731 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2802e050 con 0x7fad240039a0 2023-11-28T17:10:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.153+0000 7fad36ffd640 1 -- 172.21.15.116:0/620760731 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad280035f0 con 0x7fad240039a0 2023-11-28T17:10:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.153+0000 7fad3dac9640 1 -- 172.21.15.116:0/620760731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad240039a0 msgr2=0x7fad24003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.153+0000 7fad3dac9640 1 --2- 172.21.15.116:0/620760731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad240039a0 0x7fad24003dc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fad28001750 tx=0x7fad240057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.153+0000 7fad37fff640 1 -- 172.21.15.116:0/620760731 reap_dead start 2023-11-28T17:10:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.153+0000 7fad3dac9640 1 -- 172.21.15.116:0/620760731 shutdown_connections 2023-11-28T17:10:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.153+0000 7fad3dac9640 1 -- 172.21.15.116:0/620760731 >> 172.21.15.116:0/620760731 conn(0x7fad380b0cf0 msgr2=0x7fad380b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:07.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.154+0000 7fad3dac9640 1 -- 172.21.15.116:0/620760731 shutdown_connections 2023-11-28T17:10:07.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.154+0000 7fad3dac9640 1 -- 172.21.15.116:0/620760731 wait complete. 2023-11-28T17:10:07.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.155+0000 7fad3dac9640 1 Processor -- start 2023-11-28T17:10:07.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.155+0000 7fad3dac9640 1 -- start start 2023-11-28T17:10:07.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.156+0000 7fad3dac9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad240039a0 0x7fad381f2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:07.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.156+0000 7fad3dac9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad3813f8e0 0x7fad380c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:07.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.156+0000 7fad3dac9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad381f56f0 0x7fad380c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:07.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.156+0000 7fad3dac9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad380c4c00 con 0x7fad381f56f0 2023-11-28T17:10:07.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.156+0000 7fad3dac9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381427d0 con 0x7fad240039a0 2023-11-28T17:10:07.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.156+0000 7fad3dac9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad38142ab0 con 0x7fad3813f8e0 2023-11-28T17:10:07.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.157+0000 7fad37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad381f56f0 0x7fad380c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:07.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.157+0000 7fad37fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad381f56f0 0x7fad380c4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60314/0 (socket says 172.21.15.116:60314) 2023-11-28T17:10:07.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.157+0000 7fad37fff640 1 -- 172.21.15.116:0/1282533352 learned_addr learned my addr 172.21.15.116:0/1282533352 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:07.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.157+0000 7fad2ffff640 1 --2- 172.21.15.116:0/1282533352 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad3813f8e0 0x7fad380c46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:07.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.157+0000 7fad377fe640 1 --2- 172.21.15.116:0/1282533352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad240039a0 0x7fad381f2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:07.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.157+0000 7fad37fff640 1 -- 172.21.15.116:0/1282533352 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad3813f8e0 msgr2=0x7fad380c46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:07.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.157+0000 7fad37fff640 1 --2- 172.21.15.116:0/1282533352 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad3813f8e0 0x7fad380c46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:07.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.157+0000 7fad37fff640 1 -- 172.21.15.116:0/1282533352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad240039a0 msgr2=0x7fad381f2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:07.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.157+0000 7fad37fff640 1 --2- 172.21.15.116:0/1282533352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad240039a0 0x7fad381f2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:07.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.157+0000 7fad37fff640 1 -- 172.21.15.116:0/1282533352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad38142d90 con 0x7fad381f56f0 2023-11-28T17:10:07.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.157+0000 7fad37fff640 1 --2- 172.21.15.116:0/1282533352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad381f56f0 0x7fad380c4da0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fad38146e70 tx=0x7fad3000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:07.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.158+0000 7fad34ff9640 1 -- 172.21.15.116:0/1282533352 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad30007110 con 0x7fad381f56f0 2023-11-28T17:10:07.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.158+0000 7fad34ff9640 1 -- 172.21.15.116:0/1282533352 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad300072b0 con 0x7fad381f56f0 2023-11-28T17:10:07.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.158+0000 7fad34ff9640 1 -- 172.21.15.116:0/1282533352 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad30007660 con 0x7fad381f56f0 2023-11-28T17:10:07.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.158+0000 7fad3dac9640 1 -- 172.21.15.116:0/1282533352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad38143070 con 0x7fad381f56f0 2023-11-28T17:10:07.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.158+0000 7fad3dac9640 1 -- 172.21.15.116:0/1282533352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad38143630 con 0x7fad381f56f0 2023-11-28T17:10:07.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.159+0000 7fad3dac9640 1 -- 172.21.15.116:0/1282533352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad38143820 con 0x7fad381f56f0 2023-11-28T17:10:07.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.160+0000 7fad34ff9640 1 -- 172.21.15.116:0/1282533352 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fad300077c0 con 0x7fad381f56f0 2023-11-28T17:10:07.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.161+0000 7fad34ff9640 1 --2- 172.21.15.116:0/1282533352 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fad1806b2a0 0x7fad1806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:07.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.161+0000 7fad34ff9640 1 -- 172.21.15.116:0/1282533352 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fad30089530 con 0x7fad381f56f0 2023-11-28T17:10:07.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.161+0000 7fad377fe640 1 --2- 172.21.15.116:0/1282533352 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fad1806b2a0 0x7fad1806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:07.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.162+0000 7fad377fe640 1 --2- 172.21.15.116:0/1282533352 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fad1806b2a0 0x7fad1806d760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fad28001750 tx=0x7fad28009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:07.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.165+0000 7fad34ff9640 1 -- 172.21.15.116:0/1282533352 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad30054340 con 0x7fad381f56f0 2023-11-28T17:10:07.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.394+0000 7fad3dac9640 1 -- 172.21.15.116:0/1282533352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fad38069360 con 0x7fad381f56f0 2023-11-28T17:10:07.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.917+0000 7fad34ff9640 1 -- 172.21.15.116:0/1282533352 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7fad30057960 con 0x7fad381f56f0 2023-11-28T17:10:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.921+0000 7fad3dac9640 1 -- 172.21.15.116:0/1282533352 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fad1806b2a0 msgr2=0x7fad1806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.921+0000 7fad3dac9640 1 --2- 172.21.15.116:0/1282533352 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fad1806b2a0 0x7fad1806d760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fad28001750 tx=0x7fad28009030 comp rx=0 tx=0).stop 2023-11-28T17:10:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.921+0000 7fad3dac9640 1 -- 172.21.15.116:0/1282533352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad381f56f0 msgr2=0x7fad380c4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.921+0000 7fad3dac9640 1 --2- 172.21.15.116:0/1282533352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad381f56f0 0x7fad380c4da0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fad38146e70 tx=0x7fad3000ccd0 comp rx=0 tx=0).stop 2023-11-28T17:10:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.921+0000 7fad3dac9640 1 -- 172.21.15.116:0/1282533352 shutdown_connections 2023-11-28T17:10:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.921+0000 7fad3dac9640 1 --2- 172.21.15.116:0/1282533352 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fad1806b2a0 0x7fad1806d760 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.921+0000 7fad3dac9640 1 --2- 172.21.15.116:0/1282533352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad381f56f0 0x7fad380c4da0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:07.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.921+0000 7fad3dac9640 1 --2- 172.21.15.116:0/1282533352 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad3813f8e0 0x7fad380c46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:07.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.921+0000 7fad3dac9640 1 --2- 172.21.15.116:0/1282533352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad240039a0 0x7fad381f2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:07.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.921+0000 7fad3dac9640 1 -- 172.21.15.116:0/1282533352 >> 172.21.15.116:0/1282533352 conn(0x7fad380b0cf0 msgr2=0x7fad380ade60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:07.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.921+0000 7fad3dac9640 1 -- 172.21.15.116:0/1282533352 shutdown_connections 2023-11-28T17:10:07.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:07.922+0000 7fad3dac9640 1 -- 172.21.15.116:0/1282533352 wait complete. 2023-11-28T17:10:07.947 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T17:10:08.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.161+0000 7fe74c63d640 1 Processor -- start 2023-11-28T17:10:08.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.161+0000 7fe74c63d640 1 -- start start 2023-11-28T17:10:08.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.162+0000 7fe74c63d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe74415a4e0 0x7fe7440cb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:08.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.162+0000 7fe74c63d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe7440cbe40 0x7fe74406d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:08.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.162+0000 7fe74c63d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe74406dbc0 0x7fe74415dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:08.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.162+0000 7fe74c63d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe74406e050 con 0x7fe74406dbc0 2023-11-28T17:10:08.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.162+0000 7fe74c63d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7440cd1e0 con 0x7fe7440cbe40 2023-11-28T17:10:08.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.162+0000 7fe74c63d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7440cd7a0 con 0x7fe74415a4e0 2023-11-28T17:10:08.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.162+0000 7fe74abda640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe74406dbc0 0x7fe74415dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:08.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.162+0000 7fe74abda640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe74406dbc0 0x7fe74415dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60194/0 (socket says 172.21.15.116:60194) 2023-11-28T17:10:08.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.162+0000 7fe74abda640 1 -- 172.21.15.116:0/2613487886 learned_addr learned my addr 172.21.15.116:0/2613487886 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.163+0000 7fe74abda640 1 -- 172.21.15.116:0/2613487886 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe74415a4e0 msgr2=0x7fe7440cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.163+0000 7fe74abda640 1 --2- 172.21.15.116:0/2613487886 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe74415a4e0 0x7fe7440cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.163+0000 7fe74abda640 1 -- 172.21.15.116:0/2613487886 >> v2:172.21.15.156:3300/0 conn(0x7fe7440cbe40 msgr2=0x7fe74406d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.163+0000 7fe74abda640 1 --2- 172.21.15.116:0/2613487886 >> v2:172.21.15.156:3300/0 conn(0x7fe7440cbe40 0x7fe74406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.163+0000 7fe74abda640 1 -- 172.21.15.116:0/2613487886 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7440abf60 con 0x7fe74406dbc0 2023-11-28T17:10:08.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.164+0000 7fe74abda640 1 --2- 172.21.15.116:0/2613487886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe74406dbc0 0x7fe74415dec0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe744067fb0 tx=0x7fe73c02dfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1e6e84b15d8736 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.164+0000 7fe7493d7640 1 -- 172.21.15.116:0/2613487886 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c02eb60 con 0x7fe74406dbc0 2023-11-28T17:10:08.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.164+0000 7fe7493d7640 1 -- 172.21.15.116:0/2613487886 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73c02ecc0 con 0x7fe74406dbc0 2023-11-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.164+0000 7fe7493d7640 1 -- 172.21.15.116:0/2613487886 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c036610 con 0x7fe74406dbc0 2023-11-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.165+0000 7fe74c63d640 1 -- 172.21.15.116:0/2613487886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe74406dbc0 msgr2=0x7fe74415dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.165+0000 7fe74c63d640 1 --2- 172.21.15.116:0/2613487886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe74406dbc0 0x7fe74415dec0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe744067fb0 tx=0x7fe73c02dfc0 comp rx=0 tx=0).stop 2023-11-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.165+0000 7fe74c63d640 1 -- 172.21.15.116:0/2613487886 shutdown_connections 2023-11-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.165+0000 7fe74c63d640 1 --2- 172.21.15.116:0/2613487886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe74406dbc0 0x7fe74415dec0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.165+0000 7fe74c63d640 1 --2- 172.21.15.116:0/2613487886 >> v2:172.21.15.156:3300/0 conn(0x7fe7440cbe40 0x7fe74406d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.165+0000 7fe74c63d640 1 --2- 172.21.15.116:0/2613487886 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe74415a4e0 0x7fe7440cb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.165+0000 7fe74c63d640 1 -- 172.21.15.116:0/2613487886 >> 172.21.15.116:0/2613487886 conn(0x7fe7440b0c80 msgr2=0x7fe7440b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.165+0000 7fe74c63d640 1 -- 172.21.15.116:0/2613487886 shutdown_connections 2023-11-28T17:10:08.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.165+0000 7fe74c63d640 1 -- 172.21.15.116:0/2613487886 wait complete. 2023-11-28T17:10:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.166+0000 7fe74c63d640 1 Processor -- start 2023-11-28T17:10:08.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.167+0000 7fe74c63d640 1 -- start start 2023-11-28T17:10:08.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.167+0000 7fe74c63d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe74406dbc0 0x7fe74415d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.167+0000 7fe74c63d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7440cbe40 0x7fe74415b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.168+0000 7fe74c63d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe74415a4e0 0x7fe744154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.168+0000 7fe74c63d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74415d840 con 0x7fe7440cbe40 2023-11-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.168+0000 7fe74c63d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7441f7540 con 0x7fe74406dbc0 2023-11-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.168+0000 7fe74c63d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7441f7820 con 0x7fe74415a4e0 2023-11-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.168+0000 7fe749bd8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7440cbe40 0x7fe74415b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.168+0000 7fe749bd8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7440cbe40 0x7fe74415b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60200/0 (socket says 172.21.15.116:60200) 2023-11-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.168+0000 7fe749bd8640 1 -- 172.21.15.116:0/4004217372 learned_addr learned my addr 172.21.15.116:0/4004217372 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.168+0000 7fe74a3d9640 1 --2- 172.21.15.116:0/4004217372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe74406dbc0 0x7fe74415d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:08.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.168+0000 7fe749bd8640 1 -- 172.21.15.116:0/4004217372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe74415a4e0 msgr2=0x7fe744154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.168+0000 7fe749bd8640 1 --2- 172.21.15.116:0/4004217372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe74415a4e0 0x7fe744154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.168+0000 7fe749bd8640 1 -- 172.21.15.116:0/4004217372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe74406dbc0 msgr2=0x7fe74415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.168+0000 7fe749bd8640 1 --2- 172.21.15.116:0/4004217372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe74406dbc0 0x7fe74415d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.168+0000 7fe749bd8640 1 -- 172.21.15.116:0/4004217372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73c009b30 con 0x7fe7440cbe40 2023-11-28T17:10:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.168+0000 7fe74a3d9640 1 --2- 172.21.15.116:0/4004217372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe74406dbc0 0x7fe74415d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:08.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.169+0000 7fe749bd8640 1 --2- 172.21.15.116:0/4004217372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7440cbe40 0x7fe74415b560 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe73400b4a0 tx=0x7fe73400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.169+0000 7fe73affd640 1 -- 172.21.15.116:0/4004217372 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734010610 con 0x7fe7440cbe40 2023-11-28T17:10:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.169+0000 7fe73affd640 1 -- 172.21.15.116:0/4004217372 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe734010770 con 0x7fe7440cbe40 2023-11-28T17:10:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.169+0000 7fe73affd640 1 -- 172.21.15.116:0/4004217372 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe734010b20 con 0x7fe7440cbe40 2023-11-28T17:10:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.169+0000 7fe74c63d640 1 -- 172.21.15.116:0/4004217372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7441f7b00 con 0x7fe7440cbe40 2023-11-28T17:10:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.169+0000 7fe74c63d640 1 -- 172.21.15.116:0/4004217372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74415baa0 con 0x7fe7440cbe40 2023-11-28T17:10:08.172 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.170+0000 7fe74c63d640 1 -- 172.21.15.116:0/4004217372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe74404e880 con 0x7fe7440cbe40 2023-11-28T17:10:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.171+0000 7fe73affd640 1 -- 172.21.15.116:0/4004217372 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe734010c80 con 0x7fe7440cbe40 2023-11-28T17:10:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.172+0000 7fe73affd640 1 --2- 172.21.15.116:0/4004217372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe72806b2a0 0x7fe72806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.172+0000 7fe73affd640 1 -- 172.21.15.116:0/4004217372 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fe734089850 con 0x7fe7440cbe40 2023-11-28T17:10:08.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.172+0000 7fe74a3d9640 1 --2- 172.21.15.116:0/4004217372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe72806b2a0 0x7fe72806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:08.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.173+0000 7fe74a3d9640 1 --2- 172.21.15.116:0/4004217372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe72806b2a0 0x7fe72806d760 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe740006fd0 tx=0x7fe740008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:08.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.176+0000 7fe73affd640 1 -- 172.21.15.116:0/4004217372 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe734054660 con 0x7fe7440cbe40 2023-11-28T17:10:08.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:08.366+0000 7fe74c63d640 1 -- 172.21.15.116:0/4004217372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe744069360 con 0x7fe7440cbe40 2023-11-28T17:10:09.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.197+0000 7fe73affd640 1 -- 172.21.15.116:0/4004217372 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe7340140d0 con 0x7fe7440cbe40 2023-11-28T17:10:09.199 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' removed 2023-11-28T17:10:09.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.203+0000 7fe74c63d640 1 -- 172.21.15.116:0/4004217372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe72806b2a0 msgr2=0x7fe72806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:09.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.203+0000 7fe74c63d640 1 --2- 172.21.15.116:0/4004217372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe72806b2a0 0x7fe72806d760 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe740006fd0 tx=0x7fe740008040 comp rx=0 tx=0).stop 2023-11-28T17:10:09.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.203+0000 7fe74c63d640 1 -- 172.21.15.116:0/4004217372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7440cbe40 msgr2=0x7fe74415b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:09.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.203+0000 7fe74c63d640 1 --2- 172.21.15.116:0/4004217372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7440cbe40 0x7fe74415b560 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe73400b4a0 tx=0x7fe73400b960 comp rx=0 tx=0).stop 2023-11-28T17:10:09.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.204+0000 7fe74c63d640 1 -- 172.21.15.116:0/4004217372 shutdown_connections 2023-11-28T17:10:09.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.204+0000 7fe74c63d640 1 --2- 172.21.15.116:0/4004217372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe72806b2a0 0x7fe72806d760 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:09.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.204+0000 7fe74c63d640 1 --2- 172.21.15.116:0/4004217372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe74415a4e0 0x7fe744154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:09.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.204+0000 7fe74c63d640 1 --2- 172.21.15.116:0/4004217372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7440cbe40 0x7fe74415b560 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:09.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.204+0000 7fe74c63d640 1 --2- 172.21.15.116:0/4004217372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe74406dbc0 0x7fe74415d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:09.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.204+0000 7fe74c63d640 1 -- 172.21.15.116:0/4004217372 >> 172.21.15.116:0/4004217372 conn(0x7fe7440b0c80 msgr2=0x7fe7440ac0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:09.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.205+0000 7fe74c63d640 1 -- 172.21.15.116:0/4004217372 shutdown_connections 2023-11-28T17:10:09.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.205+0000 7fe74c63d640 1 -- 172.21.15.116:0/4004217372 wait complete. 2023-11-28T17:10:09.230 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T17:10:09.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.429+0000 7f2306a90640 1 Processor -- start 2023-11-28T17:10:09.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.429+0000 7f2306a90640 1 -- start start 2023-11-28T17:10:09.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.431+0000 7f2306a90640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f230015b0c0 0x7f230015ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:09.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.431+0000 7f2306a90640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f23000ce940 0x7f23000cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:09.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.431+0000 7f2306a90640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23000cd320 con 0x7f2300071b40 2023-11-28T17:10:09.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.431+0000 7f2306a90640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23000ced80 con 0x7f23000ce940 2023-11-28T17:10:09.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.431+0000 7f2306a90640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23000cbee0 con 0x7f230015b0c0 2023-11-28T17:10:09.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.431+0000 7f22fffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f23000ce940 0x7f23000cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:09.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.431+0000 7f230502d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2300071b40 0x7f230015b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57674/0 (socket says 172.21.15.116:57674) 2023-11-28T17:10:09.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.431+0000 7f230502d640 1 -- 172.21.15.116:0/2099527535 learned_addr learned my addr 172.21.15.116:0/2099527535 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:09.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.432+0000 7f230482c640 1 --2- 172.21.15.116:0/2099527535 >> v2:172.21.15.156:3300/0 conn(0x7f230015b0c0 0x7f230015ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:09.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.432+0000 7f22fffff640 1 -- 172.21.15.116:0/2099527535 >> v2:172.21.15.156:3300/0 conn(0x7f230015b0c0 msgr2=0x7f230015ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:09.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.432+0000 7f22fffff640 1 --2- 172.21.15.116:0/2099527535 >> v2:172.21.15.156:3300/0 conn(0x7f230015b0c0 0x7f230015ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:09.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.432+0000 7f22fffff640 1 -- 172.21.15.116:0/2099527535 >> v1:172.21.15.116:6789/0 conn(0x7f2300071b40 legacy=0x7f230015b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:09.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.432+0000 7f22fffff640 1 -- 172.21.15.116:0/2099527535 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23000cc710 con 0x7f23000ce940 2023-11-28T17:10:09.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.432+0000 7f230482c640 1 --2- 172.21.15.116:0/2099527535 >> v2:172.21.15.156:3300/0 conn(0x7f230015b0c0 0x7f230015ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:09.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.432+0000 7f22ff7fe640 1 -- 172.21.15.116:0/2099527535 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278401549 0 0) 0x7f23000cd320 con 0x7f2300071b40 2023-11-28T17:10:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.433+0000 7f22fffff640 1 --2- 172.21.15.116:0/2099527535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23000ce940 0x7f23000cb9a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f230006b470 tx=0x7f22ec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2ec8faf65eb6e04 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.433+0000 7f22ff7fe640 1 -- 172.21.15.116:0/2099527535 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec02e070 con 0x7f23000ce940 2023-11-28T17:10:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.433+0000 7f22ff7fe640 1 -- 172.21.15.116:0/2099527535 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ec0315a0 con 0x7f23000ce940 2023-11-28T17:10:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.433+0000 7f22ff7fe640 1 -- 172.21.15.116:0/2099527535 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec031930 con 0x7f23000ce940 2023-11-28T17:10:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.434+0000 7f2306a90640 1 -- 172.21.15.116:0/2099527535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23000ce940 msgr2=0x7f23000cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:09.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.434+0000 7f2306a90640 1 --2- 172.21.15.116:0/2099527535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23000ce940 0x7f23000cb9a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f230006b470 tx=0x7f22ec02b840 comp rx=0 tx=0).stop 2023-11-28T17:10:09.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.434+0000 7f2306a90640 1 -- 172.21.15.116:0/2099527535 shutdown_connections 2023-11-28T17:10:09.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.434+0000 7f2306a90640 1 --2- 172.21.15.116:0/2099527535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23000ce940 0x7f23000cb9a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:09.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.434+0000 7f2306a90640 1 --2- 172.21.15.116:0/2099527535 >> v2:172.21.15.156:3300/0 conn(0x7f230015b0c0 0x7f230015ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:09.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.434+0000 7f2306a90640 1 -- 172.21.15.116:0/2099527535 >> 172.21.15.116:0/2099527535 conn(0x7f23000b0cf0 msgr2=0x7f23000b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:09.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.434+0000 7f2306a90640 1 -- 172.21.15.116:0/2099527535 shutdown_connections 2023-11-28T17:10:09.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.434+0000 7f2306a90640 1 -- 172.21.15.116:0/2099527535 wait complete. 2023-11-28T17:10:09.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.436+0000 7f2306a90640 1 Processor -- start 2023-11-28T17:10:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.436+0000 7f2306a90640 1 -- start start 2023-11-28T17:10:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.437+0000 7f2306a90640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2300071b40 0x7f2300152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.437+0000 7f2306a90640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23000ce940 0x7f23001f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.437+0000 7f2306a90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f230015b0c0 0x7f23001f3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.437+0000 7f2306a90640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23000b2690 con 0x7f230015b0c0 2023-11-28T17:10:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.437+0000 7f2306a90640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23001f3970 con 0x7f2300071b40 2023-11-28T17:10:09.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.437+0000 7f2306a90640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23001f3c50 con 0x7f23000ce940 2023-11-28T17:10:09.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.437+0000 7f230502d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f230015b0c0 0x7f23001f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:09.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.437+0000 7f230502d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f230015b0c0 0x7f23001f3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60222/0 (socket says 172.21.15.116:60222) 2023-11-28T17:10:09.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.437+0000 7f230502d640 1 -- 172.21.15.116:0/4262681350 learned_addr learned my addr 172.21.15.116:0/4262681350 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:09.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.437+0000 7f230502d640 1 -- 172.21.15.116:0/4262681350 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23000ce940 msgr2=0x7f23001f2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:09.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.437+0000 7f230502d640 1 --2- 172.21.15.116:0/4262681350 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23000ce940 0x7f23001f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:09.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.437+0000 7f230502d640 1 -- 172.21.15.116:0/4262681350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2300071b40 msgr2=0x7f2300152b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:09.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.437+0000 7f230502d640 1 --2- 172.21.15.116:0/4262681350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2300071b40 0x7f2300152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:09.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.437+0000 7f230502d640 1 -- 172.21.15.116:0/4262681350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec009400 con 0x7f230015b0c0 2023-11-28T17:10:09.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.438+0000 7f230502d640 1 --2- 172.21.15.116:0/4262681350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f230015b0c0 0x7f23001f3380 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f22f800c780 tx=0x7f22f800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:09.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.438+0000 7f22fd7fa640 1 -- 172.21.15.116:0/4262681350 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f800ce90 con 0x7f230015b0c0 2023-11-28T17:10:09.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.438+0000 7f22fd7fa640 1 -- 172.21.15.116:0/4262681350 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f800a640 con 0x7f230015b0c0 2023-11-28T17:10:09.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.438+0000 7f22fd7fa640 1 -- 172.21.15.116:0/4262681350 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f800a9d0 con 0x7f230015b0c0 2023-11-28T17:10:09.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.438+0000 7f2306a90640 1 -- 172.21.15.116:0/4262681350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23000b28a0 con 0x7f230015b0c0 2023-11-28T17:10:09.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.438+0000 7f2306a90640 1 -- 172.21.15.116:0/4262681350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2300158230 con 0x7f230015b0c0 2023-11-28T17:10:09.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.439+0000 7f2306a90640 1 -- 172.21.15.116:0/4262681350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23000b2dc0 con 0x7f230015b0c0 2023-11-28T17:10:09.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.440+0000 7f22fd7fa640 1 -- 172.21.15.116:0/4262681350 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f22f800ab30 con 0x7f230015b0c0 2023-11-28T17:10:09.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.441+0000 7f22fd7fa640 1 --2- 172.21.15.116:0/4262681350 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f22e406b2a0 0x7f22e406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:09.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.441+0000 7f22fd7fa640 1 -- 172.21.15.116:0/4262681350 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5386+0+0 (secure 0 0 0) 0x7f22f8088fe0 con 0x7f230015b0c0 2023-11-28T17:10:09.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.441+0000 7f230482c640 1 --2- 172.21.15.116:0/4262681350 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f22e406b2a0 0x7f22e406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:09.445 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.442+0000 7f230482c640 1 --2- 172.21.15.116:0/4262681350 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f22e406b2a0 0x7f22e406d760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f22f4006fd0 tx=0x7f22f4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:09.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.445+0000 7f22fd7fa640 1 -- 172.21.15.116:0/4262681350 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f23000b2dc0 con 0x7f230015b0c0 2023-11-28T17:10:09.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:09.634+0000 7f2306a90640 1 -- 172.21.15.116:0/4262681350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f23000b2dc0 con 0x7f230015b0c0 2023-11-28T17:10:10.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.201+0000 7f22fd7fa640 1 -- 172.21.15.116:0/4262681350 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7f23000b2dc0 con 0x7f230015b0c0 2023-11-28T17:10:10.203 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' removed 2023-11-28T17:10:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.205+0000 7f2306a90640 1 -- 172.21.15.116:0/4262681350 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f22e406b2a0 msgr2=0x7f22e406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.205+0000 7f2306a90640 1 --2- 172.21.15.116:0/4262681350 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f22e406b2a0 0x7f22e406d760 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f22f4006fd0 tx=0x7f22f4008040 comp rx=0 tx=0).stop 2023-11-28T17:10:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.205+0000 7f2306a90640 1 -- 172.21.15.116:0/4262681350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f230015b0c0 msgr2=0x7f23001f3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.205+0000 7f2306a90640 1 --2- 172.21.15.116:0/4262681350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f230015b0c0 0x7f23001f3380 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f22f800c780 tx=0x7f22f800cc40 comp rx=0 tx=0).stop 2023-11-28T17:10:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.205+0000 7f2306a90640 1 -- 172.21.15.116:0/4262681350 shutdown_connections 2023-11-28T17:10:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.205+0000 7f2306a90640 1 --2- 172.21.15.116:0/4262681350 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f22e406b2a0 0x7f22e406d760 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.205+0000 7f2306a90640 1 --2- 172.21.15.116:0/4262681350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f230015b0c0 0x7f23001f3380 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:10.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.205+0000 7f2306a90640 1 --2- 172.21.15.116:0/4262681350 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23000ce940 0x7f23001f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:10.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.205+0000 7f2306a90640 1 --2- 172.21.15.116:0/4262681350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2300071b40 0x7f2300152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:10.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.205+0000 7f2306a90640 1 -- 172.21.15.116:0/4262681350 >> 172.21.15.116:0/4262681350 conn(0x7f23000b0cf0 msgr2=0x7f230006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:10.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.205+0000 7f2306a90640 1 -- 172.21.15.116:0/4262681350 shutdown_connections 2023-11-28T17:10:10.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.205+0000 7f2306a90640 1 -- 172.21.15.116:0/4262681350 wait complete. 2023-11-28T17:10:10.237 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T17:10:10.237 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T17:10:10.237 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T17:10:10.237 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:10:10.238 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T17:10:10.238 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T17:10:10.238 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T17:10:10.238 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:10:10.238 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T17:10:10.238 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T17:10:10.239 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T17:10:10.239 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:10:10.239 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T17:10:10.239 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T17:10:10.239 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T17:10:10.239 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:10:10.257 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T17:10:10.258 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T17:10:10.258 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T17:10:10.258 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T17:10:10.268 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T17:10:10.268 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T17:10:10.269 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T17:10:10.270 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T17:10:10.305 INFO:tasks.ceph.mds.d:Started 2023-11-28T17:10:10.305 DEBUG:teuthology.parallel:result is None 2023-11-28T17:10:10.311 INFO:tasks.ceph.mds.b:Started 2023-11-28T17:10:10.311 DEBUG:teuthology.parallel:result is None 2023-11-28T17:10:10.317 INFO:tasks.ceph.mds.a:Started 2023-11-28T17:10:10.317 DEBUG:teuthology.parallel:result is None 2023-11-28T17:10:10.324 INFO:tasks.ceph.mds.c:Started 2023-11-28T17:10:10.325 DEBUG:teuthology.parallel:result is None 2023-11-28T17:10:10.325 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T17:10:10.459 INFO:tasks.ceph.mds.d.smithi156.stdout:starting mds.d at 2023-11-28T17:10:10.459 INFO:tasks.ceph.mds.b.smithi156.stdout:starting mds.b at 2023-11-28T17:10:10.461 INFO:tasks.ceph.mds.b.smithi156.stderr:2023-11-28T17:10:10.459+0000 7f7664a7d640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2023-11-28T17:10:10.469 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2023-11-28T17:10:10.479 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2023-11-28T17:10:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.505+0000 7fe8b0f1d640 1 Processor -- start 2023-11-28T17:10:10.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.505+0000 7fe8b0f1d640 1 -- start start 2023-11-28T17:10:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.507+0000 7fe8b0f1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe8ac146a20 0x7fe8ac0b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.507+0000 7fe8b0f1d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe8ac071500 0x7fe8ac15e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.507+0000 7fe8b0f1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8ac15db90 con 0x7fe8ac15e560 2023-11-28T17:10:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.507+0000 7fe8b0f1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8ac0b3650 con 0x7fe8ac146a20 2023-11-28T17:10:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.507+0000 7fe8b0f1d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8ac15d840 con 0x7fe8ac071500 2023-11-28T17:10:10.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.507+0000 7fe8aa59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe8ac146a20 0x7fe8ac0b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:10.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.507+0000 7fe8aad9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe8ac15e560 0x7fe8ac06a4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57704/0 (socket says 172.21.15.116:57704) 2023-11-28T17:10:10.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.507+0000 7fe8aad9d640 1 -- 172.21.15.116:0/2756586913 learned_addr learned my addr 172.21.15.116:0/2756586913 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:10.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.507+0000 7fe8a9d9b640 1 --2- 172.21.15.116:0/2756586913 >> v2:172.21.15.156:3300/0 conn(0x7fe8ac071500 0x7fe8ac15e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:10.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.507+0000 7fe8aa59c640 1 -- 172.21.15.116:0/2756586913 >> v2:172.21.15.156:3300/0 conn(0x7fe8ac071500 msgr2=0x7fe8ac15e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:10.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.507+0000 7fe8aa59c640 1 --2- 172.21.15.116:0/2756586913 >> v2:172.21.15.156:3300/0 conn(0x7fe8ac071500 0x7fe8ac15e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:10.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.507+0000 7fe8aa59c640 1 -- 172.21.15.116:0/2756586913 >> v1:172.21.15.116:6789/0 conn(0x7fe8ac15e560 legacy=0x7fe8ac06a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:10.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.507+0000 7fe8aa59c640 1 -- 172.21.15.116:0/2756586913 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ac0b38f0 con 0x7fe8ac146a20 2023-11-28T17:10:10.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.507+0000 7fe8a959a640 1 -- 172.21.15.116:0/2756586913 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874369629 0 0) 0x7fe8ac15db90 con 0x7fe8ac15e560 2023-11-28T17:10:10.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.509+0000 7fe8aa59c640 1 --2- 172.21.15.116:0/2756586913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac146a20 0x7fe8ac0b3110 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac072980 tx=0x7fe89c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b44a21060cfd97 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.509+0000 7fe8a959a640 1 -- 172.21.15.116:0/2756586913 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c02e070 con 0x7fe8ac146a20 2023-11-28T17:10:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.509+0000 7fe8a959a640 1 -- 172.21.15.116:0/2756586913 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89c0315a0 con 0x7fe8ac146a20 2023-11-28T17:10:10.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.509+0000 7fe8a959a640 1 -- 172.21.15.116:0/2756586913 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c031930 con 0x7fe8ac146a20 2023-11-28T17:10:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.509+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2756586913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac146a20 msgr2=0x7fe8ac0b3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.509+0000 7fe8b0f1d640 1 --2- 172.21.15.116:0/2756586913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac146a20 0x7fe8ac0b3110 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac072980 tx=0x7fe89c02b840 comp rx=0 tx=0).stop 2023-11-28T17:10:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.509+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2756586913 shutdown_connections 2023-11-28T17:10:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.509+0000 7fe8b0f1d640 1 --2- 172.21.15.116:0/2756586913 >> v2:172.21.15.156:3300/0 conn(0x7fe8ac071500 0x7fe8ac15e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.509+0000 7fe8b0f1d640 1 --2- 172.21.15.116:0/2756586913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac146a20 0x7fe8ac0b3110 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.509+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2756586913 >> 172.21.15.116:0/2756586913 conn(0x7fe8ac06b5d0 msgr2=0x7fe8ac06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.509+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2756586913 shutdown_connections 2023-11-28T17:10:10.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.510+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2756586913 wait complete. 2023-11-28T17:10:10.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.511+0000 7fe8b0f1d640 1 Processor -- start 2023-11-28T17:10:10.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.511+0000 7fe8b0f1d640 1 -- start start 2023-11-28T17:10:10.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.512+0000 7fe8b0f1d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8ac071500 0x7fe8ac12e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:10.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.512+0000 7fe8b0f1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac146a20 0x7fe8ac15a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:10.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.512+0000 7fe8b0f1d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8ac15ab20 0x7fe8ac140680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:10.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.512+0000 7fe8b0f1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ac13f5e0 con 0x7fe8ac146a20 2023-11-28T17:10:10.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.512+0000 7fe8b0f1d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ac13f8c0 con 0x7fe8ac071500 2023-11-28T17:10:10.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.512+0000 7fe8b0f1d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ac13fba0 con 0x7fe8ac15ab20 2023-11-28T17:10:10.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.513+0000 7fe8a9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac146a20 0x7fe8ac15a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:10.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.513+0000 7fe8a9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac146a20 0x7fe8ac15a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60282/0 (socket says 172.21.15.116:60282) 2023-11-28T17:10:10.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.513+0000 7fe8a9d9b640 1 -- 172.21.15.116:0/2624004247 learned_addr learned my addr 172.21.15.116:0/2624004247 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:10.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.513+0000 7fe8aa59c640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8ac071500 0x7fe8ac12e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:10.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.513+0000 7fe8aad9d640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8ac15ab20 0x7fe8ac140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:10.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.513+0000 7fe8a9d9b640 1 -- 172.21.15.116:0/2624004247 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8ac15ab20 msgr2=0x7fe8ac140680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:10.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.513+0000 7fe8a9d9b640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8ac15ab20 0x7fe8ac140680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:10.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.513+0000 7fe8a9d9b640 1 -- 172.21.15.116:0/2624004247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8ac071500 msgr2=0x7fe8ac12e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:10.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.513+0000 7fe8a9d9b640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8ac071500 0x7fe8ac12e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:10.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.513+0000 7fe8a9d9b640 1 -- 172.21.15.116:0/2624004247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c009400 con 0x7fe8ac146a20 2023-11-28T17:10:10.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.513+0000 7fe8aa59c640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8ac071500 0x7fe8ac12e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:10:10.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.513+0000 7fe8a9d9b640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac146a20 0x7fe8ac15a5e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe8a000d800 tx=0x7fe8a000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:10.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.514+0000 7fe89affd640 1 -- 172.21.15.116:0/2624004247 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a000b6c0 con 0x7fe8ac146a20 2023-11-28T17:10:10.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.514+0000 7fe89affd640 1 -- 172.21.15.116:0/2624004247 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a000b820 con 0x7fe8ac146a20 2023-11-28T17:10:10.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.514+0000 7fe89affd640 1 -- 172.21.15.116:0/2624004247 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a000bbb0 con 0x7fe8ac146a20 2023-11-28T17:10:10.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.514+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2624004247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8ac13fe80 con 0x7fe8ac146a20 2023-11-28T17:10:10.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.514+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2624004247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8ac13df00 con 0x7fe8ac146a20 2023-11-28T17:10:10.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.515+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2624004247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8ac0693d0 con 0x7fe8ac146a20 2023-11-28T17:10:10.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.516+0000 7fe89affd640 1 -- 172.21.15.116:0/2624004247 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe8a000bd10 con 0x7fe8ac146a20 2023-11-28T17:10:10.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.517+0000 7fe89affd640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe88c06b2a0 0x7fe88c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:10.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.517+0000 7fe89affd640 1 -- 172.21.15.116:0/2624004247 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe8a0089e80 con 0x7fe8ac146a20 2023-11-28T17:10:10.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.517+0000 7fe8aa59c640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe88c06b2a0 0x7fe88c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:10.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.518+0000 7fe8aa59c640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe88c06b2a0 0x7fe88c06d760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe89c036000 tx=0x7fe89c03d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:10.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.521+0000 7fe89affd640 1 -- 172.21.15.116:0/2624004247 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8a0054fe0 con 0x7fe8ac146a20 2023-11-28T17:10:10.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:10.711+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2624004247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe8ac15b5e0 con 0x7fe8ac146a20 2023-11-28T17:10:12.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.028+0000 7fe89affd640 1 -- 172.21.15.116:0/2624004247 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v30) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe8a0058600 con 0x7fe8ac146a20 2023-11-28T17:10:12.030 INFO:teuthology.orchestra.run.smithi116.stderr: removed all blocklist entries 2023-11-28T17:10:12.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.032+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2624004247 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe88c06b2a0 msgr2=0x7fe88c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:12.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.032+0000 7fe8b0f1d640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe88c06b2a0 0x7fe88c06d760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe89c036000 tx=0x7fe89c03d040 comp rx=0 tx=0).stop 2023-11-28T17:10:12.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.032+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2624004247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac146a20 msgr2=0x7fe8ac15a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:12.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.032+0000 7fe8b0f1d640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac146a20 0x7fe8ac15a5e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe8a000d800 tx=0x7fe8a000dcc0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.032+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2624004247 shutdown_connections 2023-11-28T17:10:12.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.032+0000 7fe8b0f1d640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe88c06b2a0 0x7fe88c06d760 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.032+0000 7fe8b0f1d640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8ac15ab20 0x7fe8ac140680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.032+0000 7fe8b0f1d640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac146a20 0x7fe8ac15a5e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.032+0000 7fe8b0f1d640 1 --2- 172.21.15.116:0/2624004247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8ac071500 0x7fe8ac12e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.032+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2624004247 >> 172.21.15.116:0/2624004247 conn(0x7fe8ac06b5d0 msgr2=0x7fe8ac15ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:12.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.032+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2624004247 shutdown_connections 2023-11-28T17:10:12.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.032+0000 7fe8b0f1d640 1 -- 172.21.15.116:0/2624004247 wait complete. 2023-11-28T17:10:12.053 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T17:10:12.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.251+0000 7f3c04a07640 1 Processor -- start 2023-11-28T17:10:12.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.251+0000 7f3c04a07640 1 -- start start 2023-11-28T17:10:12.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.253+0000 7f3c04a07640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3c00144430 0x7f3c0015f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:12.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.253+0000 7f3c04a07640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3c000703f0 0x7f3c00071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:12.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.253+0000 7f3c04a07640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c00070bd0 con 0x7f3c000721b0 2023-11-28T17:10:12.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.253+0000 7f3c04a07640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c00144810 con 0x7f3c00144430 2023-11-28T17:10:12.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.253+0000 7f3c04a07640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c00070eb0 con 0x7f3c000703f0 2023-11-28T17:10:12.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.254+0000 7f3bfe59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3c00144430 0x7f3c0015f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:12.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.254+0000 7f3bf7fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3c000703f0 0x7f3c00071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:12.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.254+0000 7f3bfe59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3c00144430 0x7f3c0015f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:53014/0 (socket says 172.21.15.116:53014) 2023-11-28T17:10:12.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.254+0000 7f3bfe59c640 1 -- 172.21.15.116:0/2342248249 learned_addr learned my addr 172.21.15.116:0/2342248249 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:12.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.254+0000 7f3bfed9d640 1 --1- 172.21.15.116:0/2342248249 >> v1:172.21.15.156:6789/0 conn(0x7f3c000721b0 0x7f3c0006c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.116:58964/0 (socket says 172.21.15.116:58964) 2023-11-28T17:10:12.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.255+0000 7f3bfe59c640 1 -- 172.21.15.116:0/2342248249 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3c000703f0 msgr2=0x7f3c00071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:12.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.255+0000 7f3bfe59c640 1 --2- 172.21.15.116:0/2342248249 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3c000703f0 0x7f3c00071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.255+0000 7f3bfe59c640 1 -- 172.21.15.116:0/2342248249 >> v1:172.21.15.156:6789/0 conn(0x7f3c000721b0 legacy=0x7f3c0006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:12.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.255+0000 7f3bfe59c640 1 -- 172.21.15.116:0/2342248249 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c00071190 con 0x7f3c00144430 2023-11-28T17:10:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.256+0000 7f3bfe59c640 1 --2- 172.21.15.116:0/2342248249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c00144430 0x7f3c0015f7b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0009870 tx=0x7f3bf002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fe27d31bd893939 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:12.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.257+0000 7f3bfdd9b640 1 -- 172.21.15.116:0/2342248249 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0030840 con 0x7f3c00144430 2023-11-28T17:10:12.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.257+0000 7f3bfdd9b640 1 -- 172.21.15.116:0/2342248249 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf00309a0 con 0x7f3c00144430 2023-11-28T17:10:12.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.257+0000 7f3bfdd9b640 1 -- 172.21.15.116:0/2342248249 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0030d30 con 0x7f3c00144430 2023-11-28T17:10:12.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.258+0000 7f3c04a07640 1 -- 172.21.15.116:0/2342248249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c00144430 msgr2=0x7f3c0015f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:12.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.258+0000 7f3c04a07640 1 --2- 172.21.15.116:0/2342248249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c00144430 0x7f3c0015f7b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0009870 tx=0x7f3bf002b980 comp rx=0 tx=0).stop 2023-11-28T17:10:12.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.258+0000 7f3c04a07640 1 -- 172.21.15.116:0/2342248249 shutdown_connections 2023-11-28T17:10:12.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.258+0000 7f3c04a07640 1 --2- 172.21.15.116:0/2342248249 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3c000703f0 0x7f3c00071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.258+0000 7f3c04a07640 1 --2- 172.21.15.116:0/2342248249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c00144430 0x7f3c0015f7b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.258+0000 7f3c04a07640 1 -- 172.21.15.116:0/2342248249 >> 172.21.15.116:0/2342248249 conn(0x7f3c0006a4c0 msgr2=0x7f3c0006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:12.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.258+0000 7f3c04a07640 1 -- 172.21.15.116:0/2342248249 shutdown_connections 2023-11-28T17:10:12.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.258+0000 7f3c04a07640 1 -- 172.21.15.116:0/2342248249 wait complete. 2023-11-28T17:10:12.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.260+0000 7f3c04a07640 1 Processor -- start 2023-11-28T17:10:12.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.260+0000 7f3c04a07640 1 -- start start 2023-11-28T17:10:12.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.261+0000 7f3c04a07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c000703f0 0x7f3c0015e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:12.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.261+0000 7f3c04a07640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c000721b0 0x7f3c000ac960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:12.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.261+0000 7f3c04a07640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3c00144430 0x7f3c000ab250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.262+0000 7f3c04a07640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c000a9f60 con 0x7f3c000703f0 2023-11-28T17:10:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.262+0000 7f3c04a07640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c000acea0 con 0x7f3c000721b0 2023-11-28T17:10:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.262+0000 7f3c04a07640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c000aa1b0 con 0x7f3c00144430 2023-11-28T17:10:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.262+0000 7f3bfe59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c000703f0 0x7f3c0015e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.262+0000 7f3bfe59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c000703f0 0x7f3c0015e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60296/0 (socket says 172.21.15.116:60296) 2023-11-28T17:10:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.262+0000 7f3bfe59c640 1 -- 172.21.15.116:0/1061886265 learned_addr learned my addr 172.21.15.116:0/1061886265 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.262+0000 7f3bf7fff640 1 --2- 172.21.15.116:0/1061886265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c000721b0 0x7f3c000ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.262+0000 7f3bfe59c640 1 -- 172.21.15.116:0/1061886265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3c00144430 msgr2=0x7f3c000ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.262+0000 7f3bfe59c640 1 --2- 172.21.15.116:0/1061886265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3c00144430 0x7f3c000ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.262+0000 7f3bfe59c640 1 -- 172.21.15.116:0/1061886265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c000721b0 msgr2=0x7f3c000ac960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.262+0000 7f3bfe59c640 1 --2- 172.21.15.116:0/1061886265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c000721b0 0x7f3c000ac960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.262+0000 7f3bfe59c640 1 -- 172.21.15.116:0/1061886265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf00094d0 con 0x7f3c000703f0 2023-11-28T17:10:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.263+0000 7f3bfe59c640 1 --2- 172.21.15.116:0/1061886265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c000703f0 0x7f3c0015e590 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0009990 tx=0x7f3bf0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.263+0000 7f3bf6ffd640 1 -- 172.21.15.116:0/1061886265 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0039c90 con 0x7f3c000703f0 2023-11-28T17:10:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.263+0000 7f3bf6ffd640 1 -- 172.21.15.116:0/1061886265 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf0039e10 con 0x7f3c000703f0 2023-11-28T17:10:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.263+0000 7f3bf6ffd640 1 -- 172.21.15.116:0/1061886265 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf00446c0 con 0x7f3c000703f0 2023-11-28T17:10:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.263+0000 7f3c04a07640 1 -- 172.21.15.116:0/1061886265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c000aa490 con 0x7f3c000703f0 2023-11-28T17:10:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.263+0000 7f3c04a07640 1 -- 172.21.15.116:0/1061886265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c000aaa50 con 0x7f3c000703f0 2023-11-28T17:10:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.264+0000 7f3c04a07640 1 -- 172.21.15.116:0/1061886265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c00144950 con 0x7f3c000703f0 2023-11-28T17:10:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.265+0000 7f3bf6ffd640 1 -- 172.21.15.116:0/1061886265 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3bf002f390 con 0x7f3c000703f0 2023-11-28T17:10:12.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.266+0000 7f3bf6ffd640 1 --2- 172.21.15.116:0/1061886265 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3be006b2a0 0x7f3be006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:12.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.266+0000 7f3bf6ffd640 1 -- 172.21.15.116:0/1061886265 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3bf003e070 con 0x7f3c000703f0 2023-11-28T17:10:12.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.266+0000 7f3bf7fff640 1 --2- 172.21.15.116:0/1061886265 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3be006b2a0 0x7f3be006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:12.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.267+0000 7f3bf7fff640 1 --2- 172.21.15.116:0/1061886265 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3be006b2a0 0x7f3be006d760 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3c001432c0 tx=0x7f3be8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.270+0000 7f3bf6ffd640 1 -- 172.21.15.116:0/1061886265 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bf0077140 con 0x7f3c000703f0 2023-11-28T17:10:12.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.532+0000 7f3c04a07640 1 -- 172.21.15.116:0/1061886265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3c000693d0 con 0x7f3c000703f0 2023-11-28T17:10:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.533+0000 7f3bf6ffd640 1 -- 172.21.15.116:0/1061886265 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f3bf007a760 con 0x7f3c000703f0 2023-11-28T17:10:12.535 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:12.535 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-28T17:10:12.535 INFO:teuthology.orchestra.run.smithi116.stdout: "auth_dump": [ 2023-11-28T17:10:12.535 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.535 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.a", 2023-11-28T17:10:12.535 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBsHmZlDk7pERAAzfB2wxucZ6BlKfc/eBL5fA==", 2023-11-28T17:10:12.535 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.535 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:10:12.535 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-28T17:10:12.536 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-28T17:10:12.536 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.536 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.536 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.536 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.b", 2023-11-28T17:10:12.536 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBsHmZln5I1NRAAK30dyAqRuGLeJ4mGw90Rpg==", 2023-11-28T17:10:12.536 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.536 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:10:12.536 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-28T17:10:12.536 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-28T17:10:12.536 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.537 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.537 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.537 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.c", 2023-11-28T17:10:12.537 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBsHmZlvVteIxAAqG8H+6/g2owr7mI87oYHgA==", 2023-11-28T17:10:12.537 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.537 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:10:12.537 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-28T17:10:12.537 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-28T17:10:12.537 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.537 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.d", 2023-11-28T17:10:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBtHmZl6ieLCRAAVEyizVZ3mGTtPGRQeee11A==", 2023-11-28T17:10:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:10:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-28T17:10:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-28T17:10:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.0", 2023-11-28T17:10:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBxHmZlUqmdNhAAc7C+EUtBtgCaLUkr0PPrHg==", 2023-11-28T17:10:12.539 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.539 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:10:12.539 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:10:12.539 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.539 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.539 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.539 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.1", 2023-11-28T17:10:12.539 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQB1HmZlKQ07FRAAKncDe3f+D1TfLU8lhQVaGA==", 2023-11-28T17:10:12.539 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.539 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:10:12.539 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:10:12.540 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.540 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.540 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.540 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.2", 2023-11-28T17:10:12.540 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQB4HmZlvpDnLRAAG2kEjnvjJmln7TCBM9hUTw==", 2023-11-28T17:10:12.540 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.540 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:10:12.540 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:10:12.540 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.540 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.540 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.541 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.3", 2023-11-28T17:10:12.541 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQB8HmZlGD45CxAA2LESF1EBeNjPX6fjvSyEFA==", 2023-11-28T17:10:12.541 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.541 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:10:12.541 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:10:12.541 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.541 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.541 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.4", 2023-11-28T17:10:12.541 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQCBHmZlJOgqAxAAc23riVIAOTXw28t8OrTjQg==", 2023-11-28T17:10:12.541 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.541 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:10:12.542 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:10:12.542 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.542 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.5", 2023-11-28T17:10:12.542 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQCEHmZlvTMIHBAAuVkPBcYAAuTQUVqdkZ5liw==", 2023-11-28T17:10:12.542 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.542 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:10:12.542 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:10:12.542 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.542 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.543 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.6", 2023-11-28T17:10:12.543 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQCHHmZls5zfJBAAmH97PMKvbQXpeznCSkt6IQ==", 2023-11-28T17:10:12.543 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.543 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:10:12.543 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:10:12.543 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.543 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.543 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.543 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.7", 2023-11-28T17:10:12.543 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQCKHmZlahyvLRAADZqskufX275VqtUDlUHU7Q==", 2023-11-28T17:10:12.543 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.543 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:10:12.543 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:10:12.544 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.544 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.544 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.0", 2023-11-28T17:10:12.544 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBtHmZlPRewFxAAfLJPZrTljlPIoghT5ZzoBA==", 2023-11-28T17:10:12.544 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.544 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:10:12.544 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-28T17:10:12.544 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-28T17:10:12.544 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-28T17:10:12.544 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.544 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.545 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.1", 2023-11-28T17:10:12.545 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBtHmZlWG0nLhAA4xwi4Ut4dfJWwY275L/mHQ==", 2023-11-28T17:10:12.545 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.545 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:10:12.545 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-28T17:10:12.545 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-28T17:10:12.545 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-28T17:10:12.545 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.545 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.545 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.2", 2023-11-28T17:10:12.545 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBtHmZleq8ZIxAAOZsBs+OlL6TXLGqwtX11pw==", 2023-11-28T17:10:12.545 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.546 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:10:12.546 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-28T17:10:12.546 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-28T17:10:12.546 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-28T17:10:12.546 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.546 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.546 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.3", 2023-11-28T17:10:12.546 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBtHmZlkoZmORAAPWGkPoFfqsaCfAjnWmfqtw==", 2023-11-28T17:10:12.546 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.546 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:10:12.546 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-28T17:10:12.546 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-28T17:10:12.546 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-28T17:10:12.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.547 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.547 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.admin", 2023-11-28T17:10:12.547 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBrHmZlYzCkBRAAuc8Y4PZZyyT7/RQ+2mQD6Q==", 2023-11-28T17:10:12.547 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.547 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-28T17:10:12.547 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-28T17:10:12.547 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-28T17:10:12.547 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.547 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.548 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.548 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.bootstrap-mds", 2023-11-28T17:10:12.548 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQCUHmZlma/MCxAAcqmkOMMWwmq0DrdzJvOm3w==", 2023-11-28T17:10:12.548 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.548 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile bootstrap-mds" 2023-11-28T17:10:12.548 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.548 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.548 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.bootstrap-mgr", 2023-11-28T17:10:12.548 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQCUHmZlWTXNCxAAL7sVtAM4GlI/4icrzXSngg==", 2023-11-28T17:10:12.548 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.548 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile bootstrap-mgr" 2023-11-28T17:10:12.548 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.549 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.549 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.bootstrap-osd", 2023-11-28T17:10:12.549 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQCUHmZlTFrNCxAAdiaom0EgIRVU8/4LDJxfmg==", 2023-11-28T17:10:12.549 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.549 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile bootstrap-osd" 2023-11-28T17:10:12.549 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.549 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.549 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.bootstrap-rbd", 2023-11-28T17:10:12.549 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQCUHmZlDIbNCxAAFETa+Zhy/KaR+kFcliYZng==", 2023-11-28T17:10:12.549 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.549 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile bootstrap-rbd" 2023-11-28T17:10:12.550 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.550 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.550 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.550 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-11-28T17:10:12.550 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQCUHmZlEaLNCxAA6AWQd3RjWxBj5QSGeknPdQ==", 2023-11-28T17:10:12.550 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.550 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-11-28T17:10:12.550 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.550 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.550 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.550 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.bootstrap-rgw", 2023-11-28T17:10:12.550 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQCUHmZlNrvNCxAABmOy87Rm2clV8uQFQw3O/g==", 2023-11-28T17:10:12.550 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.551 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile bootstrap-rgw" 2023-11-28T17:10:12.551 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.551 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.551 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mgr.x", 2023-11-28T17:10:12.551 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBsHmZlRWsIBhAAmDx975LSXMvBGPQf/P3etQ==", 2023-11-28T17:10:12.551 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.551 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-28T17:10:12.551 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile mgr", 2023-11-28T17:10:12.551 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.551 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:12.551 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:12.552 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mgr.y", 2023-11-28T17:10:12.552 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBrHmZl159eNRAAqk03SlXduu04+ZNQe8YLUg==", 2023-11-28T17:10:12.552 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:10:12.552 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-28T17:10:12.552 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile mgr", 2023-11-28T17:10:12.552 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:10:12.552 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.552 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:12.552 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2023-11-28T17:10:12.552 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-28T17:10:12.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.537+0000 7f3c04a07640 1 -- 172.21.15.116:0/1061886265 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3be006b2a0 msgr2=0x7f3be006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:12.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.537+0000 7f3c04a07640 1 --2- 172.21.15.116:0/1061886265 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3be006b2a0 0x7f3be006d760 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3c001432c0 tx=0x7f3be8008040 comp rx=0 tx=0).stop 2023-11-28T17:10:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.537+0000 7f3c04a07640 1 -- 172.21.15.116:0/1061886265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c000703f0 msgr2=0x7f3c0015e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.537+0000 7f3c04a07640 1 --2- 172.21.15.116:0/1061886265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c000703f0 0x7f3c0015e590 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0009990 tx=0x7f3bf0004730 comp rx=0 tx=0).stop 2023-11-28T17:10:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.537+0000 7f3c04a07640 1 -- 172.21.15.116:0/1061886265 shutdown_connections 2023-11-28T17:10:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.537+0000 7f3c04a07640 1 --2- 172.21.15.116:0/1061886265 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3be006b2a0 0x7f3be006d760 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.537+0000 7f3c04a07640 1 --2- 172.21.15.116:0/1061886265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3c00144430 0x7f3c000ab250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.537+0000 7f3c04a07640 1 --2- 172.21.15.116:0/1061886265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c000721b0 0x7f3c000ac960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.537+0000 7f3c04a07640 1 --2- 172.21.15.116:0/1061886265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c000703f0 0x7f3c0015e590 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.538+0000 7f3c04a07640 1 -- 172.21.15.116:0/1061886265 >> 172.21.15.116:0/1061886265 conn(0x7f3c0006a4c0 msgr2=0x7f3c00143540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:12.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.538+0000 7f3c04a07640 1 -- 172.21.15.116:0/1061886265 shutdown_connections 2023-11-28T17:10:12.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.538+0000 7f3c04a07640 1 -- 172.21.15.116:0/1061886265 wait complete. 2023-11-28T17:10:12.557 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-11-28T17:10:12.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.763+0000 7fbe4c92c640 1 Processor -- start 2023-11-28T17:10:12.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.763+0000 7fbe4c92c640 1 -- start start 2023-11-28T17:10:12.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.765+0000 7fbe4c92c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe480b3940 0x7fbe4806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:12.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.765+0000 7fbe4c92c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbe48162490 0x7fbe4806c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:12.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.765+0000 7fbe4c92c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe480b29c0 con 0x7fbe480b3d20 2023-11-28T17:10:12.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.765+0000 7fbe4c92c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe48162af0 con 0x7fbe480b3940 2023-11-28T17:10:12.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.765+0000 7fbe4c92c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe480b2090 con 0x7fbe48162490 2023-11-28T17:10:12.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.765+0000 7fbe4659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe480b3940 0x7fbe4806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:12.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.765+0000 7fbe4659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe480b3940 0x7fbe4806c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60310/0 (socket says 172.21.15.116:60310) 2023-11-28T17:10:12.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.765+0000 7fbe4659c640 1 -- 172.21.15.116:0/3516199774 learned_addr learned my addr 172.21.15.116:0/3516199774 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:12.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.766+0000 7fbe46d9d640 1 --2- 172.21.15.116:0/3516199774 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbe48162490 0x7fbe4806c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:12.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.766+0000 7fbe4659c640 1 -- 172.21.15.116:0/3516199774 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbe48162490 msgr2=0x7fbe4806c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:12.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.766+0000 7fbe4659c640 1 --2- 172.21.15.116:0/3516199774 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbe48162490 0x7fbe4806c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.766+0000 7fbe4659c640 1 -- 172.21.15.116:0/3516199774 >> v1:172.21.15.156:6789/0 conn(0x7fbe480b3d20 legacy=0x7fbe48161d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:12.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.766+0000 7fbe4659c640 1 -- 172.21.15.116:0/3516199774 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe480b2370 con 0x7fbe480b3940 2023-11-28T17:10:12.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.767+0000 7fbe4659c640 1 --2- 172.21.15.116:0/3516199774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe480b3940 0x7fbe4806c250 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbe38009870 tx=0x7fbe3802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd014477486ac6b3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:12.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.768+0000 7fbe45d9b640 1 -- 172.21.15.116:0/3516199774 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe38030840 con 0x7fbe480b3940 2023-11-28T17:10:12.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.768+0000 7fbe45d9b640 1 -- 172.21.15.116:0/3516199774 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe380309a0 con 0x7fbe480b3940 2023-11-28T17:10:12.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.768+0000 7fbe45d9b640 1 -- 172.21.15.116:0/3516199774 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe38030d30 con 0x7fbe480b3940 2023-11-28T17:10:12.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.768+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3516199774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe480b3940 msgr2=0x7fbe4806c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:12.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.768+0000 7fbe4c92c640 1 --2- 172.21.15.116:0/3516199774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe480b3940 0x7fbe4806c250 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbe38009870 tx=0x7fbe3802b980 comp rx=0 tx=0).stop 2023-11-28T17:10:12.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.768+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3516199774 shutdown_connections 2023-11-28T17:10:12.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.768+0000 7fbe4c92c640 1 --2- 172.21.15.116:0/3516199774 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbe48162490 0x7fbe4806c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.769+0000 7fbe4c92c640 1 --2- 172.21.15.116:0/3516199774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe480b3940 0x7fbe4806c250 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.769+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3516199774 >> 172.21.15.116:0/3516199774 conn(0x7fbe4806a4c0 msgr2=0x7fbe4806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:12.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.769+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3516199774 shutdown_connections 2023-11-28T17:10:12.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.769+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3516199774 wait complete. 2023-11-28T17:10:12.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.770+0000 7fbe4c92c640 1 Processor -- start 2023-11-28T17:10:12.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.770+0000 7fbe4c92c640 1 -- start start 2023-11-28T17:10:12.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.771+0000 7fbe4c92c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbe480b3940 0x7fbe48148620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:12.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.771+0000 7fbe4c92c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe480b3d20 0x7fbe481452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:12.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.771+0000 7fbe4c92c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe48162490 0x7fbe48156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:12.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.771+0000 7fbe4c92c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4814ba80 con 0x7fbe48162490 2023-11-28T17:10:12.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.771+0000 7fbe4c92c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe48148b60 con 0x7fbe480b3d20 2023-11-28T17:10:12.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.771+0000 7fbe4c92c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe481face0 con 0x7fbe480b3940 2023-11-28T17:10:12.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.772+0000 7fbe46d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe48162490 0x7fbe48156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:12.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.772+0000 7fbe46d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe48162490 0x7fbe48156b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60318/0 (socket says 172.21.15.116:60318) 2023-11-28T17:10:12.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.772+0000 7fbe46d9d640 1 -- 172.21.15.116:0/3868247659 learned_addr learned my addr 172.21.15.116:0/3868247659 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:12.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.772+0000 7fbe4659c640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbe480b3940 0x7fbe48148620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:12.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.772+0000 7fbe46d9d640 1 -- 172.21.15.116:0/3868247659 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbe480b3940 msgr2=0x7fbe48148620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:12.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.772+0000 7fbe46d9d640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbe480b3940 0x7fbe48148620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.772+0000 7fbe3dd9b640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe480b3d20 0x7fbe481452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:12.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.772+0000 7fbe46d9d640 1 -- 172.21.15.116:0/3868247659 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe480b3d20 msgr2=0x7fbe481452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:12.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.772+0000 7fbe46d9d640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe480b3d20 0x7fbe481452a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:12.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.772+0000 7fbe46d9d640 1 -- 172.21.15.116:0/3868247659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe380094d0 con 0x7fbe48162490 2023-11-28T17:10:12.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.772+0000 7fbe4659c640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbe480b3940 0x7fbe48148620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:12.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.772+0000 7fbe3dd9b640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe480b3d20 0x7fbe481452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:12.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.772+0000 7fbe46d9d640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe48162490 0x7fbe48156b70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbe40007e90 tx=0x7fbe4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:12.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.773+0000 7fbe3f7fe640 1 -- 172.21.15.116:0/3868247659 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe40014420 con 0x7fbe48162490 2023-11-28T17:10:12.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.773+0000 7fbe3f7fe640 1 -- 172.21.15.116:0/3868247659 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe40014580 con 0x7fbe48162490 2023-11-28T17:10:12.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.773+0000 7fbe3f7fe640 1 -- 172.21.15.116:0/3868247659 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe40014930 con 0x7fbe48162490 2023-11-28T17:10:12.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.773+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3868247659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe481fafc0 con 0x7fbe48162490 2023-11-28T17:10:12.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.773+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3868247659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe481fb580 con 0x7fbe48162490 2023-11-28T17:10:12.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.774+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3868247659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe481fb740 con 0x7fbe48162490 2023-11-28T17:10:12.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.775+0000 7fbe3f7fe640 1 -- 172.21.15.116:0/3868247659 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fbe40014a90 con 0x7fbe48162490 2023-11-28T17:10:12.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.775+0000 7fbe3f7fe640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbe2806b2a0 0x7fbe2806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:12.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.776+0000 7fbe3f7fe640 1 -- 172.21.15.116:0/3868247659 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbe4008a200 con 0x7fbe48162490 2023-11-28T17:10:12.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.776+0000 7fbe4659c640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbe2806b2a0 0x7fbe2806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:12.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.777+0000 7fbe4659c640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbe2806b2a0 0x7fbe2806d760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbe38036000 tx=0x7fbe3803b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:12.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:12.780+0000 7fbe3f7fe640 1 -- 172.21.15.116:0/3868247659 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe40055320 con 0x7fbe48162490 2023-11-28T17:10:13.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.036+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3868247659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fbe480693d0 con 0x7fbe48162490 2023-11-28T17:10:13.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.040+0000 7fbe3f7fe640 1 -- 172.21.15.116:0/3868247659 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7fbe40058940 con 0x7fbe48162490 2023-11-28T17:10:13.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.043+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3868247659 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbe2806b2a0 msgr2=0x7fbe2806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:13.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.043+0000 7fbe4c92c640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbe2806b2a0 0x7fbe2806d760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbe38036000 tx=0x7fbe3803b040 comp rx=0 tx=0).stop 2023-11-28T17:10:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.043+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3868247659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe48162490 msgr2=0x7fbe48156b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.043+0000 7fbe4c92c640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe48162490 0x7fbe48156b70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbe40007e90 tx=0x7fbe4000e4b0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.043+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3868247659 shutdown_connections 2023-11-28T17:10:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.043+0000 7fbe4c92c640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbe2806b2a0 0x7fbe2806d760 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.043+0000 7fbe4c92c640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe48162490 0x7fbe48156b70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.043+0000 7fbe4c92c640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe480b3d20 0x7fbe481452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.043+0000 7fbe4c92c640 1 --2- 172.21.15.116:0/3868247659 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbe480b3940 0x7fbe48148620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.043+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3868247659 >> 172.21.15.116:0/3868247659 conn(0x7fbe4806a4c0 msgr2=0x7fbe4806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.043+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3868247659 shutdown_connections 2023-11-28T17:10:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.043+0000 7fbe4c92c640 1 -- 172.21.15.116:0/3868247659 wait complete. 2023-11-28T17:10:13.063 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-11-28T17:10:13.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.269+0000 7f36fdd63640 1 Processor -- start 2023-11-28T17:10:13.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.270+0000 7f36fdd63640 1 -- start start 2023-11-28T17:10:13.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.271+0000 7f36fdd63640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f36f80cdcf0 0x7f36f80cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.271+0000 7f36fdd63640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36f80cf210 0x7f36f80cfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.271+0000 7f36fdd63640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f80ced80 con 0x7f36f815d3b0 2023-11-28T17:10:13.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.271+0000 7f36fdd63640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f80ce170 con 0x7f36f80cdcf0 2023-11-28T17:10:13.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.271+0000 7f36fdd63640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80ce450 con 0x7f36f80cf210 2023-11-28T17:10:13.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.271+0000 7f36fcd61640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f36f815d3b0 0x7f36f80d18f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57718/0 (socket says 172.21.15.116:57718) 2023-11-28T17:10:13.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.271+0000 7f36fcd61640 1 -- 172.21.15.116:0/2101943166 learned_addr learned my addr 172.21.15.116:0/2101943166 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 -- 172.21.15.116:0/2101943166 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546260770 0 0) 0x7f36f80ced80 con 0x7f36f815d3b0 2023-11-28T17:10:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f7fff640 1 --2- 172.21.15.116:0/2101943166 >> v2:172.21.15.156:3300/0 conn(0x7f36f80cdcf0 0x7f36f80cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 -- 172.21.15.116:0/2101943166 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e4003520 con 0x7f36f815d3b0 2023-11-28T17:10:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36fd562640 1 --2- 172.21.15.116:0/2101943166 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36f80cf210 0x7f36f80cfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 -- 172.21.15.116:0/2101943166 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 823240355 0 0) 0x7f36e4003520 con 0x7f36f815d3b0 2023-11-28T17:10:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 -- 172.21.15.116:0/2101943166 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36f80cf210 msgr2=0x7f36f80cfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 --2- 172.21.15.116:0/2101943166 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36f80cf210 0x7f36f80cfdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 -- 172.21.15.116:0/2101943166 >> v2:172.21.15.156:3300/0 conn(0x7f36f80cdcf0 msgr2=0x7f36f80cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 --2- 172.21.15.116:0/2101943166 >> v2:172.21.15.156:3300/0 conn(0x7f36f80cdcf0 0x7f36f80cf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 -- 172.21.15.116:0/2101943166 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f80ce730 con 0x7f36f815d3b0 2023-11-28T17:10:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 -- 172.21.15.116:0/2101943166 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f36e80019f0 con 0x7f36f815d3b0 2023-11-28T17:10:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f7fff640 1 --2- 172.21.15.116:0/2101943166 >> v2:172.21.15.156:3300/0 conn(0x7f36f80cdcf0 0x7f36f80cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:10:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 -- 172.21.15.116:0/2101943166 >> v1:172.21.15.116:6789/0 conn(0x7f36f815d3b0 legacy=0x7f36f80d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 --2- 172.21.15.116:0/2101943166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36e4003970 0x7f36e4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 --2- 172.21.15.116:0/2101943166 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36f80cf210 0x7f36e40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 --2- 172.21.15.116:0/2101943166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e4004d00 0x7f36e4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 -- 172.21.15.116:0/2101943166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80ce170 con 0x7f36e4004d00 2023-11-28T17:10:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 -- 172.21.15.116:0/2101943166 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80ce450 con 0x7f36e4003970 2023-11-28T17:10:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36f77fe640 1 -- 172.21.15.116:0/2101943166 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e400ba30 con 0x7f36f80cf210 2023-11-28T17:10:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.272+0000 7f36fd562640 1 --2- 172.21.15.116:0/2101943166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e4004d00 0x7f36e4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.273+0000 7f36fcd61640 1 --2- 172.21.15.116:0/2101943166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36e4003970 0x7f36e4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.273+0000 7f36fd562640 1 -- 172.21.15.116:0/2101943166 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36f80cf210 msgr2=0x7f36e40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.273+0000 7f36fd562640 1 --2- 172.21.15.116:0/2101943166 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36f80cf210 0x7f36e40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.273+0000 7f36fd562640 1 -- 172.21.15.116:0/2101943166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36e4003970 msgr2=0x7f36e4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:13.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.273+0000 7f36fd562640 1 --2- 172.21.15.116:0/2101943166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36e4003970 0x7f36e4003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.273+0000 7f36fd562640 1 -- 172.21.15.116:0/2101943166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e4003520 con 0x7f36e4004d00 2023-11-28T17:10:13.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.274+0000 7f36fd562640 1 --2- 172.21.15.116:0/2101943166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e4004d00 0x7f36e4005180 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f36f000ba60 tx=0x7f36e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8489390587da9f95 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:13.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.274+0000 7f36f77fe640 1 -- 172.21.15.116:0/2101943166 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0032680 con 0x7f36e4004d00 2023-11-28T17:10:13.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.274+0000 7f36f77fe640 1 -- 172.21.15.116:0/2101943166 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f00327e0 con 0x7f36e4004d00 2023-11-28T17:10:13.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.274+0000 7f36f77fe640 1 -- 172.21.15.116:0/2101943166 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0032b70 con 0x7f36e4004d00 2023-11-28T17:10:13.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.275+0000 7f36fdd63640 1 -- 172.21.15.116:0/2101943166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e4004d00 msgr2=0x7f36e4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:13.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.275+0000 7f36fdd63640 1 --2- 172.21.15.116:0/2101943166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e4004d00 0x7f36e4005180 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f36f000ba60 tx=0x7f36e40057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.275+0000 7f36fd562640 1 -- 172.21.15.116:0/2101943166 reap_dead start 2023-11-28T17:10:13.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.275+0000 7f36fdd63640 1 -- 172.21.15.116:0/2101943166 shutdown_connections 2023-11-28T17:10:13.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.275+0000 7f36fdd63640 1 -- 172.21.15.116:0/2101943166 >> 172.21.15.116:0/2101943166 conn(0x7f36f80d2f00 msgr2=0x7f36f80d3310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:13.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.275+0000 7f36fdd63640 1 -- 172.21.15.116:0/2101943166 shutdown_connections 2023-11-28T17:10:13.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.275+0000 7f36fdd63640 1 -- 172.21.15.116:0/2101943166 wait complete. 2023-11-28T17:10:13.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.277+0000 7f36fdd63640 1 Processor -- start 2023-11-28T17:10:13.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.277+0000 7f36fdd63640 1 -- start start 2023-11-28T17:10:13.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.278+0000 7f36fdd63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e4004d00 0x7f36f80c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.278+0000 7f36fdd63640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36f8154250 0x7f36f80c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.278+0000 7f36fdd63640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36f80c8b70 0x7f36f815e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.278+0000 7f36fdd63640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80c9020 con 0x7f36e4004d00 2023-11-28T17:10:13.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.278+0000 7f36fdd63640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f815d9f0 con 0x7f36f80c8b70 2023-11-28T17:10:13.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.278+0000 7f36fdd63640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80ccb00 con 0x7f36f8154250 2023-11-28T17:10:13.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.278+0000 7f36fcd61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e4004d00 0x7f36f80c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:13.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.278+0000 7f36fcd61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e4004d00 0x7f36f80c80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60326/0 (socket says 172.21.15.116:60326) 2023-11-28T17:10:13.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.278+0000 7f36fcd61640 1 -- 172.21.15.116:0/1915213879 learned_addr learned my addr 172.21.15.116:0/1915213879 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:13.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.278+0000 7f36fcd61640 1 -- 172.21.15.116:0/1915213879 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36f8154250 msgr2=0x7f36f80c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:13.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.278+0000 7f36fcd61640 1 --2- 172.21.15.116:0/1915213879 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36f8154250 0x7f36f80c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.278+0000 7f36fcd61640 1 -- 172.21.15.116:0/1915213879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36f80c8b70 msgr2=0x7f36f815e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:13.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.278+0000 7f36fcd61640 1 --2- 172.21.15.116:0/1915213879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36f80c8b70 0x7f36f815e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.278+0000 7f36fcd61640 1 -- 172.21.15.116:0/1915213879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f80ccde0 con 0x7f36e4004d00 2023-11-28T17:10:13.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.279+0000 7f36fcd61640 1 --2- 172.21.15.116:0/1915213879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e4004d00 0x7f36f80c80f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f36e8001370 tx=0x7f36e800b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:13.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.279+0000 7f36f57fa640 1 -- 172.21.15.116:0/1915213879 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e8004d60 con 0x7f36e4004d00 2023-11-28T17:10:13.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.279+0000 7f36f57fa640 1 -- 172.21.15.116:0/1915213879 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e80070d0 con 0x7f36e4004d00 2023-11-28T17:10:13.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.279+0000 7f36fdd63640 1 -- 172.21.15.116:0/1915213879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f815dc90 con 0x7f36e4004d00 2023-11-28T17:10:13.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.279+0000 7f36f57fa640 1 -- 172.21.15.116:0/1915213879 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e8007440 con 0x7f36e4004d00 2023-11-28T17:10:13.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.279+0000 7f36fdd63640 1 -- 172.21.15.116:0/1915213879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f80cc7b0 con 0x7f36e4004d00 2023-11-28T17:10:13.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.280+0000 7f36fdd63640 1 -- 172.21.15.116:0/1915213879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f8067da0 con 0x7f36e4004d00 2023-11-28T17:10:13.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.281+0000 7f36f57fa640 1 -- 172.21.15.116:0/1915213879 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f36e80075a0 con 0x7f36e4004d00 2023-11-28T17:10:13.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.282+0000 7f36f57fa640 1 --2- 172.21.15.116:0/1915213879 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f36d806b2a0 0x7f36d806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.282+0000 7f36f57fa640 1 -- 172.21.15.116:0/1915213879 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f36e8089bb0 con 0x7f36e4004d00 2023-11-28T17:10:13.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.282+0000 7f36f7fff640 1 --2- 172.21.15.116:0/1915213879 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f36d806b2a0 0x7f36d806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.283+0000 7f36f7fff640 1 --2- 172.21.15.116:0/1915213879 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f36d806b2a0 0x7f36d806d760 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f36e0006fd0 tx=0x7f36e0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:13.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.287+0000 7f36f57fa640 1 -- 172.21.15.116:0/1915213879 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f36e8054e00 con 0x7f36e4004d00 2023-11-28T17:10:13.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.524+0000 7f36fdd63640 1 -- 172.21.15.116:0/1915213879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f36f815e1b0 con 0x7f36e4004d00 2023-11-28T17:10:13.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.528+0000 7f36f57fa640 1 -- 172.21.15.116:0/1915213879 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f36f815e1b0 con 0x7f36e4004d00 2023-11-28T17:10:13.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.530+0000 7f36fdd63640 1 -- 172.21.15.116:0/1915213879 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f36d806b2a0 msgr2=0x7f36d806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:13.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.530+0000 7f36fdd63640 1 --2- 172.21.15.116:0/1915213879 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f36d806b2a0 0x7f36d806d760 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f36e0006fd0 tx=0x7f36e0008040 comp rx=0 tx=0).stop 2023-11-28T17:10:13.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.530+0000 7f36fdd63640 1 -- 172.21.15.116:0/1915213879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e4004d00 msgr2=0x7f36f80c80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:13.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.530+0000 7f36fdd63640 1 --2- 172.21.15.116:0/1915213879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e4004d00 0x7f36f80c80f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f36e8001370 tx=0x7f36e800b560 comp rx=0 tx=0).stop 2023-11-28T17:10:13.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.530+0000 7f36fdd63640 1 -- 172.21.15.116:0/1915213879 shutdown_connections 2023-11-28T17:10:13.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.530+0000 7f36fdd63640 1 --2- 172.21.15.116:0/1915213879 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f36d806b2a0 0x7f36d806d760 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.530+0000 7f36fdd63640 1 --2- 172.21.15.116:0/1915213879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36f80c8b70 0x7f36f815e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.530+0000 7f36fdd63640 1 --2- 172.21.15.116:0/1915213879 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36f8154250 0x7f36f80c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.531+0000 7f36fdd63640 1 --2- 172.21.15.116:0/1915213879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e4004d00 0x7f36f80c80f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.531+0000 7f36fdd63640 1 -- 172.21.15.116:0/1915213879 >> 172.21.15.116:0/1915213879 conn(0x7f36f80d2f00 msgr2=0x7f36f80b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.531+0000 7f36fdd63640 1 -- 172.21.15.116:0/1915213879 shutdown_connections 2023-11-28T17:10:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.531+0000 7f36fdd63640 1 -- 172.21.15.116:0/1915213879 wait complete. 2023-11-28T17:10:13.551 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-11-28T17:10:13.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.761+0000 7f4480eda640 1 Processor -- start 2023-11-28T17:10:13.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.761+0000 7f4480eda640 1 -- start start 2023-11-28T17:10:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.763+0000 7f4480eda640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f447c146a50 0x7f447c161a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.763+0000 7f4480eda640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f447c0b32b0 0x7f447c0cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.763+0000 7f4480eda640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f447c0cadf0 con 0x7f447c0cd970 2023-11-28T17:10:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.763+0000 7f4480eda640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f447c0ce7c0 con 0x7f447c0b32b0 2023-11-28T17:10:13.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.763+0000 7f4480eda640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f447c0ce4e0 con 0x7f447c146a50 2023-11-28T17:10:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.764+0000 7f447a59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f447c146a50 0x7f447c161a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.764+0000 7f4479d9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f447c0cd970 0x7f447c0b2ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.116:58984/0 (socket says 172.21.15.116:58984) 2023-11-28T17:10:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.764+0000 7f4479d9b640 1 -- 172.21.15.116:0/3940381115 learned_addr learned my addr 172.21.15.116:0/3940381115 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:13.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.764+0000 7f447ad9d640 1 --2- 172.21.15.116:0/3940381115 >> v2:172.21.15.156:3300/0 conn(0x7f447c0b32b0 0x7f447c0cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.765+0000 7f447ad9d640 1 -- 172.21.15.116:0/3940381115 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f447c146a50 msgr2=0x7f447c161a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.765+0000 7f447ad9d640 1 --2- 172.21.15.116:0/3940381115 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f447c146a50 0x7f447c161a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.765+0000 7f447ad9d640 1 -- 172.21.15.116:0/3940381115 >> v1:172.21.15.156:6789/0 conn(0x7f447c0cd970 legacy=0x7f447c0b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.765+0000 7f447ad9d640 1 -- 172.21.15.116:0/3940381115 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c160fa0 con 0x7f447c0b32b0 2023-11-28T17:10:13.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.765+0000 7f447a59c640 1 --2- 172.21.15.116:0/3940381115 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f447c146a50 0x7f447c161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:10:13.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.766+0000 7f447ad9d640 1 --2- 172.21.15.116:0/3940381115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f447c0b32b0 0x7f447c0cddd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f44740097a0 tx=0x7f447402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b626191a25efc7b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:13.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.767+0000 7f447959a640 1 -- 172.21.15.116:0/3940381115 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44740316f0 con 0x7f447c0b32b0 2023-11-28T17:10:13.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.767+0000 7f447959a640 1 -- 172.21.15.116:0/3940381115 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4474031850 con 0x7f447c0b32b0 2023-11-28T17:10:13.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.767+0000 7f447959a640 1 -- 172.21.15.116:0/3940381115 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4474031be0 con 0x7f447c0b32b0 2023-11-28T17:10:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.767+0000 7f4480eda640 1 -- 172.21.15.116:0/3940381115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f447c0b32b0 msgr2=0x7f447c0cddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.767+0000 7f4480eda640 1 --2- 172.21.15.116:0/3940381115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f447c0b32b0 0x7f447c0cddd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f44740097a0 tx=0x7f447402b790 comp rx=0 tx=0).stop 2023-11-28T17:10:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.767+0000 7f4480eda640 1 -- 172.21.15.116:0/3940381115 shutdown_connections 2023-11-28T17:10:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.767+0000 7f4480eda640 1 --2- 172.21.15.116:0/3940381115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f447c0b32b0 0x7f447c0cddd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.767+0000 7f4480eda640 1 --2- 172.21.15.116:0/3940381115 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f447c146a50 0x7f447c161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.767+0000 7f4480eda640 1 -- 172.21.15.116:0/3940381115 >> 172.21.15.116:0/3940381115 conn(0x7f447c06a4c0 msgr2=0x7f447c06a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.768+0000 7f4480eda640 1 -- 172.21.15.116:0/3940381115 shutdown_connections 2023-11-28T17:10:13.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.768+0000 7f4480eda640 1 -- 172.21.15.116:0/3940381115 wait complete. 2023-11-28T17:10:13.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.769+0000 7f4480eda640 1 Processor -- start 2023-11-28T17:10:13.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.769+0000 7f4480eda640 1 -- start start 2023-11-28T17:10:13.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.770+0000 7f4480eda640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f447c0b32b0 0x7f447c1403b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.770+0000 7f4480eda640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f447c0cd970 0x7f447c141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.770+0000 7f4480eda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f447c146a50 0x7f447c0a9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.770+0000 7f4480eda640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c0a9650 con 0x7f447c146a50 2023-11-28T17:10:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.770+0000 7f4480eda640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c140a20 con 0x7f447c0cd970 2023-11-28T17:10:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.770+0000 7f4480eda640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f447c140d00 con 0x7f447c0b32b0 2023-11-28T17:10:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.771+0000 7f447ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f447c146a50 0x7f447c0a9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.771+0000 7f447ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f447c146a50 0x7f447c0a9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60334/0 (socket says 172.21.15.116:60334) 2023-11-28T17:10:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.771+0000 7f447ad9d640 1 -- 172.21.15.116:0/1511792649 learned_addr learned my addr 172.21.15.116:0/1511792649 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.771+0000 7f447a59c640 1 --2- 172.21.15.116:0/1511792649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f447c0b32b0 0x7f447c1403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.771+0000 7f4479d9b640 1 --2- 172.21.15.116:0/1511792649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f447c0cd970 0x7f447c141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.771+0000 7f447ad9d640 1 -- 172.21.15.116:0/1511792649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f447c0b32b0 msgr2=0x7f447c1403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.771+0000 7f447ad9d640 1 --2- 172.21.15.116:0/1511792649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f447c0b32b0 0x7f447c1403b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.771+0000 7f447ad9d640 1 -- 172.21.15.116:0/1511792649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f447c0cd970 msgr2=0x7f447c141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.771+0000 7f447ad9d640 1 --2- 172.21.15.116:0/1511792649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f447c0cd970 0x7f447c141ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.771+0000 7f447ad9d640 1 -- 172.21.15.116:0/1511792649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c009a70 con 0x7f447c146a50 2023-11-28T17:10:13.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.772+0000 7f447ad9d640 1 --2- 172.21.15.116:0/1511792649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f447c146a50 0x7f447c0a9110 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f44740097a0 tx=0x7f44740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:13.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.772+0000 7f4472ffd640 1 -- 172.21.15.116:0/1511792649 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4474034680 con 0x7f447c146a50 2023-11-28T17:10:13.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.772+0000 7f4472ffd640 1 -- 172.21.15.116:0/1511792649 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44740347e0 con 0x7f447c146a50 2023-11-28T17:10:13.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.772+0000 7f4480eda640 1 -- 172.21.15.116:0/1511792649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4474009400 con 0x7f447c146a50 2023-11-28T17:10:13.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.772+0000 7f4472ffd640 1 -- 172.21.15.116:0/1511792649 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4474034b70 con 0x7f447c146a50 2023-11-28T17:10:13.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.772+0000 7f4480eda640 1 -- 172.21.15.116:0/1511792649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447c1412c0 con 0x7f447c146a50 2023-11-28T17:10:13.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.773+0000 7f4480eda640 1 -- 172.21.15.116:0/1511792649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f447c0b71f0 con 0x7f447c146a50 2023-11-28T17:10:13.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.774+0000 7f4472ffd640 1 -- 172.21.15.116:0/1511792649 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f447403b040 con 0x7f447c146a50 2023-11-28T17:10:13.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.775+0000 7f4472ffd640 1 --2- 172.21.15.116:0/1511792649 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f445c06b2a0 0x7f445c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:13.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.775+0000 7f4472ffd640 1 -- 172.21.15.116:0/1511792649 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f44740aae20 con 0x7f447c146a50 2023-11-28T17:10:13.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.775+0000 7f447a59c640 1 --2- 172.21.15.116:0/1511792649 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f445c06b2a0 0x7f445c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.776+0000 7f447a59c640 1 --2- 172.21.15.116:0/1511792649 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f445c06b2a0 0x7f445c06d760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f446c007560 tx=0x7f446c00b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:13.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:13.779+0000 7f4472ffd640 1 -- 172.21.15.116:0/1511792649 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44740760f0 con 0x7f447c146a50 2023-11-28T17:10:14.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.037+0000 7f4480eda640 1 -- 172.21.15.116:0/1511792649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f447c0693d0 con 0x7f447c146a50 2023-11-28T17:10:14.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.041+0000 7f4472ffd640 1 -- 172.21.15.116:0/1511792649 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4474079710 con 0x7f447c146a50 2023-11-28T17:10:14.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.044+0000 7f4480eda640 1 -- 172.21.15.116:0/1511792649 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f445c06b2a0 msgr2=0x7f445c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:14.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.044+0000 7f4480eda640 1 --2- 172.21.15.116:0/1511792649 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f445c06b2a0 0x7f445c06d760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f446c007560 tx=0x7f446c00b040 comp rx=0 tx=0).stop 2023-11-28T17:10:14.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.044+0000 7f4480eda640 1 -- 172.21.15.116:0/1511792649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f447c146a50 msgr2=0x7f447c0a9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:14.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.044+0000 7f4480eda640 1 --2- 172.21.15.116:0/1511792649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f447c146a50 0x7f447c0a9110 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f44740097a0 tx=0x7f44740046c0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.044+0000 7f4480eda640 1 -- 172.21.15.116:0/1511792649 shutdown_connections 2023-11-28T17:10:14.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.044+0000 7f4480eda640 1 --2- 172.21.15.116:0/1511792649 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f445c06b2a0 0x7f445c06d760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.044+0000 7f4480eda640 1 --2- 172.21.15.116:0/1511792649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f447c146a50 0x7f447c0a9110 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.044+0000 7f4480eda640 1 --2- 172.21.15.116:0/1511792649 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f447c0cd970 0x7f447c141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.044+0000 7f4480eda640 1 --2- 172.21.15.116:0/1511792649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f447c0b32b0 0x7f447c1403b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.044+0000 7f4480eda640 1 -- 172.21.15.116:0/1511792649 >> 172.21.15.116:0/1511792649 conn(0x7f447c06a4c0 msgr2=0x7f447c0cd140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:14.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.044+0000 7f4480eda640 1 -- 172.21.15.116:0/1511792649 shutdown_connections 2023-11-28T17:10:14.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.044+0000 7f4480eda640 1 -- 172.21.15.116:0/1511792649 wait complete. 2023-11-28T17:10:14.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-11-28T17:10:14.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.257+0000 7fdd95231640 1 Processor -- start 2023-11-28T17:10:14.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.257+0000 7fdd95231640 1 -- start start 2023-11-28T17:10:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.259+0000 7fdd95231640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdd901401c0 0x7fdd90071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.259+0000 7fdd95231640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdd9006c960 0x7fdd900b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.259+0000 7fdd95231640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd901405a0 con 0x7fdd90071710 2023-11-28T17:10:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.259+0000 7fdd95231640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd90140880 con 0x7fdd9006c960 2023-11-28T17:10:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.259+0000 7fdd95231640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd9015db00 con 0x7fdd901401c0 2023-11-28T17:10:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.259+0000 7fdd8659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdd90071710 0x7fdd9006c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57730/0 (socket says 172.21.15.116:57730) 2023-11-28T17:10:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.259+0000 7fdd8659c640 1 -- 172.21.15.116:0/3390109593 learned_addr learned my addr 172.21.15.116:0/3390109593 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.260+0000 7fdd8f59e640 1 --2- 172.21.15.116:0/3390109593 >> v2:172.21.15.116:3300/0 conn(0x7fdd9006c960 0x7fdd900b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.260+0000 7fdd8ed9d640 1 --2- 172.21.15.116:0/3390109593 >> v2:172.21.15.156:3300/0 conn(0x7fdd901401c0 0x7fdd90071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.260+0000 7fdd8e59c640 1 -- 172.21.15.116:0/3390109593 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739042249 0 0) 0x7fdd901405a0 con 0x7fdd90071710 2023-11-28T17:10:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.260+0000 7fdd8e59c640 1 -- 172.21.15.116:0/3390109593 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd7c003520 con 0x7fdd90071710 2023-11-28T17:10:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.260+0000 7fdd8f59e640 1 -- 172.21.15.116:0/3390109593 >> v2:172.21.15.156:3300/0 conn(0x7fdd901401c0 msgr2=0x7fdd90071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.260+0000 7fdd8f59e640 1 --2- 172.21.15.116:0/3390109593 >> v2:172.21.15.156:3300/0 conn(0x7fdd901401c0 0x7fdd90071eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.260+0000 7fdd8f59e640 1 -- 172.21.15.116:0/3390109593 >> v1:172.21.15.116:6789/0 conn(0x7fdd90071710 legacy=0x7fdd9006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.260+0000 7fdd8e59c640 1 -- 172.21.15.116:0/3390109593 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1517943445 0 0) 0x7fdd7c003520 con 0x7fdd90071710 2023-11-28T17:10:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.260+0000 7fdd8f59e640 1 -- 172.21.15.116:0/3390109593 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9015dde0 con 0x7fdd9006c960 2023-11-28T17:10:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.261+0000 7fdd8f59e640 1 --2- 172.21.15.116:0/3390109593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9006c960 0x7fdd900b3430 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdd9006bdd0 tx=0x7fdd8802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f77512d870917c3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.261+0000 7fdd8e59c640 1 -- 172.21.15.116:0/3390109593 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd78001670 con 0x7fdd9006c960 2023-11-28T17:10:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.261+0000 7fdd8e59c640 1 -- 172.21.15.116:0/3390109593 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8802c8c0 con 0x7fdd9006c960 2023-11-28T17:10:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.261+0000 7fdd8e59c640 1 -- 172.21.15.116:0/3390109593 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8802cc50 con 0x7fdd9006c960 2023-11-28T17:10:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.262+0000 7fdd95231640 1 -- 172.21.15.116:0/3390109593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9006c960 msgr2=0x7fdd900b3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.262+0000 7fdd95231640 1 --2- 172.21.15.116:0/3390109593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9006c960 0x7fdd900b3430 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdd9006bdd0 tx=0x7fdd8802bd90 comp rx=0 tx=0).stop 2023-11-28T17:10:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.262+0000 7fdd95231640 1 -- 172.21.15.116:0/3390109593 shutdown_connections 2023-11-28T17:10:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.262+0000 7fdd95231640 1 --2- 172.21.15.116:0/3390109593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd9006c960 0x7fdd900b3430 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.262+0000 7fdd95231640 1 --2- 172.21.15.116:0/3390109593 >> v2:172.21.15.156:3300/0 conn(0x7fdd901401c0 0x7fdd90071eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.262+0000 7fdd95231640 1 -- 172.21.15.116:0/3390109593 >> 172.21.15.116:0/3390109593 conn(0x7fdd9006a4c0 msgr2=0x7fdd9006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.262+0000 7fdd95231640 1 -- 172.21.15.116:0/3390109593 shutdown_connections 2023-11-28T17:10:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.262+0000 7fdd95231640 1 -- 172.21.15.116:0/3390109593 wait complete. 2023-11-28T17:10:14.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.264+0000 7fdd95231640 1 Processor -- start 2023-11-28T17:10:14.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.264+0000 7fdd95231640 1 -- start start 2023-11-28T17:10:14.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.265+0000 7fdd95231640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd9006c960 0x7fdd90156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.265+0000 7fdd95231640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd90071710 0x7fdd90159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.265+0000 7fdd95231640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd901401c0 0x7fdd90157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.265+0000 7fdd95231640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd901592e0 con 0x7fdd90071710 2023-11-28T17:10:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.265+0000 7fdd95231640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd901586d0 con 0x7fdd9006c960 2023-11-28T17:10:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.265+0000 7fdd95231640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd901589b0 con 0x7fdd901401c0 2023-11-28T17:10:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.265+0000 7fdd8659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd90071710 0x7fdd90159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:14.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.265+0000 7fdd8659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd90071710 0x7fdd90159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60358/0 (socket says 172.21.15.116:60358) 2023-11-28T17:10:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.265+0000 7fdd8659c640 1 -- 172.21.15.116:0/489570644 learned_addr learned my addr 172.21.15.116:0/489570644 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.265+0000 7fdd8659c640 1 -- 172.21.15.116:0/489570644 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd901401c0 msgr2=0x7fdd90157e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.265+0000 7fdd8659c640 1 --2- 172.21.15.116:0/489570644 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd901401c0 0x7fdd90157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.265+0000 7fdd8659c640 1 -- 172.21.15.116:0/489570644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd9006c960 msgr2=0x7fdd90156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.265+0000 7fdd8659c640 1 --2- 172.21.15.116:0/489570644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd9006c960 0x7fdd90156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.265+0000 7fdd8659c640 1 -- 172.21.15.116:0/489570644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd88009a70 con 0x7fdd90071710 2023-11-28T17:10:14.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.266+0000 7fdd8659c640 1 --2- 172.21.15.116:0/489570644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd90071710 0x7fdd90159770 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800c760 tx=0x7fdd7800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.266+0000 7fdd87fff640 1 -- 172.21.15.116:0/489570644 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd78005ad0 con 0x7fdd90071710 2023-11-28T17:10:14.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.266+0000 7fdd87fff640 1 -- 172.21.15.116:0/489570644 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd78005c30 con 0x7fdd90071710 2023-11-28T17:10:14.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.266+0000 7fdd95231640 1 -- 172.21.15.116:0/489570644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd90158c90 con 0x7fdd90071710 2023-11-28T17:10:14.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.266+0000 7fdd87fff640 1 -- 172.21.15.116:0/489570644 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd78007070 con 0x7fdd90071710 2023-11-28T17:10:14.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.266+0000 7fdd95231640 1 -- 172.21.15.116:0/489570644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd901583c0 con 0x7fdd90071710 2023-11-28T17:10:14.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.267+0000 7fdd95231640 1 -- 172.21.15.116:0/489570644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd900693d0 con 0x7fdd90071710 2023-11-28T17:10:14.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.268+0000 7fdd87fff640 1 -- 172.21.15.116:0/489570644 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fdd780071d0 con 0x7fdd90071710 2023-11-28T17:10:14.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.269+0000 7fdd87fff640 1 --2- 172.21.15.116:0/489570644 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdd7006b2a0 0x7fdd7006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:14.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.269+0000 7fdd87fff640 1 -- 172.21.15.116:0/489570644 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdd78089720 con 0x7fdd90071710 2023-11-28T17:10:14.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.269+0000 7fdd8ed9d640 1 --2- 172.21.15.116:0/489570644 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdd7006b2a0 0x7fdd7006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:14.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.270+0000 7fdd8ed9d640 1 --2- 172.21.15.116:0/489570644 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdd7006b2a0 0x7fdd7006d760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdd80006a90 tx=0x7fdd80006a20 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:14.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.273+0000 7fdd87fff640 1 -- 172.21.15.116:0/489570644 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd780547b0 con 0x7fdd90071710 2023-11-28T17:10:14.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.527+0000 7fdd95231640 1 -- 172.21.15.116:0/489570644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fdd9015d750 con 0x7fdd90071710 2023-11-28T17:10:14.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.531+0000 7fdd87fff640 1 -- 172.21.15.116:0/489570644 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7fdd78057dd0 con 0x7fdd90071710 2023-11-28T17:10:14.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.533+0000 7fdd95231640 1 -- 172.21.15.116:0/489570644 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdd7006b2a0 msgr2=0x7fdd7006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:14.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.533+0000 7fdd95231640 1 --2- 172.21.15.116:0/489570644 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdd7006b2a0 0x7fdd7006d760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdd80006a90 tx=0x7fdd80006a20 comp rx=0 tx=0).stop 2023-11-28T17:10:14.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.533+0000 7fdd95231640 1 -- 172.21.15.116:0/489570644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd90071710 msgr2=0x7fdd90159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:14.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.533+0000 7fdd95231640 1 --2- 172.21.15.116:0/489570644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd90071710 0x7fdd90159770 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800c760 tx=0x7fdd7800cc20 comp rx=0 tx=0).stop 2023-11-28T17:10:14.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.534+0000 7fdd95231640 1 -- 172.21.15.116:0/489570644 shutdown_connections 2023-11-28T17:10:14.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.534+0000 7fdd95231640 1 --2- 172.21.15.116:0/489570644 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdd7006b2a0 0x7fdd7006d760 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.534+0000 7fdd95231640 1 --2- 172.21.15.116:0/489570644 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd901401c0 0x7fdd90157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.534+0000 7fdd95231640 1 --2- 172.21.15.116:0/489570644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd90071710 0x7fdd90159770 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.534+0000 7fdd95231640 1 --2- 172.21.15.116:0/489570644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd9006c960 0x7fdd90156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.534+0000 7fdd95231640 1 -- 172.21.15.116:0/489570644 >> 172.21.15.116:0/489570644 conn(0x7fdd9006a4c0 msgr2=0x7fdd9006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:14.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.534+0000 7fdd95231640 1 -- 172.21.15.116:0/489570644 shutdown_connections 2023-11-28T17:10:14.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.534+0000 7fdd95231640 1 -- 172.21.15.116:0/489570644 wait complete. 2023-11-28T17:10:14.552 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-11-28T17:10:14.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.755+0000 7f279271d640 1 Processor -- start 2023-11-28T17:10:14.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.755+0000 7f279271d640 1 -- start start 2023-11-28T17:10:14.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.757+0000 7f279271d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f278c144430 0x7f278c15f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:14.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.757+0000 7f279271d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f278c06c250 0x7f278c06c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:14.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.757+0000 7f279271d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c070bd0 con 0x7f278c0703f0 2023-11-28T17:10:14.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.757+0000 7f279271d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f278c144810 con 0x7f278c144430 2023-11-28T17:10:14.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.757+0000 7f279271d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f278c06cc20 con 0x7f278c06c250 2023-11-28T17:10:14.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.757+0000 7f278bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f278c144430 0x7f278c15f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:14.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.757+0000 7f27837fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f278c0703f0 0x7f278c071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57744/0 (socket says 172.21.15.116:57744) 2023-11-28T17:10:14.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.757+0000 7f27837fe640 1 -- 172.21.15.116:0/773908327 learned_addr learned my addr 172.21.15.116:0/773908327 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:14.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.758+0000 7f2790cba640 1 --2- 172.21.15.116:0/773908327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f278c06c250 0x7f278c06c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:14.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.758+0000 7f278b7fe640 1 -- 172.21.15.116:0/773908327 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69544972 0 0) 0x7f278c070bd0 con 0x7f278c0703f0 2023-11-28T17:10:14.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.758+0000 7f278b7fe640 1 -- 172.21.15.116:0/773908327 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2778003520 con 0x7f278c0703f0 2023-11-28T17:10:14.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.758+0000 7f278bfff640 1 -- 172.21.15.116:0/773908327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f278c06c250 msgr2=0x7f278c06c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:14.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.758+0000 7f278bfff640 1 --2- 172.21.15.116:0/773908327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f278c06c250 0x7f278c06c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.758+0000 7f278bfff640 1 -- 172.21.15.116:0/773908327 >> v1:172.21.15.116:6789/0 conn(0x7f278c0703f0 legacy=0x7f278c071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:14.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.758+0000 7f278bfff640 1 -- 172.21.15.116:0/773908327 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c070eb0 con 0x7f278c144430 2023-11-28T17:10:14.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.758+0000 7f2790cba640 1 --2- 172.21.15.116:0/773908327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f278c06c250 0x7f278c06c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:14.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.759+0000 7f278bfff640 1 --2- 172.21.15.116:0/773908327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f278c144430 0x7f278c15f7b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f277c009d90 tx=0x7f277c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972b2b13c9881663 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.759+0000 7f278b7fe640 1 -- 172.21.15.116:0/773908327 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c02cc50 con 0x7f278c144430 2023-11-28T17:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.759+0000 7f278b7fe640 1 -- 172.21.15.116:0/773908327 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277c02cdb0 con 0x7f278c144430 2023-11-28T17:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.759+0000 7f278b7fe640 1 -- 172.21.15.116:0/773908327 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c034660 con 0x7f278c144430 2023-11-28T17:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.760+0000 7f279271d640 1 -- 172.21.15.116:0/773908327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f278c144430 msgr2=0x7f278c15f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.760+0000 7f279271d640 1 --2- 172.21.15.116:0/773908327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f278c144430 0x7f278c15f7b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f277c009d90 tx=0x7f277c02bea0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.760+0000 7f279271d640 1 -- 172.21.15.116:0/773908327 shutdown_connections 2023-11-28T17:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.760+0000 7f279271d640 1 --2- 172.21.15.116:0/773908327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f278c06c250 0x7f278c06c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.760+0000 7f279271d640 1 --2- 172.21.15.116:0/773908327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f278c144430 0x7f278c15f7b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.760+0000 7f279271d640 1 -- 172.21.15.116:0/773908327 >> 172.21.15.116:0/773908327 conn(0x7f278c06a4c0 msgr2=0x7f278c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.760+0000 7f279271d640 1 -- 172.21.15.116:0/773908327 shutdown_connections 2023-11-28T17:10:14.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.760+0000 7f279271d640 1 -- 172.21.15.116:0/773908327 wait complete. 2023-11-28T17:10:14.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.762+0000 7f279271d640 1 Processor -- start 2023-11-28T17:10:14.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.762+0000 7f279271d640 1 -- start start 2023-11-28T17:10:14.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.763+0000 7f279271d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f278c06c250 0x7f278c1f4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:14.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.763+0000 7f279271d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f278c0703f0 0x7f278c13bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:14.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.763+0000 7f279271d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f278c144430 0x7f278c15a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:14.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.763+0000 7f279271d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c1e4430 con 0x7f278c06c250 2023-11-28T17:10:14.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.763+0000 7f279271d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c1e4710 con 0x7f278c144430 2023-11-28T17:10:14.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.763+0000 7f279271d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f278c1f4890 con 0x7f278c0703f0 2023-11-28T17:10:14.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.764+0000 7f278bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f278c06c250 0x7f278c1f4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:14.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.764+0000 7f278bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f278c06c250 0x7f278c1f4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60368/0 (socket says 172.21.15.116:60368) 2023-11-28T17:10:14.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.764+0000 7f278bfff640 1 -- 172.21.15.116:0/35753062 learned_addr learned my addr 172.21.15.116:0/35753062 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:14.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.764+0000 7f27837fe640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f278c0703f0 0x7f278c13bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.764+0000 7f2790cba640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f278c144430 0x7f278c15a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.764+0000 7f278bfff640 1 -- 172.21.15.116:0/35753062 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f278c0703f0 msgr2=0x7f278c13bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.764+0000 7f278bfff640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f278c0703f0 0x7f278c13bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.764+0000 7f278bfff640 1 -- 172.21.15.116:0/35753062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f278c144430 msgr2=0x7f278c15a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.764+0000 7f278bfff640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f278c144430 0x7f278c15a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.764+0000 7f278bfff640 1 -- 172.21.15.116:0/35753062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f277c009a70 con 0x7f278c06c250 2023-11-28T17:10:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.764+0000 7f2790cba640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f278c144430 0x7f278c15a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.764+0000 7f278bfff640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f278c06c250 0x7f278c1f4350 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f277c037040 tx=0x7f277c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.765+0000 7f27897fa640 1 -- 172.21.15.116:0/35753062 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c040650 con 0x7f278c06c250 2023-11-28T17:10:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.765+0000 7f27897fa640 1 -- 172.21.15.116:0/35753062 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277c0407b0 con 0x7f278c06c250 2023-11-28T17:10:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.765+0000 7f27897fa640 1 -- 172.21.15.116:0/35753062 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c040b60 con 0x7f278c06c250 2023-11-28T17:10:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.765+0000 7f279271d640 1 -- 172.21.15.116:0/35753062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278c1e5e50 con 0x7f278c06c250 2023-11-28T17:10:14.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.765+0000 7f279271d640 1 -- 172.21.15.116:0/35753062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c1f4e10 con 0x7f278c06c250 2023-11-28T17:10:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.766+0000 7f279271d640 1 -- 172.21.15.116:0/35753062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278c0693d0 con 0x7f278c06c250 2023-11-28T17:10:14.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.767+0000 7f27897fa640 1 -- 172.21.15.116:0/35753062 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f277c040cc0 con 0x7f278c06c250 2023-11-28T17:10:14.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.768+0000 7f27897fa640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f276c06b2a0 0x7f276c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:14.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.768+0000 7f27897fa640 1 -- 172.21.15.116:0/35753062 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f277c0ac600 con 0x7f278c06c250 2023-11-28T17:10:14.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.768+0000 7f27837fe640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f276c06b2a0 0x7f276c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:14.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.769+0000 7f27837fe640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f276c06b2a0 0x7f276c06d760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f278c0cb8e0 tx=0x7f2774009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:14.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:14.772+0000 7f27897fa640 1 -- 172.21.15.116:0/35753062 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f277c077850 con 0x7f278c06c250 2023-11-28T17:10:15.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.023+0000 7f279271d640 1 -- 172.21.15.116:0/35753062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f278c06d100 con 0x7f278c06c250 2023-11-28T17:10:15.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.027+0000 7f27897fa640 1 -- 172.21.15.116:0/35753062 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f277c07ae70 con 0x7f278c06c250 2023-11-28T17:10:15.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.030+0000 7f279271d640 1 -- 172.21.15.116:0/35753062 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f276c06b2a0 msgr2=0x7f276c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:15.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.030+0000 7f279271d640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f276c06b2a0 0x7f276c06d760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f278c0cb8e0 tx=0x7f2774009040 comp rx=0 tx=0).stop 2023-11-28T17:10:15.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.030+0000 7f279271d640 1 -- 172.21.15.116:0/35753062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f278c06c250 msgr2=0x7f278c1f4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:15.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.030+0000 7f279271d640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f278c06c250 0x7f278c1f4350 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f277c037040 tx=0x7f277c004a40 comp rx=0 tx=0).stop 2023-11-28T17:10:15.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.030+0000 7f279271d640 1 -- 172.21.15.116:0/35753062 shutdown_connections 2023-11-28T17:10:15.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.030+0000 7f279271d640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f276c06b2a0 0x7f276c06d760 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.030+0000 7f279271d640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f278c144430 0x7f278c15a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.030+0000 7f279271d640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f278c0703f0 0x7f278c13bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.030+0000 7f279271d640 1 --2- 172.21.15.116:0/35753062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f278c06c250 0x7f278c1f4350 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.030+0000 7f279271d640 1 -- 172.21.15.116:0/35753062 >> 172.21.15.116:0/35753062 conn(0x7f278c06a4c0 msgr2=0x7f278c143280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:15.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.030+0000 7f279271d640 1 -- 172.21.15.116:0/35753062 shutdown_connections 2023-11-28T17:10:15.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.030+0000 7f279271d640 1 -- 172.21.15.116:0/35753062 wait complete. 2023-11-28T17:10:15.049 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-11-28T17:10:15.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.258+0000 7f3e65415640 1 Processor -- start 2023-11-28T17:10:15.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.259+0000 7f3e65415640 1 -- start start 2023-11-28T17:10:15.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.261+0000 7f3e65415640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e6015dab0 0x7f3e600703f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:15.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.261+0000 7f3e65415640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e60154220 con 0x7f3e600b3310 2023-11-28T17:10:15.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.261+0000 7f3e65415640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6015d2f0 con 0x7f3e6006cae0 2023-11-28T17:10:15.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.261+0000 7f3e65415640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e6015a3c0 con 0x7f3e6015dab0 2023-11-28T17:10:15.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.261+0000 7f3e5f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e6015dab0 0x7f3e600703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:15.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.261+0000 7f3e567fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3e600b3310 0x7f3e60159cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57760/0 (socket says 172.21.15.116:57760) 2023-11-28T17:10:15.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.261+0000 7f3e567fc640 1 -- 172.21.15.116:0/3673360628 learned_addr learned my addr 172.21.15.116:0/3673360628 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:15.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.262+0000 7f3e5f7fe640 1 -- 172.21.15.116:0/3673360628 >> v1:172.21.15.156:6789/0 conn(0x7f3e6006cae0 legacy=0x7f3e6006cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:15.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.262+0000 7f3e5f7fe640 1 -- 172.21.15.116:0/3673360628 >> v1:172.21.15.116:6789/0 conn(0x7f3e600b3310 legacy=0x7f3e60159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:15.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.262+0000 7f3e5f7fe640 1 -- 172.21.15.116:0/3673360628 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6015a5c0 con 0x7f3e6015dab0 2023-11-28T17:10:15.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.262+0000 7f3e5e7fc640 1 -- 172.21.15.116:0/3673360628 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886304359 0 0) 0x7f3e50001110 con 0x7f3e600b3310 2023-11-28T17:10:15.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.263+0000 7f3e5f7fe640 1 --2- 172.21.15.116:0/3673360628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e6015dab0 0x7f3e600703f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3e58009d90 tx=0x7f3e5802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49991dc0207d2b97 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:15.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.263+0000 7f3e5e7fc640 1 -- 172.21.15.116:0/3673360628 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5802cc60 con 0x7f3e6015dab0 2023-11-28T17:10:15.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.263+0000 7f3e5e7fc640 1 -- 172.21.15.116:0/3673360628 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5802cdc0 con 0x7f3e6015dab0 2023-11-28T17:10:15.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.263+0000 7f3e5e7fc640 1 -- 172.21.15.116:0/3673360628 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e58034660 con 0x7f3e6015dab0 2023-11-28T17:10:15.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.264+0000 7f3e65415640 1 -- 172.21.15.116:0/3673360628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e6015dab0 msgr2=0x7f3e600703f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:15.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.264+0000 7f3e65415640 1 --2- 172.21.15.116:0/3673360628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e6015dab0 0x7f3e600703f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3e58009d90 tx=0x7f3e5802beb0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.264+0000 7f3e65415640 1 -- 172.21.15.116:0/3673360628 shutdown_connections 2023-11-28T17:10:15.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.264+0000 7f3e65415640 1 --2- 172.21.15.116:0/3673360628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e6015dab0 0x7f3e600703f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.264+0000 7f3e65415640 1 -- 172.21.15.116:0/3673360628 >> 172.21.15.116:0/3673360628 conn(0x7f3e6006a4c0 msgr2=0x7f3e6006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:15.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.264+0000 7f3e65415640 1 -- 172.21.15.116:0/3673360628 shutdown_connections 2023-11-28T17:10:15.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.264+0000 7f3e65415640 1 -- 172.21.15.116:0/3673360628 wait complete. 2023-11-28T17:10:15.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.266+0000 7f3e65415640 1 Processor -- start 2023-11-28T17:10:15.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.266+0000 7f3e65415640 1 -- start start 2023-11-28T17:10:15.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.267+0000 7f3e65415640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e6006cae0 0x7f3e6014a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.267+0000 7f3e65415640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e600b3310 0x7f3e601591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.267+0000 7f3e65415640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e6015dab0 0x7f3e600703f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.267+0000 7f3e65415640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e60154430 con 0x7f3e6015dab0 2023-11-28T17:10:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.267+0000 7f3e65415640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e601f6850 con 0x7f3e600b3310 2023-11-28T17:10:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.267+0000 7f3e65415640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e601f6b30 con 0x7f3e6006cae0 2023-11-28T17:10:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.267+0000 7f3e5f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e6015dab0 0x7f3e600703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.267+0000 7f3e5f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e6015dab0 0x7f3e600703f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60386/0 (socket says 172.21.15.116:60386) 2023-11-28T17:10:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.267+0000 7f3e5f7fe640 1 -- 172.21.15.116:0/1270609450 learned_addr learned my addr 172.21.15.116:0/1270609450 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.268+0000 7f3e5f7fe640 1 -- 172.21.15.116:0/1270609450 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e6006cae0 msgr2=0x7f3e6014a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.268+0000 7f3e5f7fe640 1 --2- 172.21.15.116:0/1270609450 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e6006cae0 0x7f3e6014a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.268+0000 7f3e5f7fe640 1 -- 172.21.15.116:0/1270609450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e600b3310 msgr2=0x7f3e601591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.268+0000 7f3e567fc640 1 --2- 172.21.15.116:0/1270609450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e600b3310 0x7f3e601591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.268+0000 7f3e5f7fe640 1 --2- 172.21.15.116:0/1270609450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e600b3310 0x7f3e601591f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.268+0000 7f3e5f7fe640 1 -- 172.21.15.116:0/1270609450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e58009a70 con 0x7f3e6015dab0 2023-11-28T17:10:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.268+0000 7f3e5f7fe640 1 --2- 172.21.15.116:0/1270609450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e6015dab0 0x7f3e600703f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3e58009eb0 tx=0x7f3e58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.268+0000 7f3e57fff640 1 -- 172.21.15.116:0/1270609450 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5803f420 con 0x7f3e6015dab0 2023-11-28T17:10:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.268+0000 7f3e57fff640 1 -- 172.21.15.116:0/1270609450 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5803f580 con 0x7f3e6015dab0 2023-11-28T17:10:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.268+0000 7f3e65415640 1 -- 172.21.15.116:0/1270609450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e601f6e10 con 0x7f3e6015dab0 2023-11-28T17:10:15.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.268+0000 7f3e57fff640 1 -- 172.21.15.116:0/1270609450 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5803f910 con 0x7f3e6015dab0 2023-11-28T17:10:15.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.268+0000 7f3e65415640 1 -- 172.21.15.116:0/1270609450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6014abf0 con 0x7f3e6015dab0 2023-11-28T17:10:15.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.269+0000 7f3e65415640 1 -- 172.21.15.116:0/1270609450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e600693d0 con 0x7f3e6015dab0 2023-11-28T17:10:15.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.270+0000 7f3e57fff640 1 -- 172.21.15.116:0/1270609450 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3e5803fa70 con 0x7f3e6015dab0 2023-11-28T17:10:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.271+0000 7f3e57fff640 1 --2- 172.21.15.116:0/1270609450 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3e4006b2d0 0x7f3e4006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.271+0000 7f3e57fff640 1 -- 172.21.15.116:0/1270609450 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3e580ac330 con 0x7f3e6015dab0 2023-11-28T17:10:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.271+0000 7f3e5effd640 1 --2- 172.21.15.116:0/1270609450 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3e4006b2d0 0x7f3e4006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.272+0000 7f3e5effd640 1 --2- 172.21.15.116:0/1270609450 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3e4006b2d0 0x7f3e4006d790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3e48006fd0 tx=0x7f3e48006bc0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:15.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.275+0000 7f3e57fff640 1 -- 172.21.15.116:0/1270609450 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e58077450 con 0x7f3e6015dab0 2023-11-28T17:10:15.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.516+0000 7f3e65415640 1 -- 172.21.15.116:0/1270609450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f3e60071b80 con 0x7f3e6015dab0 2023-11-28T17:10:15.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.521+0000 7f3e57fff640 1 -- 172.21.15.116:0/1270609450 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3e5807aa70 con 0x7f3e6015dab0 2023-11-28T17:10:15.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.524+0000 7f3e65415640 1 -- 172.21.15.116:0/1270609450 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3e4006b2d0 msgr2=0x7f3e4006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:15.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.524+0000 7f3e65415640 1 --2- 172.21.15.116:0/1270609450 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3e4006b2d0 0x7f3e4006d790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3e48006fd0 tx=0x7f3e48006bc0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.524+0000 7f3e65415640 1 -- 172.21.15.116:0/1270609450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e6015dab0 msgr2=0x7f3e600703f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:15.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.524+0000 7f3e65415640 1 --2- 172.21.15.116:0/1270609450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e6015dab0 0x7f3e600703f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3e58009eb0 tx=0x7f3e58004a40 comp rx=0 tx=0).stop 2023-11-28T17:10:15.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.524+0000 7f3e65415640 1 -- 172.21.15.116:0/1270609450 shutdown_connections 2023-11-28T17:10:15.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.524+0000 7f3e65415640 1 --2- 172.21.15.116:0/1270609450 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3e4006b2d0 0x7f3e4006d790 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.524+0000 7f3e65415640 1 --2- 172.21.15.116:0/1270609450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e6015dab0 0x7f3e600703f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.524+0000 7f3e65415640 1 --2- 172.21.15.116:0/1270609450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e600b3310 0x7f3e601591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.524+0000 7f3e65415640 1 --2- 172.21.15.116:0/1270609450 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e6006cae0 0x7f3e6014a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.524+0000 7f3e65415640 1 -- 172.21.15.116:0/1270609450 >> 172.21.15.116:0/1270609450 conn(0x7f3e6006a4c0 msgr2=0x7f3e601538f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:15.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.524+0000 7f3e65415640 1 -- 172.21.15.116:0/1270609450 shutdown_connections 2023-11-28T17:10:15.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.524+0000 7f3e65415640 1 -- 172.21.15.116:0/1270609450 wait complete. 2023-11-28T17:10:15.545 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:10:15.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.754+0000 7f35d94b2640 1 Processor -- start 2023-11-28T17:10:15.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.754+0000 7f35d94b2640 1 -- start start 2023-11-28T17:10:15.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.756+0000 7f35d94b2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f35d40d01e0 0x7f35d40ce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:15.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.756+0000 7f35d94b2640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d40cdcb0 con 0x7f35d40d1780 2023-11-28T17:10:15.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.756+0000 7f35d94b2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d40d39e0 con 0x7f35d40b20a0 2023-11-28T17:10:15.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.756+0000 7f35d94b2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d40d05f0 con 0x7f35d40d01e0 2023-11-28T17:10:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.756+0000 7f35d2ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f35d40d1780 0x7f35d40d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57772/0 (socket says 172.21.15.116:57772) 2023-11-28T17:10:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.756+0000 7f35d2ffd640 1 -- 172.21.15.116:0/1859929628 learned_addr learned my addr 172.21.15.116:0/1859929628 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.756+0000 7f35d37fe640 1 --2- 172.21.15.116:0/1859929628 >> v2:172.21.15.116:3300/0 conn(0x7f35d40d01e0 0x7f35d40ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.756+0000 7f35d1ffb640 1 -- 172.21.15.116:0/1859929628 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820917687 0 0) 0x7f35d40cdcb0 con 0x7f35d40d1780 2023-11-28T17:10:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.756+0000 7f35d1ffb640 1 -- 172.21.15.116:0/1859929628 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35bc003520 con 0x7f35d40d1780 2023-11-28T17:10:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.757+0000 7f35d37fe640 1 -- 172.21.15.116:0/1859929628 >> v1:172.21.15.156:6789/0 conn(0x7f35d40b20a0 legacy=0x7f35d40b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:15.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.757+0000 7f35d37fe640 1 -- 172.21.15.116:0/1859929628 >> v1:172.21.15.116:6789/0 conn(0x7f35d40d1780 legacy=0x7f35d40d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:15.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.757+0000 7f35d37fe640 1 -- 172.21.15.116:0/1859929628 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d40d3cc0 con 0x7f35d40d01e0 2023-11-28T17:10:15.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.758+0000 7f35d37fe640 1 --2- 172.21.15.116:0/1859929628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d40d01e0 0x7f35d40ce750 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009d90 tx=0x7f35cc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72daac2c9952486b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:15.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.758+0000 7f35d1ffb640 1 -- 172.21.15.116:0/1859929628 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc02cbd0 con 0x7f35d40d01e0 2023-11-28T17:10:15.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.758+0000 7f35d1ffb640 1 -- 172.21.15.116:0/1859929628 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35cc02cd30 con 0x7f35d40d01e0 2023-11-28T17:10:15.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.758+0000 7f35d1ffb640 1 -- 172.21.15.116:0/1859929628 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc02cea0 con 0x7f35d40d01e0 2023-11-28T17:10:15.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.759+0000 7f35d94b2640 1 -- 172.21.15.116:0/1859929628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d40d01e0 msgr2=0x7f35d40ce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:15.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.759+0000 7f35d94b2640 1 --2- 172.21.15.116:0/1859929628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d40d01e0 0x7f35d40ce750 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009d90 tx=0x7f35cc02be20 comp rx=0 tx=0).stop 2023-11-28T17:10:15.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.759+0000 7f35d94b2640 1 -- 172.21.15.116:0/1859929628 shutdown_connections 2023-11-28T17:10:15.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.759+0000 7f35d94b2640 1 --2- 172.21.15.116:0/1859929628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d40d01e0 0x7f35d40ce750 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.759+0000 7f35d94b2640 1 -- 172.21.15.116:0/1859929628 >> 172.21.15.116:0/1859929628 conn(0x7f35d409c4e0 msgr2=0x7f35d40d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:15.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.759+0000 7f35d94b2640 1 -- 172.21.15.116:0/1859929628 shutdown_connections 2023-11-28T17:10:15.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.759+0000 7f35d94b2640 1 -- 172.21.15.116:0/1859929628 wait complete. 2023-11-28T17:10:15.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.761+0000 7f35d94b2640 1 Processor -- start 2023-11-28T17:10:15.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.761+0000 7f35d94b2640 1 -- start start 2023-11-28T17:10:15.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.763+0000 7f35d94b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d40b20a0 0x7f35d41546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:15.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.763+0000 7f35d94b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35d40d01e0 0x7f35d41f6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:15.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.763+0000 7f35d94b2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35d40d1780 0x7f35d4159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:15.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.763+0000 7f35d94b2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d40b7100 con 0x7f35d40b20a0 2023-11-28T17:10:15.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.763+0000 7f35d94b2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d41f6fe0 con 0x7f35d40d01e0 2023-11-28T17:10:15.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.763+0000 7f35d94b2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d41f72c0 con 0x7f35d40d1780 2023-11-28T17:10:15.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.763+0000 7f35d2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d40b20a0 0x7f35d41546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:15.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.763+0000 7f35d2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d40b20a0 0x7f35d41546c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60410/0 (socket says 172.21.15.116:60410) 2023-11-28T17:10:15.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.763+0000 7f35d2ffd640 1 -- 172.21.15.116:0/3298086143 learned_addr learned my addr 172.21.15.116:0/3298086143 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:15.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.763+0000 7f35d2ffd640 1 -- 172.21.15.116:0/3298086143 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35d40d1780 msgr2=0x7f35d4159460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:15.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.763+0000 7f35d2ffd640 1 --2- 172.21.15.116:0/3298086143 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35d40d1780 0x7f35d4159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.764+0000 7f35d2ffd640 1 -- 172.21.15.116:0/3298086143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35d40d01e0 msgr2=0x7f35d41f6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:15.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.764+0000 7f35d27fc640 1 --2- 172.21.15.116:0/3298086143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35d40d01e0 0x7f35d41f6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:15.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.764+0000 7f35d2ffd640 1 --2- 172.21.15.116:0/3298086143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35d40d01e0 0x7f35d41f6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.764+0000 7f35d2ffd640 1 -- 172.21.15.116:0/3298086143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc009a70 con 0x7f35d40b20a0 2023-11-28T17:10:15.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.764+0000 7f35d27fc640 1 --2- 172.21.15.116:0/3298086143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35d40d01e0 0x7f35d41f6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:15.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.764+0000 7f35d2ffd640 1 --2- 172.21.15.116:0/3298086143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d40b20a0 0x7f35d41546c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f35c400c410 tx=0x7f35c400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:15.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.764+0000 7f35bb7fe640 1 -- 172.21.15.116:0/3298086143 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4005ce0 con 0x7f35d40b20a0 2023-11-28T17:10:15.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.764+0000 7f35bb7fe640 1 -- 172.21.15.116:0/3298086143 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c4005e40 con 0x7f35d40b20a0 2023-11-28T17:10:15.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.764+0000 7f35bb7fe640 1 -- 172.21.15.116:0/3298086143 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c40072f0 con 0x7f35d40b20a0 2023-11-28T17:10:15.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.764+0000 7f35d94b2640 1 -- 172.21.15.116:0/3298086143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d40b7350 con 0x7f35d40b20a0 2023-11-28T17:10:15.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.764+0000 7f35d94b2640 1 -- 172.21.15.116:0/3298086143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d4154c00 con 0x7f35d40b20a0 2023-11-28T17:10:15.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.766+0000 7f35d94b2640 1 -- 172.21.15.116:0/3298086143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d40b7870 con 0x7f35d40b20a0 2023-11-28T17:10:15.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.766+0000 7f35bb7fe640 1 -- 172.21.15.116:0/3298086143 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f35c4007510 con 0x7f35d40b20a0 2023-11-28T17:10:15.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.767+0000 7f35bb7fe640 1 --2- 172.21.15.116:0/3298086143 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f35c006b2a0 0x7f35c006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:15.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.767+0000 7f35bb7fe640 1 -- 172.21.15.116:0/3298086143 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f35c4088ea0 con 0x7f35d40b20a0 2023-11-28T17:10:15.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.768+0000 7f35d27fc640 1 --2- 172.21.15.116:0/3298086143 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f35c006b2a0 0x7f35c006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:15.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.768+0000 7f35d27fc640 1 --2- 172.21.15.116:0/3298086143 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f35c006b2a0 0x7f35c006d760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f35d41f7650 tx=0x7f35c8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:15.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.772+0000 7f35bb7fe640 1 -- 172.21.15.116:0/3298086143 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35d40b7870 con 0x7f35d40b20a0 2023-11-28T17:10:15.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.962+0000 7f35d94b2640 1 -- 172.21.15.116:0/3298086143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35d40b7870 con 0x7f35d40b20a0 2023-11-28T17:10:15.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.965+0000 7f35bb7fe640 1 -- 172.21.15.116:0/3298086143 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12869 (secure 0 0 0) 0x7f35c40541b0 con 0x7f35d40b20a0 2023-11-28T17:10:15.966 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:15.966 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:10:12.019779+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:10:15.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.967+0000 7f35d94b2640 1 -- 172.21.15.116:0/3298086143 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f35c006b2a0 msgr2=0x7f35c006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:15.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.967+0000 7f35d94b2640 1 --2- 172.21.15.116:0/3298086143 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f35c006b2a0 0x7f35c006d760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f35d41f7650 tx=0x7f35c8008040 comp rx=0 tx=0).stop 2023-11-28T17:10:15.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.967+0000 7f35d94b2640 1 -- 172.21.15.116:0/3298086143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d40b20a0 msgr2=0x7f35d41546c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:15.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.967+0000 7f35d94b2640 1 --2- 172.21.15.116:0/3298086143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d40b20a0 0x7f35d41546c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f35c400c410 tx=0x7f35c400c8d0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.967+0000 7f35d94b2640 1 -- 172.21.15.116:0/3298086143 shutdown_connections 2023-11-28T17:10:15.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.967+0000 7f35d94b2640 1 --2- 172.21.15.116:0/3298086143 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f35c006b2a0 0x7f35c006d760 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.967+0000 7f35d94b2640 1 --2- 172.21.15.116:0/3298086143 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f35d40d1780 0x7f35d4159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.967+0000 7f35d94b2640 1 --2- 172.21.15.116:0/3298086143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35d40d01e0 0x7f35d41f6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.967+0000 7f35d94b2640 1 --2- 172.21.15.116:0/3298086143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35d40b20a0 0x7f35d41546c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:15.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.967+0000 7f35d94b2640 1 -- 172.21.15.116:0/3298086143 >> 172.21.15.116:0/3298086143 conn(0x7f35d409c4e0 msgr2=0x7f35d40d2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:15.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.967+0000 7f35d94b2640 1 -- 172.21.15.116:0/3298086143 shutdown_connections 2023-11-28T17:10:15.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:15.968+0000 7f35d94b2640 1 -- 172.21.15.116:0/3298086143 wait complete. 2023-11-28T17:10:15.986 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:10:15.986 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:10:16.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.195+0000 7f86849fb640 1 Processor -- start 2023-11-28T17:10:16.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.195+0000 7f86849fb640 1 -- start start 2023-11-28T17:10:16.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.196+0000 7f86849fb640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8680152050 0x7f868015bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.196+0000 7f86849fb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86800b2fe0 0x7f86800ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.196+0000 7f86849fb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86800b2b50 con 0x7f86800abd30 2023-11-28T17:10:16.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.196+0000 7f86849fb640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f868015c520 con 0x7f8680152050 2023-11-28T17:10:16.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.196+0000 7f86849fb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86800ac2d0 con 0x7f86800b2fe0 2023-11-28T17:10:16.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.197+0000 7f867f59e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f86800abd30 0x7f86800cdc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57782/0 (socket says 172.21.15.116:57782) 2023-11-28T17:10:16.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.197+0000 7f867f59e640 1 -- 172.21.15.116:0/599767590 learned_addr learned my addr 172.21.15.116:0/599767590 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:16.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.197+0000 7f8677fff640 1 --2- 172.21.15.116:0/599767590 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86800b2fe0 0x7f86800ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:16.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.197+0000 7f867ed9d640 1 --2- 172.21.15.116:0/599767590 >> v2:172.21.15.156:3300/0 conn(0x7f8680152050 0x7f868015bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:16.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.197+0000 7f867e59c640 1 -- 172.21.15.116:0/599767590 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257620592 0 0) 0x7f86800b2b50 con 0x7f86800abd30 2023-11-28T17:10:16.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.197+0000 7f867e59c640 1 -- 172.21.15.116:0/599767590 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f866c003520 con 0x7f86800abd30 2023-11-28T17:10:16.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f867e59c640 1 -- 172.21.15.116:0/599767590 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4057575199 0 0) 0x7f866c003520 con 0x7f86800abd30 2023-11-28T17:10:16.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f867e59c640 1 -- 172.21.15.116:0/599767590 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86800b2fe0 msgr2=0x7f86800ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:16.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f867e59c640 1 --2- 172.21.15.116:0/599767590 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86800b2fe0 0x7f86800ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f867e59c640 1 -- 172.21.15.116:0/599767590 >> v2:172.21.15.156:3300/0 conn(0x7f8680152050 msgr2=0x7f868015bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:16.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f867e59c640 1 --2- 172.21.15.116:0/599767590 >> v2:172.21.15.156:3300/0 conn(0x7f8680152050 0x7f868015bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f867e59c640 1 -- 172.21.15.116:0/599767590 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86800b2220 con 0x7f86800abd30 2023-11-28T17:10:16.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f867e59c640 1 -- 172.21.15.116:0/599767590 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f86780019f0 con 0x7f86800abd30 2023-11-28T17:10:16.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f867e59c640 1 -- 172.21.15.116:0/599767590 >> v1:172.21.15.116:6789/0 conn(0x7f86800abd30 legacy=0x7f86800cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:16.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f867e59c640 1 --2- 172.21.15.116:0/599767590 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86800b2fe0 0x7f866c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f867e59c640 1 --2- 172.21.15.116:0/599767590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866c003f40 0x7f866c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f867e59c640 1 --2- 172.21.15.116:0/599767590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f866c004d00 0x7f866c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f867e59c640 1 -- 172.21.15.116:0/599767590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868015c520 con 0x7f866c003f40 2023-11-28T17:10:16.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f867e59c640 1 -- 172.21.15.116:0/599767590 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86800ac2d0 con 0x7f866c004d00 2023-11-28T17:10:16.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f867e59c640 1 -- 172.21.15.116:0/599767590 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866c00b980 con 0x7f86800b2fe0 2023-11-28T17:10:16.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f8677fff640 1 --2- 172.21.15.116:0/599767590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866c003f40 0x7f866c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:16.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f8677fff640 1 -- 172.21.15.116:0/599767590 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86800b2fe0 msgr2=0x7f866c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:16.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f8677fff640 1 --2- 172.21.15.116:0/599767590 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f86800b2fe0 0x7f866c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f8677fff640 1 -- 172.21.15.116:0/599767590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f866c004d00 msgr2=0x7f866c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:16.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f8677fff640 1 --2- 172.21.15.116:0/599767590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f866c004d00 0x7f866c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.198+0000 7f8677fff640 1 -- 172.21.15.116:0/599767590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866c003520 con 0x7f866c003f40 2023-11-28T17:10:16.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.200+0000 7f8677fff640 1 --2- 172.21.15.116:0/599767590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866c003f40 0x7f866c0047c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8668009ac0 tx=0x7f866c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe1a3a84dfc5067 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:16.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.200+0000 7f867e59c640 1 -- 172.21.15.116:0/599767590 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866802c6b0 con 0x7f866c003f40 2023-11-28T17:10:16.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.200+0000 7f867e59c640 1 -- 172.21.15.116:0/599767590 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866802c810 con 0x7f866c003f40 2023-11-28T17:10:16.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.200+0000 7f867e59c640 1 -- 172.21.15.116:0/599767590 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866802cba0 con 0x7f866c003f40 2023-11-28T17:10:16.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.200+0000 7f86849fb640 1 -- 172.21.15.116:0/599767590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866c003f40 msgr2=0x7f866c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:16.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.200+0000 7f86849fb640 1 --2- 172.21.15.116:0/599767590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866c003f40 0x7f866c0047c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8668009ac0 tx=0x7f866c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.200+0000 7f867f59e640 1 -- 172.21.15.116:0/599767590 reap_dead start 2023-11-28T17:10:16.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.200+0000 7f86849fb640 1 -- 172.21.15.116:0/599767590 shutdown_connections 2023-11-28T17:10:16.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.200+0000 7f86849fb640 1 -- 172.21.15.116:0/599767590 >> 172.21.15.116:0/599767590 conn(0x7f86800b0950 msgr2=0x7f86800b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:16.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.201+0000 7f86849fb640 1 -- 172.21.15.116:0/599767590 shutdown_connections 2023-11-28T17:10:16.204 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.201+0000 7f86849fb640 1 -- 172.21.15.116:0/599767590 wait complete. 2023-11-28T17:10:16.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.202+0000 7f86849fb640 1 Processor -- start 2023-11-28T17:10:16.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.203+0000 7f86849fb640 1 -- start start 2023-11-28T17:10:16.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.203+0000 7f86849fb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f866c003f40 0x7f8680152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.204+0000 7f86849fb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f868015a7e0 0x7f868013c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.204+0000 7f86849fb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8680071220 0x7f86800716a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.204+0000 7f86849fb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8680071be0 con 0x7f868015a7e0 2023-11-28T17:10:16.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.204+0000 7f86849fb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8680154c90 con 0x7f8680071220 2023-11-28T17:10:16.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.204+0000 7f86849fb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8680154f70 con 0x7f866c003f40 2023-11-28T17:10:16.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.204+0000 7f8677fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f868015a7e0 0x7f868013c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:16.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.204+0000 7f8677fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f868015a7e0 0x7f868013c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60440/0 (socket says 172.21.15.116:60440) 2023-11-28T17:10:16.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.204+0000 7f8677fff640 1 -- 172.21.15.116:0/2238521174 learned_addr learned my addr 172.21.15.116:0/2238521174 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:16.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.204+0000 7f8677fff640 1 -- 172.21.15.116:0/2238521174 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f866c003f40 msgr2=0x7f8680152fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:16.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.204+0000 7f8677fff640 1 --2- 172.21.15.116:0/2238521174 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f866c003f40 0x7f8680152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.204+0000 7f867f59e640 1 --2- 172.21.15.116:0/2238521174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8680071220 0x7f86800716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:16.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.204+0000 7f8677fff640 1 -- 172.21.15.116:0/2238521174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8680071220 msgr2=0x7f86800716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:16.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.204+0000 7f8677fff640 1 --2- 172.21.15.116:0/2238521174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8680071220 0x7f86800716a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.204+0000 7f8677fff640 1 -- 172.21.15.116:0/2238521174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8680155250 con 0x7f868015a7e0 2023-11-28T17:10:16.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.205+0000 7f8677fff640 1 --2- 172.21.15.116:0/2238521174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f868015a7e0 0x7f868013c670 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f866802c250 tx=0x7f8668004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:16.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.205+0000 7f86777fe640 1 -- 172.21.15.116:0/2238521174 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8668003810 con 0x7f868015a7e0 2023-11-28T17:10:16.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.205+0000 7f86777fe640 1 -- 172.21.15.116:0/2238521174 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8668003970 con 0x7f868015a7e0 2023-11-28T17:10:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.205+0000 7f86777fe640 1 -- 172.21.15.116:0/2238521174 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8668003d00 con 0x7f868015a7e0 2023-11-28T17:10:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.205+0000 7f86849fb640 1 -- 172.21.15.116:0/2238521174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8680155530 con 0x7f868015a7e0 2023-11-28T17:10:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.205+0000 7f86849fb640 1 -- 172.21.15.116:0/2238521174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868013cbb0 con 0x7f868015a7e0 2023-11-28T17:10:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.206+0000 7f86849fb640 1 -- 172.21.15.116:0/2238521174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86800ce310 con 0x7f868015a7e0 2023-11-28T17:10:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.207+0000 7f86777fe640 1 -- 172.21.15.116:0/2238521174 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f8668047450 con 0x7f868015a7e0 2023-11-28T17:10:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.208+0000 7f86777fe640 1 --2- 172.21.15.116:0/2238521174 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f866006b2d0 0x7f866006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.208+0000 7f86777fe640 1 -- 172.21.15.116:0/2238521174 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f86680aba00 con 0x7f868015a7e0 2023-11-28T17:10:16.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.208+0000 7f867ed9d640 1 --2- 172.21.15.116:0/2238521174 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f866006b2d0 0x7f866006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:16.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.209+0000 7f867ed9d640 1 --2- 172.21.15.116:0/2238521174 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f866006b2d0 0x7f866006d790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8670006e80 tx=0x7f8670006dd0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:16.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.213+0000 7f86777fe640 1 -- 172.21.15.116:0/2238521174 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8668076c70 con 0x7f868015a7e0 2023-11-28T17:10:16.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.402+0000 7f86849fb640 1 -- 172.21.15.116:0/2238521174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f868009d0b0 con 0x7f868015a7e0 2023-11-28T17:10:16.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.403+0000 7f86777fe640 1 -- 172.21.15.116:0/2238521174 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7f86680044e0 con 0x7f868015a7e0 2023-11-28T17:10:16.405 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:10:16.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.406+0000 7f86849fb640 1 -- 172.21.15.116:0/2238521174 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f866006b2d0 msgr2=0x7f866006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:16.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.406+0000 7f86849fb640 1 --2- 172.21.15.116:0/2238521174 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f866006b2d0 0x7f866006d790 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8670006e80 tx=0x7f8670006dd0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.406+0000 7f86849fb640 1 -- 172.21.15.116:0/2238521174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f868015a7e0 msgr2=0x7f868013c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:16.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.406+0000 7f86849fb640 1 --2- 172.21.15.116:0/2238521174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f868015a7e0 0x7f868013c670 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f866802c250 tx=0x7f8668004730 comp rx=0 tx=0).stop 2023-11-28T17:10:16.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.406+0000 7f86849fb640 1 -- 172.21.15.116:0/2238521174 shutdown_connections 2023-11-28T17:10:16.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.406+0000 7f86849fb640 1 --2- 172.21.15.116:0/2238521174 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f866006b2d0 0x7f866006d790 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.406+0000 7f86849fb640 1 --2- 172.21.15.116:0/2238521174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8680071220 0x7f86800716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.406+0000 7f86849fb640 1 --2- 172.21.15.116:0/2238521174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f868015a7e0 0x7f868013c670 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.406+0000 7f86849fb640 1 --2- 172.21.15.116:0/2238521174 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f866c003f40 0x7f8680152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.406+0000 7f86849fb640 1 -- 172.21.15.116:0/2238521174 >> 172.21.15.116:0/2238521174 conn(0x7f86800b0950 msgr2=0x7f868015f250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:16.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.406+0000 7f86849fb640 1 -- 172.21.15.116:0/2238521174 shutdown_connections 2023-11-28T17:10:16.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.406+0000 7f86849fb640 1 -- 172.21.15.116:0/2238521174 wait complete. 2023-11-28T17:10:16.424 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T17:10:16.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.631+0000 7f29d322d640 1 Processor -- start 2023-11-28T17:10:16.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.631+0000 7f29d322d640 1 -- start start 2023-11-28T17:10:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.633+0000 7f29d322d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f29cc13cee0 0x7f29cc06c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.633+0000 7f29d322d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f29cc140200 0x7f29cc06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.633+0000 7f29d322d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29cc15a490 0x7f29cc15a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.633+0000 7f29d322d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29cc0aea20 con 0x7f29cc140200 2023-11-28T17:10:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.633+0000 7f29d322d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29cc13d2c0 con 0x7f29cc13cee0 2023-11-28T17:10:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.633+0000 7f29d322d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc15b3a0 con 0x7f29cc15a490 2023-11-28T17:10:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.633+0000 7f29cbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f29cc140200 0x7f29cc06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.633+0000 7f29cbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f29cc140200 0x7f29cc06c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60446/0 (socket says 172.21.15.116:60446) 2023-11-28T17:10:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.633+0000 7f29cbfff640 1 -- 172.21.15.116:0/3448033877 learned_addr learned my addr 172.21.15.116:0/3448033877 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.634+0000 7f29d0fc9640 1 --2- 172.21.15.116:0/3448033877 >> v2:172.21.15.156:3300/0 conn(0x7f29cc13cee0 0x7f29cc06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.634+0000 7f29cbfff640 1 -- 172.21.15.116:0/3448033877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29cc15a490 msgr2=0x7f29cc15a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.634+0000 7f29cbfff640 1 --2- 172.21.15.116:0/3448033877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29cc15a490 0x7f29cc15a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.634+0000 7f29cbfff640 1 -- 172.21.15.116:0/3448033877 >> v2:172.21.15.156:3300/0 conn(0x7f29cc13cee0 msgr2=0x7f29cc06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.634+0000 7f29cbfff640 1 --2- 172.21.15.116:0/3448033877 >> v2:172.21.15.156:3300/0 conn(0x7f29cc13cee0 0x7f29cc06c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.634+0000 7f29cbfff640 1 -- 172.21.15.116:0/3448033877 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29cc15ae60 con 0x7f29cc140200 2023-11-28T17:10:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.634+0000 7f29d0fc9640 1 --2- 172.21.15.116:0/3448033877 >> v2:172.21.15.156:3300/0 conn(0x7f29cc13cee0 0x7f29cc06c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.635+0000 7f29cbfff640 1 --2- 172.21.15.116:0/3448033877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc140200 0x7f29cc06c790 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f29cc0b2280 tx=0x7f29b802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23947e9b45b69199 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.635+0000 7f29cb7fe640 1 -- 172.21.15.116:0/3448033877 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b80317e0 con 0x7f29cc140200 2023-11-28T17:10:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.635+0000 7f29cb7fe640 1 -- 172.21.15.116:0/3448033877 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b8031940 con 0x7f29cc140200 2023-11-28T17:10:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.635+0000 7f29cb7fe640 1 -- 172.21.15.116:0/3448033877 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8031cd0 con 0x7f29cc140200 2023-11-28T17:10:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.636+0000 7f29d322d640 1 -- 172.21.15.116:0/3448033877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc140200 msgr2=0x7f29cc06c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:16.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.636+0000 7f29d322d640 1 --2- 172.21.15.116:0/3448033877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc140200 0x7f29cc06c790 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f29cc0b2280 tx=0x7f29b802b880 comp rx=0 tx=0).stop 2023-11-28T17:10:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.636+0000 7f29d322d640 1 -- 172.21.15.116:0/3448033877 shutdown_connections 2023-11-28T17:10:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.636+0000 7f29d322d640 1 --2- 172.21.15.116:0/3448033877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29cc15a490 0x7f29cc15a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.636+0000 7f29d322d640 1 --2- 172.21.15.116:0/3448033877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc140200 0x7f29cc06c790 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.636+0000 7f29d322d640 1 --2- 172.21.15.116:0/3448033877 >> v2:172.21.15.156:3300/0 conn(0x7f29cc13cee0 0x7f29cc06c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.636+0000 7f29d322d640 1 -- 172.21.15.116:0/3448033877 >> 172.21.15.116:0/3448033877 conn(0x7f29cc06a4c0 msgr2=0x7f29cc06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.636+0000 7f29d322d640 1 -- 172.21.15.116:0/3448033877 shutdown_connections 2023-11-28T17:10:16.638 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.636+0000 7f29d322d640 1 -- 172.21.15.116:0/3448033877 wait complete. 2023-11-28T17:10:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.638+0000 7f29d322d640 1 Processor -- start 2023-11-28T17:10:16.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.638+0000 7f29d322d640 1 -- start start 2023-11-28T17:10:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.639+0000 7f29d322d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29cc13cee0 0x7f29cc136f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.639+0000 7f29d322d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29cc140200 0x7f29cc136550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.639+0000 7f29d322d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc15a490 0x7f29cc071600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.639+0000 7f29d322d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc0711e0 con 0x7f29cc15a490 2023-11-28T17:10:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.639+0000 7f29d322d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc13eb20 con 0x7f29cc13cee0 2023-11-28T17:10:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.639+0000 7f29d322d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc13ed50 con 0x7f29cc140200 2023-11-28T17:10:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.639+0000 7f29d17ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc15a490 0x7f29cc071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:16.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.639+0000 7f29d17ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc15a490 0x7f29cc071600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60448/0 (socket says 172.21.15.116:60448) 2023-11-28T17:10:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.640+0000 7f29d17ca640 1 -- 172.21.15.116:0/3720177016 learned_addr learned my addr 172.21.15.116:0/3720177016 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.640+0000 7f29d0fc9640 1 --2- 172.21.15.116:0/3720177016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29cc13cee0 0x7f29cc136f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.640+0000 7f29cbfff640 1 --2- 172.21.15.116:0/3720177016 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29cc140200 0x7f29cc136550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.640+0000 7f29d17ca640 1 -- 172.21.15.116:0/3720177016 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29cc140200 msgr2=0x7f29cc136550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.640+0000 7f29d17ca640 1 --2- 172.21.15.116:0/3720177016 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29cc140200 0x7f29cc136550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.640+0000 7f29d17ca640 1 -- 172.21.15.116:0/3720177016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29cc13cee0 msgr2=0x7f29cc136f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.640+0000 7f29d17ca640 1 --2- 172.21.15.116:0/3720177016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29cc13cee0 0x7f29cc136f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.640+0000 7f29d17ca640 1 -- 172.21.15.116:0/3720177016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b80094d0 con 0x7f29cc15a490 2023-11-28T17:10:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.640+0000 7f29d17ca640 1 --2- 172.21.15.116:0/3720177016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc15a490 0x7f29cc071600 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f29c400b9a0 tx=0x7f29c400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.640+0000 7f29c97fa640 1 -- 172.21.15.116:0/3720177016 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c400cba0 con 0x7f29cc15a490 2023-11-28T17:10:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.641+0000 7f29c97fa640 1 -- 172.21.15.116:0/3720177016 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c400cd20 con 0x7f29cc15a490 2023-11-28T17:10:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.641+0000 7f29c97fa640 1 -- 172.21.15.116:0/3720177016 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4014680 con 0x7f29cc15a490 2023-11-28T17:10:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.641+0000 7f29d322d640 1 -- 172.21.15.116:0/3720177016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29cc13f030 con 0x7f29cc15a490 2023-11-28T17:10:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.641+0000 7f29d322d640 1 -- 172.21.15.116:0/3720177016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29cc070e00 con 0x7f29cc15a490 2023-11-28T17:10:16.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.642+0000 7f29d322d640 1 -- 172.21.15.116:0/3720177016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29cc071320 con 0x7f29cc15a490 2023-11-28T17:10:16.644 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.643+0000 7f29c97fa640 1 -- 172.21.15.116:0/3720177016 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f29c401d3b0 con 0x7f29cc15a490 2023-11-28T17:10:16.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.644+0000 7f29c97fa640 1 --2- 172.21.15.116:0/3720177016 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f29b006b2a0 0x7f29b006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:16.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.644+0000 7f29c97fa640 1 -- 172.21.15.116:0/3720177016 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f29c4089820 con 0x7f29cc15a490 2023-11-28T17:10:16.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.644+0000 7f29d0fc9640 1 --2- 172.21.15.116:0/3720177016 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f29b006b2a0 0x7f29b006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:16.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.645+0000 7f29d0fc9640 1 --2- 172.21.15.116:0/3720177016 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f29b006b2a0 0x7f29b006d760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f29cc1592a0 tx=0x7f29c0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:16.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.649+0000 7f29c97fa640 1 -- 172.21.15.116:0/3720177016 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29cc071320 con 0x7f29cc15a490 2023-11-28T17:10:16.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.840+0000 7f29d322d640 1 -- 172.21.15.116:0/3720177016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f29cc071320 con 0x7f29cc15a490 2023-11-28T17:10:16.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.841+0000 7f29c97fa640 1 -- 172.21.15.116:0/3720177016 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7f29cc071320 con 0x7f29cc15a490 2023-11-28T17:10:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:16.842 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2023-11-28T17:10:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:10:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 1, 2023-11-28T17:10:16.842 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": ".mgr" 2023-11-28T17:10:16.843 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:16.843 INFO:teuthology.orchestra.run.smithi116.stdout:] 2023-11-28T17:10:16.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.843+0000 7f29d322d640 1 -- 172.21.15.116:0/3720177016 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f29b006b2a0 msgr2=0x7f29b006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:16.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.843+0000 7f29d322d640 1 --2- 172.21.15.116:0/3720177016 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f29b006b2a0 0x7f29b006d760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f29cc1592a0 tx=0x7f29c0008040 comp rx=0 tx=0).stop 2023-11-28T17:10:16.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.843+0000 7f29d322d640 1 -- 172.21.15.116:0/3720177016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc15a490 msgr2=0x7f29cc071600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:16.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.843+0000 7f29d322d640 1 --2- 172.21.15.116:0/3720177016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc15a490 0x7f29cc071600 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f29c400b9a0 tx=0x7f29c400be60 comp rx=0 tx=0).stop 2023-11-28T17:10:16.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.843+0000 7f29d322d640 1 -- 172.21.15.116:0/3720177016 shutdown_connections 2023-11-28T17:10:16.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.843+0000 7f29d322d640 1 --2- 172.21.15.116:0/3720177016 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f29b006b2a0 0x7f29b006d760 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.843+0000 7f29d322d640 1 --2- 172.21.15.116:0/3720177016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29cc15a490 0x7f29cc071600 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.843+0000 7f29d322d640 1 --2- 172.21.15.116:0/3720177016 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29cc140200 0x7f29cc136550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.844+0000 7f29d322d640 1 --2- 172.21.15.116:0/3720177016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29cc13cee0 0x7f29cc136f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:16.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.844+0000 7f29d322d640 1 -- 172.21.15.116:0/3720177016 >> 172.21.15.116:0/3720177016 conn(0x7f29cc06a4c0 msgr2=0x7f29cc150870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:16.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.844+0000 7f29d322d640 1 -- 172.21.15.116:0/3720177016 shutdown_connections 2023-11-28T17:10:16.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:16.844+0000 7f29d322d640 1 -- 172.21.15.116:0/3720177016 wait complete. 2023-11-28T17:10:16.864 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T17:10:16.864 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T17:10:17.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.078+0000 7f50c7378640 1 Processor -- start 2023-11-28T17:10:17.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.078+0000 7f50c7378640 1 -- start start 2023-11-28T17:10:17.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.080+0000 7f50c7378640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f50c0160820 0x7f50c00ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:17.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.080+0000 7f50c7378640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f50c00d1ea0 0x7f50c00b6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:17.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.080+0000 7f50c7378640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c0160c00 con 0x7f50c00d19f0 2023-11-28T17:10:17.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.080+0000 7f50c7378640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c00b8e90 con 0x7f50c0160820 2023-11-28T17:10:17.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.080+0000 7f50c7378640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c00cdbe0 con 0x7f50c00d1ea0 2023-11-28T17:10:17.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.080+0000 7f50c5114640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f50c0160820 0x7f50c00ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:17.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.080+0000 7f50c5114640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f50c0160820 0x7f50c00ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60460/0 (socket says 172.21.15.116:60460) 2023-11-28T17:10:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.080+0000 7f50c5114640 1 -- 172.21.15.116:0/4044975899 learned_addr learned my addr 172.21.15.116:0/4044975899 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.081+0000 7f50c5915640 1 --2- 172.21.15.116:0/4044975899 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f50c00d1ea0 0x7f50c00b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.081+0000 7f50c5114640 1 -- 172.21.15.116:0/4044975899 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f50c00d1ea0 msgr2=0x7f50c00b6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.081+0000 7f50c5114640 1 --2- 172.21.15.116:0/4044975899 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f50c00d1ea0 0x7f50c00b6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.081+0000 7f50c5114640 1 -- 172.21.15.116:0/4044975899 >> v1:172.21.15.156:6789/0 conn(0x7f50c00d19f0 legacy=0x7f50c00d30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.081+0000 7f50c5114640 1 -- 172.21.15.116:0/4044975899 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c00cdec0 con 0x7f50c0160820 2023-11-28T17:10:17.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.081+0000 7f50c5915640 1 --2- 172.21.15.116:0/4044975899 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f50c00d1ea0 0x7f50c00b6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:17.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.082+0000 7f50c5114640 1 --2- 172.21.15.116:0/4044975899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0160820 0x7f50c00ce190 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f50c00b5360 tx=0x7f50b002be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c120561f5f9f56 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:17.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.082+0000 7f50bffff640 1 -- 172.21.15.116:0/4044975899 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b002ca30 con 0x7f50c0160820 2023-11-28T17:10:17.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.082+0000 7f50bffff640 1 -- 172.21.15.116:0/4044975899 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b002cb90 con 0x7f50c0160820 2023-11-28T17:10:17.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.082+0000 7f50bffff640 1 -- 172.21.15.116:0/4044975899 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00343f0 con 0x7f50c0160820 2023-11-28T17:10:17.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.083+0000 7f50c7378640 1 -- 172.21.15.116:0/4044975899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0160820 msgr2=0x7f50c00ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:17.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.083+0000 7f50c7378640 1 --2- 172.21.15.116:0/4044975899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0160820 0x7f50c00ce190 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f50c00b5360 tx=0x7f50b002be90 comp rx=0 tx=0).stop 2023-11-28T17:10:17.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.083+0000 7f50c7378640 1 -- 172.21.15.116:0/4044975899 shutdown_connections 2023-11-28T17:10:17.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.083+0000 7f50c7378640 1 --2- 172.21.15.116:0/4044975899 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f50c00d1ea0 0x7f50c00b6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:17.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.083+0000 7f50c7378640 1 --2- 172.21.15.116:0/4044975899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0160820 0x7f50c00ce190 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:17.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.083+0000 7f50c7378640 1 -- 172.21.15.116:0/4044975899 >> 172.21.15.116:0/4044975899 conn(0x7f50c0072010 msgr2=0x7f50c0072420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:17.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.083+0000 7f50c7378640 1 -- 172.21.15.116:0/4044975899 shutdown_connections 2023-11-28T17:10:17.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.083+0000 7f50c7378640 1 -- 172.21.15.116:0/4044975899 wait complete. 2023-11-28T17:10:17.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.085+0000 7f50c7378640 1 Processor -- start 2023-11-28T17:10:17.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.085+0000 7f50c7378640 1 -- start start 2023-11-28T17:10:17.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.086+0000 7f50c7378640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f50c00d19f0 0x7f50c00cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:17.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.086+0000 7f50c7378640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50c00d1ea0 0x7f50c015f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:17.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.086+0000 7f50c7378640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0160820 0x7f50c01f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:17.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.086+0000 7f50c7378640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c00ca2a0 con 0x7f50c0160820 2023-11-28T17:10:17.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.086+0000 7f50c7378640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c01f36c0 con 0x7f50c00d1ea0 2023-11-28T17:10:17.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.086+0000 7f50c7378640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c01f39a0 con 0x7f50c00d19f0 2023-11-28T17:10:17.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.086+0000 7f50c5915640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0160820 0x7f50c01f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:17.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.086+0000 7f50c5915640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0160820 0x7f50c01f2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60466/0 (socket says 172.21.15.116:60466) 2023-11-28T17:10:17.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.086+0000 7f50c5915640 1 -- 172.21.15.116:0/2460590901 learned_addr learned my addr 172.21.15.116:0/2460590901 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:17.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.087+0000 7f50c4913640 1 --2- 172.21.15.116:0/2460590901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50c00d1ea0 0x7f50c015f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:17.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.087+0000 7f50c5915640 1 -- 172.21.15.116:0/2460590901 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f50c00d19f0 msgr2=0x7f50c00cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:17.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.087+0000 7f50c5915640 1 --2- 172.21.15.116:0/2460590901 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f50c00d19f0 0x7f50c00cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:17.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.087+0000 7f50c5915640 1 -- 172.21.15.116:0/2460590901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50c00d1ea0 msgr2=0x7f50c015f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:17.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.087+0000 7f50c5915640 1 --2- 172.21.15.116:0/2460590901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50c00d1ea0 0x7f50c015f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:17.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.087+0000 7f50c5915640 1 -- 172.21.15.116:0/2460590901 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b0009b30 con 0x7f50c0160820 2023-11-28T17:10:17.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.087+0000 7f50c4913640 1 --2- 172.21.15.116:0/2460590901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50c00d1ea0 0x7f50c015f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:17.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.087+0000 7f50c5915640 1 --2- 172.21.15.116:0/2460590901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0160820 0x7f50c01f2c00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f50b800c7f0 tx=0x7f50b800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:17.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.087+0000 7f50bdffb640 1 -- 172.21.15.116:0/2460590901 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b8007820 con 0x7f50c0160820 2023-11-28T17:10:17.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.087+0000 7f50bdffb640 1 -- 172.21.15.116:0/2460590901 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b8007980 con 0x7f50c0160820 2023-11-28T17:10:17.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.087+0000 7f50c7378640 1 -- 172.21.15.116:0/2460590901 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c00ca4f0 con 0x7f50c0160820 2023-11-28T17:10:17.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.087+0000 7f50bdffb640 1 -- 172.21.15.116:0/2460590901 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b8007d70 con 0x7f50c0160820 2023-11-28T17:10:17.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.087+0000 7f50c7378640 1 -- 172.21.15.116:0/2460590901 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c00cb210 con 0x7f50c0160820 2023-11-28T17:10:17.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.088+0000 7f50c7378640 1 -- 172.21.15.116:0/2460590901 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c00d2ab0 con 0x7f50c0160820 2023-11-28T17:10:17.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.089+0000 7f50bdffb640 1 -- 172.21.15.116:0/2460590901 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f50b80113f0 con 0x7f50c0160820 2023-11-28T17:10:17.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.090+0000 7f50bdffb640 1 --2- 172.21.15.116:0/2460590901 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f50ac06b2a0 0x7f50ac06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:17.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.091+0000 7f50bdffb640 1 -- 172.21.15.116:0/2460590901 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f50b8089c50 con 0x7f50c0160820 2023-11-28T17:10:17.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.091+0000 7f50c5114640 1 --2- 172.21.15.116:0/2460590901 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f50ac06b2a0 0x7f50ac06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:17.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.092+0000 7f50c5114640 1 --2- 172.21.15.116:0/2460590901 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f50ac06b2a0 0x7f50ac06d760 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f50b0035000 tx=0x7f50b002e000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:17.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.095+0000 7f50bdffb640 1 -- 172.21.15.116:0/2460590901 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50b8054d70 con 0x7f50c0160820 2023-11-28T17:10:17.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:17.288+0000 7f50c7378640 1 -- 172.21.15.116:0/2460590901 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f50c0069360 con 0x7f50c0160820 2023-11-28T17:10:18.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.256+0000 7f50bdffb640 1 -- 172.21.15.116:0/2460590901 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f50b801f0b0 con 0x7f50c0160820 2023-11-28T17:10:18.258 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' created 2023-11-28T17:10:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.260+0000 7f50c7378640 1 -- 172.21.15.116:0/2460590901 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f50ac06b2a0 msgr2=0x7f50ac06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.260+0000 7f50c7378640 1 --2- 172.21.15.116:0/2460590901 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f50ac06b2a0 0x7f50ac06d760 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f50b0035000 tx=0x7f50b002e000 comp rx=0 tx=0).stop 2023-11-28T17:10:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.260+0000 7f50c7378640 1 -- 172.21.15.116:0/2460590901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0160820 msgr2=0x7f50c01f2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.260+0000 7f50c7378640 1 --2- 172.21.15.116:0/2460590901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0160820 0x7f50c01f2c00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f50b800c7f0 tx=0x7f50b800ccb0 comp rx=0 tx=0).stop 2023-11-28T17:10:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.260+0000 7f50c7378640 1 -- 172.21.15.116:0/2460590901 shutdown_connections 2023-11-28T17:10:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.260+0000 7f50c7378640 1 --2- 172.21.15.116:0/2460590901 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f50ac06b2a0 0x7f50ac06d760 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.260+0000 7f50c7378640 1 --2- 172.21.15.116:0/2460590901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0160820 0x7f50c01f2c00 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.260+0000 7f50c7378640 1 --2- 172.21.15.116:0/2460590901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50c00d1ea0 0x7f50c015f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.260+0000 7f50c7378640 1 --2- 172.21.15.116:0/2460590901 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f50c00d19f0 0x7f50c00cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.260+0000 7f50c7378640 1 -- 172.21.15.116:0/2460590901 >> 172.21.15.116:0/2460590901 conn(0x7f50c0072010 msgr2=0x7f50c0157bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:18.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.261+0000 7f50c7378640 1 -- 172.21.15.116:0/2460590901 shutdown_connections 2023-11-28T17:10:18.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.261+0000 7f50c7378640 1 -- 172.21.15.116:0/2460590901 wait complete. 2023-11-28T17:10:18.286 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T17:10:18.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.502+0000 7f87d3ae0640 1 Processor -- start 2023-11-28T17:10:18.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.502+0000 7f87d3ae0640 1 -- start start 2023-11-28T17:10:18.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.504+0000 7f87d3ae0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f87cc15ae10 0x7f87cc15e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:18.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.504+0000 7f87d3ae0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc0d1640 con 0x7f87cc160e60 2023-11-28T17:10:18.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.504+0000 7f87d3ae0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87cc15e830 con 0x7f87cc0d0850 2023-11-28T17:10:18.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.504+0000 7f87d3ae0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87cc161400 con 0x7f87cc15ae10 2023-11-28T17:10:18.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.504+0000 7f87d107b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f87cc160e60 0x7f87cc0d0140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49026/0 (socket says 172.21.15.116:49026) 2023-11-28T17:10:18.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.504+0000 7f87d107b640 1 -- 172.21.15.116:0/545350027 learned_addr learned my addr 172.21.15.116:0/545350027 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.505+0000 7f87d087a640 1 -- 172.21.15.116:0/545350027 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076326180 0 0) 0x7f87cc0d1640 con 0x7f87cc160e60 2023-11-28T17:10:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.505+0000 7f87d087a640 1 -- 172.21.15.116:0/545350027 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b8003520 con 0x7f87cc160e60 2023-11-28T17:10:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.505+0000 7f87d187c640 1 --2- 172.21.15.116:0/545350027 >> v2:172.21.15.156:3300/0 conn(0x7f87cc15ae10 0x7f87cc15e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.505+0000 7f87d087a640 1 -- 172.21.15.116:0/545350027 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1624806581 0 0) 0x7f87b8003520 con 0x7f87cc160e60 2023-11-28T17:10:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.505+0000 7f87d087a640 1 -- 172.21.15.116:0/545350027 >> v2:172.21.15.156:3300/0 conn(0x7f87cc15ae10 msgr2=0x7f87cc15e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.505+0000 7f87d087a640 1 --2- 172.21.15.116:0/545350027 >> v2:172.21.15.156:3300/0 conn(0x7f87cc15ae10 0x7f87cc15e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.505+0000 7f87d087a640 1 -- 172.21.15.116:0/545350027 >> v1:172.21.15.156:6789/0 conn(0x7f87cc0d0850 legacy=0x7f87cc0b8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.505+0000 7f87d087a640 1 -- 172.21.15.116:0/545350027 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87cc1616e0 con 0x7f87cc160e60 2023-11-28T17:10:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.505+0000 7f87d087a640 1 -- 172.21.15.116:0/545350027 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f87bc001670 con 0x7f87cc160e60 2023-11-28T17:10:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.505+0000 7f87d087a640 1 -- 172.21.15.116:0/545350027 >> v1:172.21.15.116:6789/0 conn(0x7f87cc160e60 legacy=0x7f87cc0d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:18.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.505+0000 7f87d087a640 1 --2- 172.21.15.116:0/545350027 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87b8003970 0x7f87b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.505+0000 7f87d087a640 1 --2- 172.21.15.116:0/545350027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87b80047c0 0x7f87b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.505+0000 7f87d087a640 1 --2- 172.21.15.116:0/545350027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87b80052a0 0x7f87b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.506+0000 7f87d087a640 1 -- 172.21.15.116:0/545350027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc161400 con 0x7f87b80047c0 2023-11-28T17:10:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.506+0000 7f87d087a640 1 -- 172.21.15.116:0/545350027 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b800c170 con 0x7f87b80052a0 2023-11-28T17:10:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.506+0000 7f87d087a640 1 -- 172.21.15.116:0/545350027 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b800c330 con 0x7f87b8003970 2023-11-28T17:10:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.506+0000 7f87d107b640 1 --2- 172.21.15.116:0/545350027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87b80047c0 0x7f87b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.506+0000 7f87d187c640 1 --2- 172.21.15.116:0/545350027 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87b8003970 0x7f87b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.506+0000 7f87d207d640 1 --2- 172.21.15.116:0/545350027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87b80052a0 0x7f87b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.506+0000 7f87d107b640 1 -- 172.21.15.116:0/545350027 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87b8003970 msgr2=0x7f87b8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.506+0000 7f87d107b640 1 --2- 172.21.15.116:0/545350027 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87b8003970 0x7f87b8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.506+0000 7f87d107b640 1 -- 172.21.15.116:0/545350027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87b80052a0 msgr2=0x7f87b800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.506+0000 7f87d107b640 1 --2- 172.21.15.116:0/545350027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87b80052a0 0x7f87b800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:18.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.506+0000 7f87d107b640 1 -- 172.21.15.116:0/545350027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b8003520 con 0x7f87b80047c0 2023-11-28T17:10:18.510 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.506+0000 7f87d207d640 1 -- 172.21.15.116:0/545350027 reap_dead start 2023-11-28T17:10:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.507+0000 7f87d107b640 1 --2- 172.21.15.116:0/545350027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87b80047c0 0x7f87b8004c40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00c210 tx=0x7f87b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a016123e11b9b152 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.508+0000 7f87d087a640 1 -- 172.21.15.116:0/545350027 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc005ad0 con 0x7f87b80047c0 2023-11-28T17:10:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.508+0000 7f87d087a640 1 -- 172.21.15.116:0/545350027 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87bc005c30 con 0x7f87b80047c0 2023-11-28T17:10:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.508+0000 7f87d087a640 1 -- 172.21.15.116:0/545350027 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc02cad0 con 0x7f87b80047c0 2023-11-28T17:10:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.508+0000 7f87d3ae0640 1 -- 172.21.15.116:0/545350027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87b80047c0 msgr2=0x7f87b8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.508+0000 7f87d3ae0640 1 --2- 172.21.15.116:0/545350027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87b80047c0 0x7f87b8004c40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00c210 tx=0x7f87b80057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.508+0000 7f87d3ae0640 1 -- 172.21.15.116:0/545350027 shutdown_connections 2023-11-28T17:10:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.508+0000 7f87d3ae0640 1 --2- 172.21.15.116:0/545350027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87b80047c0 0x7f87b8004c40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.508+0000 7f87d3ae0640 1 -- 172.21.15.116:0/545350027 >> 172.21.15.116:0/545350027 conn(0x7f87cc071fe0 msgr2=0x7f87cc0723f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.508+0000 7f87d3ae0640 1 -- 172.21.15.116:0/545350027 shutdown_connections 2023-11-28T17:10:18.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.508+0000 7f87d3ae0640 1 -- 172.21.15.116:0/545350027 wait complete. 2023-11-28T17:10:18.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.510+0000 7f87d3ae0640 1 Processor -- start 2023-11-28T17:10:18.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.510+0000 7f87d3ae0640 1 -- start start 2023-11-28T17:10:18.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.511+0000 7f87d3ae0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87b80047c0 0x7f87cc0c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:18.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.511+0000 7f87d3ae0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87cc0c65e0 0x7f87cc0c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:18.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.511+0000 7f87d3ae0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87cc0d3080 0x7f87cc0d3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:18.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.511+0000 7f87d3ae0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc0d3a40 con 0x7f87b80047c0 2023-11-28T17:10:18.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.511+0000 7f87d3ae0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc0c9ba0 con 0x7f87cc0c65e0 2023-11-28T17:10:18.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.511+0000 7f87d3ae0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc0c8cf0 con 0x7f87cc0d3080 2023-11-28T17:10:18.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.512+0000 7f87d187c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87b80047c0 0x7f87cc0c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:18.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.512+0000 7f87d187c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87b80047c0 0x7f87cc0c7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57446/0 (socket says 172.21.15.116:57446) 2023-11-28T17:10:18.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.512+0000 7f87d187c640 1 -- 172.21.15.116:0/1382132738 learned_addr learned my addr 172.21.15.116:0/1382132738 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:18.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.512+0000 7f87d107b640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87cc0c65e0 0x7f87cc0c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:18.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.512+0000 7f87d207d640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87cc0d3080 0x7f87cc0d3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:18.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.512+0000 7f87d187c640 1 -- 172.21.15.116:0/1382132738 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87cc0d3080 msgr2=0x7f87cc0d3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:18.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.512+0000 7f87d187c640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87cc0d3080 0x7f87cc0d3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:18.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.512+0000 7f87d187c640 1 -- 172.21.15.116:0/1382132738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87cc0c65e0 msgr2=0x7f87cc0c87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:18.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.512+0000 7f87d187c640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87cc0c65e0 0x7f87cc0c87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:18.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.512+0000 7f87d187c640 1 -- 172.21.15.116:0/1382132738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87cc0c8f90 con 0x7f87b80047c0 2023-11-28T17:10:18.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.512+0000 7f87d207d640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87cc0d3080 0x7f87cc0d3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:18.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.512+0000 7f87d187c640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87b80047c0 0x7f87cc0c7a00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f87c800f600 tx=0x7f87c800fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:18.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.513+0000 7f87c27fc640 1 -- 172.21.15.116:0/1382132738 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8014870 con 0x7f87b80047c0 2023-11-28T17:10:18.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.513+0000 7f87c27fc640 1 -- 172.21.15.116:0/1382132738 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c80149d0 con 0x7f87b80047c0 2023-11-28T17:10:18.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.513+0000 7f87c27fc640 1 -- 172.21.15.116:0/1382132738 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8014d40 con 0x7f87b80047c0 2023-11-28T17:10:18.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.513+0000 7f87d3ae0640 1 -- 172.21.15.116:0/1382132738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87cc0c9270 con 0x7f87b80047c0 2023-11-28T17:10:18.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.513+0000 7f87d3ae0640 1 -- 172.21.15.116:0/1382132738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87cc0c9830 con 0x7f87b80047c0 2023-11-28T17:10:18.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.514+0000 7f87d3ae0640 1 -- 172.21.15.116:0/1382132738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87cc0693d0 con 0x7f87b80047c0 2023-11-28T17:10:18.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.515+0000 7f87c27fc640 1 -- 172.21.15.116:0/1382132738 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f87c800d490 con 0x7f87b80047c0 2023-11-28T17:10:18.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.516+0000 7f87c27fc640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f87b006b2a0 0x7f87b006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:18.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.516+0000 7f87c27fc640 1 -- 172.21.15.116:0/1382132738 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f87c808a5a0 con 0x7f87b80047c0 2023-11-28T17:10:18.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.516+0000 7f87d107b640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f87b006b2a0 0x7f87b006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:18.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.517+0000 7f87d107b640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f87b006b2a0 0x7f87b006d760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f87bc034000 tx=0x7f87bc009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:18.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.521+0000 7f87c27fc640 1 -- 172.21.15.116:0/1382132738 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87c80554c0 con 0x7f87b80047c0 2023-11-28T17:10:18.712 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:18.710+0000 7f87d3ae0640 1 -- 172.21.15.116:0/1382132738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f87cc15e2c0 con 0x7f87b80047c0 2023-11-28T17:10:19.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.263+0000 7f87c27fc640 1 -- 172.21.15.116:0/1382132738 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7f87c8058ae0 con 0x7f87b80047c0 2023-11-28T17:10:19.265 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' created 2023-11-28T17:10:19.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.267+0000 7f87d3ae0640 1 -- 172.21.15.116:0/1382132738 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f87b006b2a0 msgr2=0x7f87b006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:19.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.267+0000 7f87d3ae0640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f87b006b2a0 0x7f87b006d760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f87bc034000 tx=0x7f87bc009030 comp rx=0 tx=0).stop 2023-11-28T17:10:19.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.267+0000 7f87d3ae0640 1 -- 172.21.15.116:0/1382132738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87b80047c0 msgr2=0x7f87cc0c7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:19.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.267+0000 7f87d3ae0640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87b80047c0 0x7f87cc0c7a00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f87c800f600 tx=0x7f87c800fac0 comp rx=0 tx=0).stop 2023-11-28T17:10:19.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.267+0000 7f87d3ae0640 1 -- 172.21.15.116:0/1382132738 shutdown_connections 2023-11-28T17:10:19.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.267+0000 7f87d3ae0640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f87b006b2a0 0x7f87b006d760 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:19.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.267+0000 7f87d3ae0640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87cc0d3080 0x7f87cc0d3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:19.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.267+0000 7f87d3ae0640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87cc0c65e0 0x7f87cc0c87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:19.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.267+0000 7f87d3ae0640 1 --2- 172.21.15.116:0/1382132738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87b80047c0 0x7f87cc0c7a00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:19.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.267+0000 7f87d3ae0640 1 -- 172.21.15.116:0/1382132738 >> 172.21.15.116:0/1382132738 conn(0x7f87cc071fe0 msgr2=0x7f87cc0d2450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:19.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.268+0000 7f87d3ae0640 1 -- 172.21.15.116:0/1382132738 shutdown_connections 2023-11-28T17:10:19.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.268+0000 7f87d3ae0640 1 -- 172.21.15.116:0/1382132738 wait complete. 2023-11-28T17:10:19.302 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T17:10:19.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.487+0000 7fd81fb61640 1 Processor -- start 2023-11-28T17:10:19.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.487+0000 7fd81fb61640 1 -- start start 2023-11-28T17:10:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.489+0000 7fd81fb61640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd8180b4600 0x7fd8180cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:19.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.489+0000 7fd81fb61640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd81814ac30 0x7fd818070ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:19.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.489+0000 7fd81fb61640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd818070420 con 0x7fd81814a740 2023-11-28T17:10:19.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.489+0000 7fd81fb61640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd818070aa0 con 0x7fd8180b4600 2023-11-28T17:10:19.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.489+0000 7fd81fb61640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8180b2fc0 con 0x7fd81814ac30 2023-11-28T17:10:19.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.489+0000 7fd81d0fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd8180b4600 0x7fd8180cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:19.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.489+0000 7fd81d0fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd8180b4600 0x7fd8180cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57462/0 (socket says 172.21.15.116:57462) 2023-11-28T17:10:19.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.489+0000 7fd81d0fc640 1 -- 172.21.15.116:0/2502676807 learned_addr learned my addr 172.21.15.116:0/2502676807 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.490+0000 7fd81e0fe640 1 --2- 172.21.15.116:0/2502676807 >> v2:172.21.15.156:3300/0 conn(0x7fd81814ac30 0x7fd818070ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.490+0000 7fd81d0fc640 1 -- 172.21.15.116:0/2502676807 >> v2:172.21.15.156:3300/0 conn(0x7fd81814ac30 msgr2=0x7fd818070ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.490+0000 7fd81d0fc640 1 --2- 172.21.15.116:0/2502676807 >> v2:172.21.15.156:3300/0 conn(0x7fd81814ac30 0x7fd818070ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.490+0000 7fd81d0fc640 1 -- 172.21.15.116:0/2502676807 >> v1:172.21.15.116:6789/0 conn(0x7fd81814a740 legacy=0x7fd8180b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.490+0000 7fd81d0fc640 1 -- 172.21.15.116:0/2502676807 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8180706c0 con 0x7fd8180b4600 2023-11-28T17:10:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.490+0000 7fd81e0fe640 1 --2- 172.21.15.116:0/2502676807 >> v2:172.21.15.156:3300/0 conn(0x7fd81814ac30 0x7fd818070ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.491+0000 7fd81d0fc640 1 --2- 172.21.15.116:0/2502676807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8180b4600 0x7fd8180cc7a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd818166360 tx=0x7fd80802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5f99f42f6aaadee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:19.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.491+0000 7fd81c8fb640 1 -- 172.21.15.116:0/2502676807 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808030870 con 0x7fd8180b4600 2023-11-28T17:10:19.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.491+0000 7fd81c8fb640 1 -- 172.21.15.116:0/2502676807 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8080309d0 con 0x7fd8180b4600 2023-11-28T17:10:19.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.491+0000 7fd81c8fb640 1 -- 172.21.15.116:0/2502676807 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808030d60 con 0x7fd8180b4600 2023-11-28T17:10:19.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.492+0000 7fd81fb61640 1 -- 172.21.15.116:0/2502676807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8180b4600 msgr2=0x7fd8180cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:19.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.492+0000 7fd81fb61640 1 --2- 172.21.15.116:0/2502676807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8180b4600 0x7fd8180cc7a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd818166360 tx=0x7fd80802b9b0 comp rx=0 tx=0).stop 2023-11-28T17:10:19.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.492+0000 7fd81fb61640 1 -- 172.21.15.116:0/2502676807 shutdown_connections 2023-11-28T17:10:19.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.492+0000 7fd81fb61640 1 --2- 172.21.15.116:0/2502676807 >> v2:172.21.15.156:3300/0 conn(0x7fd81814ac30 0x7fd818070ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:19.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.492+0000 7fd81fb61640 1 --2- 172.21.15.116:0/2502676807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd8180b4600 0x7fd8180cc7a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:19.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.492+0000 7fd81fb61640 1 -- 172.21.15.116:0/2502676807 >> 172.21.15.116:0/2502676807 conn(0x7fd8180b0cf0 msgr2=0x7fd8180b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:19.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.492+0000 7fd81fb61640 1 -- 172.21.15.116:0/2502676807 shutdown_connections 2023-11-28T17:10:19.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.492+0000 7fd81fb61640 1 -- 172.21.15.116:0/2502676807 wait complete. 2023-11-28T17:10:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.494+0000 7fd81fb61640 1 Processor -- start 2023-11-28T17:10:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.494+0000 7fd81fb61640 1 -- start start 2023-11-28T17:10:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.495+0000 7fd81fb61640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8180b4600 0x7fd8181421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.495+0000 7fd81fb61640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd81814a740 0x7fd818142ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:19.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.495+0000 7fd81fb61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81814ac30 0x7fd818143220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.495+0000 7fd81fb61640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd818142b40 con 0x7fd81814ac30 2023-11-28T17:10:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.495+0000 7fd81fb61640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd818145540 con 0x7fd81814a740 2023-11-28T17:10:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.495+0000 7fd81fb61640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd818144780 con 0x7fd8180b4600 2023-11-28T17:10:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.495+0000 7fd81e0fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81814ac30 0x7fd818143220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.495+0000 7fd81e0fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81814ac30 0x7fd818143220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57476/0 (socket says 172.21.15.116:57476) 2023-11-28T17:10:19.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.495+0000 7fd81e0fe640 1 -- 172.21.15.116:0/3418925709 learned_addr learned my addr 172.21.15.116:0/3418925709 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.496+0000 7fd81e0fe640 1 -- 172.21.15.116:0/3418925709 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8180b4600 msgr2=0x7fd8181421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.496+0000 7fd81e0fe640 1 --2- 172.21.15.116:0/3418925709 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8180b4600 0x7fd8181421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.496+0000 7fd81e0fe640 1 -- 172.21.15.116:0/3418925709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd81814a740 msgr2=0x7fd818142ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.496+0000 7fd81e0fe640 1 --2- 172.21.15.116:0/3418925709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd81814a740 0x7fd818142ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:19.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.496+0000 7fd81e0fe640 1 -- 172.21.15.116:0/3418925709 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8080094d0 con 0x7fd81814ac30 2023-11-28T17:10:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.496+0000 7fd81e0fe640 1 --2- 172.21.15.116:0/3418925709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81814ac30 0x7fd818143220 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd81000c7f0 tx=0x7fd81000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.496+0000 7fd80e7fc640 1 -- 172.21.15.116:0/3418925709 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd810007820 con 0x7fd81814ac30 2023-11-28T17:10:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.496+0000 7fd80e7fc640 1 -- 172.21.15.116:0/3418925709 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd810007980 con 0x7fd81814ac30 2023-11-28T17:10:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.496+0000 7fd81fb61640 1 -- 172.21.15.116:0/3418925709 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd818144a60 con 0x7fd81814ac30 2023-11-28T17:10:19.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.496+0000 7fd80e7fc640 1 -- 172.21.15.116:0/3418925709 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd810007d70 con 0x7fd81814ac30 2023-11-28T17:10:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.496+0000 7fd81fb61640 1 -- 172.21.15.116:0/3418925709 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd818145020 con 0x7fd81814ac30 2023-11-28T17:10:19.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.497+0000 7fd81fb61640 1 -- 172.21.15.116:0/3418925709 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8180cbe70 con 0x7fd81814ac30 2023-11-28T17:10:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.499+0000 7fd80e7fc640 1 -- 172.21.15.116:0/3418925709 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fd8100113f0 con 0x7fd81814ac30 2023-11-28T17:10:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.499+0000 7fd80e7fc640 1 --2- 172.21.15.116:0/3418925709 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd7fc06b2a0 0x7fd7fc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.500+0000 7fd80e7fc640 1 -- 172.21.15.116:0/3418925709 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fd810089eb0 con 0x7fd81814ac30 2023-11-28T17:10:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.500+0000 7fd81d8fd640 1 --2- 172.21.15.116:0/3418925709 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd7fc06b2a0 0x7fd7fc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.500+0000 7fd81d8fd640 1 --2- 172.21.15.116:0/3418925709 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd7fc06b2a0 0x7fd7fc06d760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd8140012e0 tx=0x7fd814007d30 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:19.505 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.504+0000 7fd80e7fc640 1 -- 172.21.15.116:0/3418925709 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd810054d70 con 0x7fd81814ac30 2023-11-28T17:10:19.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:19.723+0000 7fd81fb61640 1 -- 172.21.15.116:0/3418925709 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd818145210 con 0x7fd81814ac30 2023-11-28T17:10:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.317+0000 7fd80e7fc640 1 -- 172.21.15.116:0/3418925709 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd81001f0c0 con 0x7fd81814ac30 2023-11-28T17:10:20.321 INFO:teuthology.orchestra.run.smithi116.stderr:new fs with metadata pool 4 and data pool 5 2023-11-28T17:10:20.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.334+0000 7fd81fb61640 1 -- 172.21.15.116:0/3418925709 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd7fc06b2a0 msgr2=0x7fd7fc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:20.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.334+0000 7fd81fb61640 1 --2- 172.21.15.116:0/3418925709 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd7fc06b2a0 0x7fd7fc06d760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd8140012e0 tx=0x7fd814007d30 comp rx=0 tx=0).stop 2023-11-28T17:10:20.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.334+0000 7fd81fb61640 1 -- 172.21.15.116:0/3418925709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81814ac30 msgr2=0x7fd818143220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:20.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.334+0000 7fd81fb61640 1 --2- 172.21.15.116:0/3418925709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81814ac30 0x7fd818143220 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd81000c7f0 tx=0x7fd81000ccb0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.334+0000 7fd81fb61640 1 -- 172.21.15.116:0/3418925709 shutdown_connections 2023-11-28T17:10:20.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.334+0000 7fd81fb61640 1 --2- 172.21.15.116:0/3418925709 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd7fc06b2a0 0x7fd7fc06d760 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.335+0000 7fd81fb61640 1 --2- 172.21.15.116:0/3418925709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd81814ac30 0x7fd818143220 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.335+0000 7fd81fb61640 1 --2- 172.21.15.116:0/3418925709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd81814a740 0x7fd818142ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.335+0000 7fd81fb61640 1 --2- 172.21.15.116:0/3418925709 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8180b4600 0x7fd8181421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.335+0000 7fd81fb61640 1 -- 172.21.15.116:0/3418925709 >> 172.21.15.116:0/3418925709 conn(0x7fd8180b0cf0 msgr2=0x7fd8180cc760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:20.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.335+0000 7fd81fb61640 1 -- 172.21.15.116:0/3418925709 shutdown_connections 2023-11-28T17:10:20.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.335+0000 7fd81fb61640 1 -- 172.21.15.116:0/3418925709 wait complete. 2023-11-28T17:10:20.366 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:10:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.542+0000 7fd9c18ff640 1 Processor -- start 2023-11-28T17:10:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.543+0000 7fd9c18ff640 1 -- start start 2023-11-28T17:10:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.544+0000 7fd9c18ff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd9bc159f30 0x7fd9bc0701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.544+0000 7fd9c18ff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9bc15ae90 0x7fd9bc0ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.544+0000 7fd9c18ff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd9bc0ceb90 0x7fd9bc0cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.544+0000 7fd9c18ff640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9bc070730 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.544+0000 7fd9c18ff640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9bc070ee0 con 0x7fd9bc159f30 2023-11-28T17:10:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.544+0000 7fd9c18ff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc0cf550 con 0x7fd9bc15ae90 2023-11-28T17:10:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.544+0000 7fd9c10fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd9bc0ceb90 0x7fd9bc0cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.545+0000 7fd9c10fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd9bc0ceb90 0x7fd9bc0cf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57490/0 (socket says 172.21.15.116:57490) 2023-11-28T17:10:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.545+0000 7fd9c10fe640 1 -- 172.21.15.116:0/214253589 learned_addr learned my addr 172.21.15.116:0/214253589 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.545+0000 7fd9bbfff640 1 --2- 172.21.15.116:0/214253589 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9bc15ae90 0x7fd9bc0ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.545+0000 7fd9c08fd640 1 --2- 172.21.15.116:0/214253589 >> v2:172.21.15.156:3300/0 conn(0x7fd9bc159f30 0x7fd9bc0701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.545+0000 7fd9c10fe640 1 -- 172.21.15.116:0/214253589 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9bc15ae90 msgr2=0x7fd9bc0ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.545+0000 7fd9c10fe640 1 --2- 172.21.15.116:0/214253589 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9bc15ae90 0x7fd9bc0ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.545+0000 7fd9c10fe640 1 -- 172.21.15.116:0/214253589 >> v2:172.21.15.156:3300/0 conn(0x7fd9bc159f30 msgr2=0x7fd9bc0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.545+0000 7fd9c10fe640 1 --2- 172.21.15.116:0/214253589 >> v2:172.21.15.156:3300/0 conn(0x7fd9bc159f30 0x7fd9bc0701f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.545+0000 7fd9c10fe640 1 -- 172.21.15.116:0/214253589 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9bc15b2f0 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.545+0000 7fd9bbfff640 1 --2- 172.21.15.116:0/214253589 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9bc15ae90 0x7fd9bc0ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.546+0000 7fd9c10fe640 1 --2- 172.21.15.116:0/214253589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9bc0ceb90 0x7fd9bc0cf010 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0adfe0 tx=0x7fd9b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b28f6316fd32e61 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.546+0000 7fd9bb7fe640 1 -- 172.21.15.116:0/214253589 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b402ca70 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.546+0000 7fd9bb7fe640 1 -- 172.21.15.116:0/214253589 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b402cbd0 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.546+0000 7fd9bb7fe640 1 -- 172.21.15.116:0/214253589 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b40343f0 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.547+0000 7fd9c18ff640 1 -- 172.21.15.116:0/214253589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9bc0ceb90 msgr2=0x7fd9bc0cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.547+0000 7fd9c18ff640 1 --2- 172.21.15.116:0/214253589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9bc0ceb90 0x7fd9bc0cf010 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0adfe0 tx=0x7fd9b402bed0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.547+0000 7fd9c18ff640 1 -- 172.21.15.116:0/214253589 shutdown_connections 2023-11-28T17:10:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.547+0000 7fd9c18ff640 1 --2- 172.21.15.116:0/214253589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9bc0ceb90 0x7fd9bc0cf010 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.547+0000 7fd9c18ff640 1 --2- 172.21.15.116:0/214253589 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9bc15ae90 0x7fd9bc0ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.547+0000 7fd9c18ff640 1 --2- 172.21.15.116:0/214253589 >> v2:172.21.15.156:3300/0 conn(0x7fd9bc159f30 0x7fd9bc0701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.547+0000 7fd9c18ff640 1 -- 172.21.15.116:0/214253589 >> 172.21.15.116:0/214253589 conn(0x7fd9bc0b0950 msgr2=0x7fd9bc0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:20.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.547+0000 7fd9c18ff640 1 -- 172.21.15.116:0/214253589 shutdown_connections 2023-11-28T17:10:20.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.547+0000 7fd9c18ff640 1 -- 172.21.15.116:0/214253589 wait complete. 2023-11-28T17:10:20.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.549+0000 7fd9c18ff640 1 Processor -- start 2023-11-28T17:10:20.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.549+0000 7fd9c18ff640 1 -- start start 2023-11-28T17:10:20.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.550+0000 7fd9c18ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9bc0ceb90 0x7fd9bc151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:20.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.550+0000 7fd9c18ff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9bc159f30 0x7fd9bc0ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:20.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.550+0000 7fd9c18ff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd9bc15ae90 0x7fd9bc06cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:20.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.550+0000 7fd9c18ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc153610 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.550+0000 7fd9c18ff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc153050 con 0x7fd9bc15ae90 2023-11-28T17:10:20.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.550+0000 7fd9c18ff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc153330 con 0x7fd9bc159f30 2023-11-28T17:10:20.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.550+0000 7fd9c08fd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9bc0ceb90 0x7fd9bc151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:20.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.550+0000 7fd9c08fd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9bc0ceb90 0x7fd9bc151a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57494/0 (socket says 172.21.15.116:57494) 2023-11-28T17:10:20.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.550+0000 7fd9c08fd640 1 -- 172.21.15.116:0/3466372113 learned_addr learned my addr 172.21.15.116:0/3466372113 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:20.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.551+0000 7fd9c08fd640 1 -- 172.21.15.116:0/3466372113 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9bc159f30 msgr2=0x7fd9bc0ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:20.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.551+0000 7fd9c08fd640 1 --2- 172.21.15.116:0/3466372113 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9bc159f30 0x7fd9bc0ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.551+0000 7fd9c08fd640 1 -- 172.21.15.116:0/3466372113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd9bc15ae90 msgr2=0x7fd9bc06cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:20.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.551+0000 7fd9c10fe640 1 --2- 172.21.15.116:0/3466372113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd9bc15ae90 0x7fd9bc06cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:20.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.551+0000 7fd9c08fd640 1 --2- 172.21.15.116:0/3466372113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd9bc15ae90 0x7fd9bc06cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.551+0000 7fd9c08fd640 1 -- 172.21.15.116:0/3466372113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b4009b30 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.551+0000 7fd9c08fd640 1 --2- 172.21.15.116:0/3466372113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9bc0ceb90 0x7fd9bc151a60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000c7f0 tx=0x7fd9b000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.551+0000 7fd9b97fa640 1 -- 172.21.15.116:0/3466372113 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0004220 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.551+0000 7fd9b97fa640 1 -- 172.21.15.116:0/3466372113 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b00043a0 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.551+0000 7fd9b97fa640 1 -- 172.21.15.116:0/3466372113 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0007860 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.551+0000 7fd9c18ff640 1 -- 172.21.15.116:0/3466372113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9bc099ba0 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.551+0000 7fd9c18ff640 1 -- 172.21.15.116:0/3466372113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9bc151fa0 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.552+0000 7fd9c18ff640 1 -- 172.21.15.116:0/3466372113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9bc099a00 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.553+0000 7fd9b97fa640 1 -- 172.21.15.116:0/3466372113 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fd9b0007bd0 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.554+0000 7fd9b97fa640 1 --2- 172.21.15.116:0/3466372113 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd9a006b2a0 0x7fd9a006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.554+0000 7fd9b97fa640 1 -- 172.21.15.116:0/3466372113 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9b0089b50 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.554+0000 7fd9bbfff640 1 --2- 172.21.15.116:0/3466372113 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd9a006b2a0 0x7fd9a006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.555+0000 7fd9bbfff640 1 --2- 172.21.15.116:0/3466372113 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd9a006b2a0 0x7fd9a006d760 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8006b10 tx=0x7fd9a8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.558+0000 7fd9b97fa640 1 -- 172.21.15.116:0/3466372113 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9bc099a00 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.741+0000 7fd9c18ff640 1 -- 172.21.15.116:0/3466372113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9bc099a00 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.743+0000 7fd9b97fa640 1 -- 172.21.15.116:0/3466372113 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16534 (secure 0 0 0) 0x7fd9bc099a00 con 0x7fd9bc0ceb90 2023-11-28T17:10:20.745 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:20.745 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":33,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:10:19.725457+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:10:17.338444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-28T17:10:18.766338+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:10:20.747 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.745+0000 7fd9c18ff640 1 -- 172.21.15.116:0/3466372113 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd9a006b2a0 msgr2=0x7fd9a006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:20.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.745+0000 7fd9c18ff640 1 --2- 172.21.15.116:0/3466372113 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd9a006b2a0 0x7fd9a006d760 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8006b10 tx=0x7fd9a8008040 comp rx=0 tx=0).stop 2023-11-28T17:10:20.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.745+0000 7fd9c18ff640 1 -- 172.21.15.116:0/3466372113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9bc0ceb90 msgr2=0x7fd9bc151a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:20.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.745+0000 7fd9c18ff640 1 --2- 172.21.15.116:0/3466372113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9bc0ceb90 0x7fd9bc151a60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000c7f0 tx=0x7fd9b000ccb0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.746+0000 7fd9c18ff640 1 -- 172.21.15.116:0/3466372113 shutdown_connections 2023-11-28T17:10:20.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.746+0000 7fd9c18ff640 1 --2- 172.21.15.116:0/3466372113 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd9a006b2a0 0x7fd9a006d760 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.746+0000 7fd9c18ff640 1 --2- 172.21.15.116:0/3466372113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd9bc15ae90 0x7fd9bc06cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.746+0000 7fd9c18ff640 1 --2- 172.21.15.116:0/3466372113 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9bc159f30 0x7fd9bc0ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.746+0000 7fd9c18ff640 1 --2- 172.21.15.116:0/3466372113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9bc0ceb90 0x7fd9bc151a60 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.746+0000 7fd9c18ff640 1 -- 172.21.15.116:0/3466372113 >> 172.21.15.116:0/3466372113 conn(0x7fd9bc0b0950 msgr2=0x7fd9bc0aede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:20.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.746+0000 7fd9c18ff640 1 -- 172.21.15.116:0/3466372113 shutdown_connections 2023-11-28T17:10:20.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.746+0000 7fd9c18ff640 1 -- 172.21.15.116:0/3466372113 wait complete. 2023-11-28T17:10:20.765 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:10:20.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.949+0000 7f1caa085640 1 Processor -- start 2023-11-28T17:10:20.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.949+0000 7f1caa085640 1 -- start start 2023-11-28T17:10:20.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.951+0000 7f1caa085640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1ca40cdf90 0x7f1ca40ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:20.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.951+0000 7f1caa085640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca4070560 con 0x7f1ca40ceb90 2023-11-28T17:10:20.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.951+0000 7f1caa085640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca4070840 con 0x7f1ca415ae20 2023-11-28T17:10:20.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.951+0000 7f1caa085640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca406d3e0 con 0x7f1ca40cdf90 2023-11-28T17:10:20.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.951+0000 7f1ca8882640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1ca40cdf90 0x7f1ca40ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:20.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.951+0000 7f1ca9884640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1ca40ceb90 0x7f1ca415ee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49048/0 (socket says 172.21.15.116:49048) 2023-11-28T17:10:20.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.951+0000 7f1ca9884640 1 -- 172.21.15.116:0/4292387936 learned_addr learned my addr 172.21.15.116:0/4292387936 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:20.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.952+0000 7f1ca8882640 1 -- 172.21.15.116:0/4292387936 >> v1:172.21.15.156:6789/0 conn(0x7f1ca415ae20 legacy=0x7f1ca406ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:20.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.952+0000 7f1ca8882640 1 -- 172.21.15.116:0/4292387936 >> v1:172.21.15.116:6789/0 conn(0x7f1ca40ceb90 legacy=0x7f1ca415ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:20.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.952+0000 7f1c9bfff640 1 -- 172.21.15.116:0/4292387936 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752808763 0 0) 0x7f1ca4070560 con 0x7f1ca40ceb90 2023-11-28T17:10:20.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.952+0000 7f1ca8882640 1 -- 172.21.15.116:0/4292387936 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca4070b20 con 0x7f1ca40cdf90 2023-11-28T17:10:20.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.953+0000 7f1ca8882640 1 --2- 172.21.15.116:0/4292387936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca40cdf90 0x7f1ca40ce420 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1c940097d0 tx=0x7f1c9402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6503bcf39cc6e8bf server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.953+0000 7f1c9bfff640 1 -- 172.21.15.116:0/4292387936 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9402e070 con 0x7f1ca40cdf90 2023-11-28T17:10:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.953+0000 7f1c9bfff640 1 -- 172.21.15.116:0/4292387936 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c940315a0 con 0x7f1ca40cdf90 2023-11-28T17:10:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.953+0000 7f1c9bfff640 1 -- 172.21.15.116:0/4292387936 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94031930 con 0x7f1ca40cdf90 2023-11-28T17:10:20.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.954+0000 7f1caa085640 1 -- 172.21.15.116:0/4292387936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca40cdf90 msgr2=0x7f1ca40ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.954+0000 7f1caa085640 1 --2- 172.21.15.116:0/4292387936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca40cdf90 0x7f1ca40ce420 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1c940097d0 tx=0x7f1c9402b840 comp rx=0 tx=0).stop 2023-11-28T17:10:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.954+0000 7f1caa085640 1 -- 172.21.15.116:0/4292387936 shutdown_connections 2023-11-28T17:10:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.954+0000 7f1caa085640 1 --2- 172.21.15.116:0/4292387936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca40cdf90 0x7f1ca40ce420 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.954+0000 7f1caa085640 1 -- 172.21.15.116:0/4292387936 >> 172.21.15.116:0/4292387936 conn(0x7f1ca40b0950 msgr2=0x7f1ca40b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.954+0000 7f1caa085640 1 -- 172.21.15.116:0/4292387936 shutdown_connections 2023-11-28T17:10:20.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.954+0000 7f1caa085640 1 -- 172.21.15.116:0/4292387936 wait complete. 2023-11-28T17:10:20.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.956+0000 7f1caa085640 1 Processor -- start 2023-11-28T17:10:20.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.956+0000 7f1caa085640 1 -- start start 2023-11-28T17:10:20.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.957+0000 7f1caa085640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ca40cdf90 0x7f1ca4159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:20.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.957+0000 7f1caa085640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ca40ceb90 0x7f1ca415d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:20.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.957+0000 7f1caa085640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca415ae20 0x7f1ca41581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:20.959 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.957+0000 7f1caa085640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca415c560 con 0x7f1ca415ae20 2023-11-28T17:10:20.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.957+0000 7f1caa085640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca415d380 con 0x7f1ca40ceb90 2023-11-28T17:10:20.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.957+0000 7f1caa085640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca415c770 con 0x7f1ca40cdf90 2023-11-28T17:10:20.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.957+0000 7f1ca9884640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca415ae20 0x7f1ca41581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:20.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.957+0000 7f1ca9884640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca415ae20 0x7f1ca41581d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57512/0 (socket says 172.21.15.116:57512) 2023-11-28T17:10:20.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.957+0000 7f1ca9884640 1 -- 172.21.15.116:0/1942681944 learned_addr learned my addr 172.21.15.116:0/1942681944 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:20.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.957+0000 7f1ca9884640 1 -- 172.21.15.116:0/1942681944 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ca40cdf90 msgr2=0x7f1ca4159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:20.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.957+0000 7f1ca8882640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ca40ceb90 0x7f1ca415d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:20.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.957+0000 7f1ca9083640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ca40cdf90 0x7f1ca4159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:20.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.958+0000 7f1ca9884640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ca40cdf90 0x7f1ca4159800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.958+0000 7f1ca9884640 1 -- 172.21.15.116:0/1942681944 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ca40ceb90 msgr2=0x7f1ca415d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:20.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.958+0000 7f1ca9884640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ca40ceb90 0x7f1ca415d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:20.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.958+0000 7f1ca9884640 1 -- 172.21.15.116:0/1942681944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c94009400 con 0x7f1ca415ae20 2023-11-28T17:10:20.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.958+0000 7f1ca8882640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ca40ceb90 0x7f1ca415d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:20.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.958+0000 7f1ca9083640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ca40cdf90 0x7f1ca4159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:20.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.958+0000 7f1ca9884640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca415ae20 0x7f1ca41581d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00c7e0 tx=0x7f1c9c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:20.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.958+0000 7f1c99ffb640 1 -- 172.21.15.116:0/1942681944 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c00a640 con 0x7f1ca415ae20 2023-11-28T17:10:20.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.958+0000 7f1c99ffb640 1 -- 172.21.15.116:0/1942681944 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9c00a7c0 con 0x7f1ca415ae20 2023-11-28T17:10:20.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.958+0000 7f1caa085640 1 -- 172.21.15.116:0/1942681944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca415b380 con 0x7f1ca415ae20 2023-11-28T17:10:20.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.958+0000 7f1c99ffb640 1 -- 172.21.15.116:0/1942681944 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c00ab70 con 0x7f1ca415ae20 2023-11-28T17:10:20.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.958+0000 7f1caa085640 1 -- 172.21.15.116:0/1942681944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca415cd30 con 0x7f1ca415ae20 2023-11-28T17:10:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.959+0000 7f1caa085640 1 -- 172.21.15.116:0/1942681944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca409d0b0 con 0x7f1ca415ae20 2023-11-28T17:10:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.960+0000 7f1c99ffb640 1 -- 172.21.15.116:0/1942681944 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f1c9c00acd0 con 0x7f1ca415ae20 2023-11-28T17:10:20.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.961+0000 7f1c99ffb640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1c8806b2a0 0x7f1c8806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.961+0000 7f1c99ffb640 1 -- 172.21.15.116:0/1942681944 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c9c089400 con 0x7f1ca415ae20 2023-11-28T17:10:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.961+0000 7f1ca9083640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1c8806b2a0 0x7f1c8806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:20.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.962+0000 7f1ca9083640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1c8806b2a0 0x7f1c8806d760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000a830 tx=0x7f1ca0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:20.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:20.966+0000 7f1c99ffb640 1 -- 172.21.15.116:0/1942681944 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c9c054300 con 0x7f1ca415ae20 2023-11-28T17:10:21.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.156+0000 7f1caa085640 1 -- 172.21.15.116:0/1942681944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ca415d4c0 con 0x7f1ca415ae20 2023-11-28T17:10:21.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.159+0000 7f1c99ffb640 1 -- 172.21.15.116:0/1942681944 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16534 (secure 0 0 0) 0x7f1c9c057920 con 0x7f1ca415ae20 2023-11-28T17:10:21.162 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:21.162 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":33,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:10:19.725457+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:10:17.338444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-28T17:10:18.766338+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:10:21.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.161+0000 7f1caa085640 1 -- 172.21.15.116:0/1942681944 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1c8806b2a0 msgr2=0x7f1c8806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:21.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.161+0000 7f1caa085640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1c8806b2a0 0x7f1c8806d760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000a830 tx=0x7f1ca0008040 comp rx=0 tx=0).stop 2023-11-28T17:10:21.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.161+0000 7f1caa085640 1 -- 172.21.15.116:0/1942681944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca415ae20 msgr2=0x7f1ca41581d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:21.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.161+0000 7f1caa085640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca415ae20 0x7f1ca41581d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00c7e0 tx=0x7f1c9c00cca0 comp rx=0 tx=0).stop 2023-11-28T17:10:21.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.161+0000 7f1caa085640 1 -- 172.21.15.116:0/1942681944 shutdown_connections 2023-11-28T17:10:21.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.161+0000 7f1caa085640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f1c8806b2a0 0x7f1c8806d760 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:21.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.161+0000 7f1caa085640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca415ae20 0x7f1ca41581d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:21.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.161+0000 7f1caa085640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ca40ceb90 0x7f1ca415d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:21.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.161+0000 7f1caa085640 1 --2- 172.21.15.116:0/1942681944 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ca40cdf90 0x7f1ca4159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:21.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.161+0000 7f1caa085640 1 -- 172.21.15.116:0/1942681944 >> 172.21.15.116:0/1942681944 conn(0x7f1ca40b0950 msgr2=0x7f1ca40add40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:21.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.162+0000 7f1caa085640 1 -- 172.21.15.116:0/1942681944 shutdown_connections 2023-11-28T17:10:21.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.162+0000 7f1caa085640 1 -- 172.21.15.116:0/1942681944 wait complete. 2023-11-28T17:10:21.183 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T17:10:21.241 INFO:tasks.ceph.mds.c.smithi116.stderr:2023-11-28T17:10:21.239+0000 7f56ff6d0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T17:10:21.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.397+0000 7fcdde23d640 1 Processor -- start 2023-11-28T17:10:21.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.397+0000 7fcdde23d640 1 -- start start 2023-11-28T17:10:21.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.398+0000 7fcdde23d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcdd813cee0 0x7fcdd813d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:21.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.398+0000 7fcdde23d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcdd813f640 0x7fcdd813fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:21.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.398+0000 7fcdde23d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fcdd8071d20 0x7fcdd8072170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:21.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.398+0000 7fcdde23d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd80726b0 con 0x7fcdd813cee0 2023-11-28T17:10:21.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.398+0000 7fcdde23d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd813d2c0 con 0x7fcdd8071d20 2023-11-28T17:10:21.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.398+0000 7fcdde23d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd81400e0 con 0x7fcdd813f640 2023-11-28T17:10:21.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.398+0000 7fcdd77fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcdd813cee0 0x7fcdd813d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:21.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.399+0000 7fcdd77fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcdd813cee0 0x7fcdd813d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57522/0 (socket says 172.21.15.116:57522) 2023-11-28T17:10:21.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.399+0000 7fcdd77fe640 1 -- 172.21.15.116:0/2893538914 learned_addr learned my addr 172.21.15.116:0/2893538914 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:21.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.399+0000 7fcdd7fff640 1 --2- 172.21.15.116:0/2893538914 >> v2:172.21.15.156:3300/0 conn(0x7fcdd8071d20 0x7fcdd8072170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:21.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.399+0000 7fcdd6ffd640 1 --2- 172.21.15.116:0/2893538914 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcdd813f640 0x7fcdd813fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:21.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.399+0000 7fcdd77fe640 1 -- 172.21.15.116:0/2893538914 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcdd813f640 msgr2=0x7fcdd813fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:21.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.399+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/2893538914 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcdd813f640 0x7fcdd813fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:21.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.399+0000 7fcdd77fe640 1 -- 172.21.15.116:0/2893538914 >> v2:172.21.15.156:3300/0 conn(0x7fcdd8071d20 msgr2=0x7fcdd8072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.399+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/2893538914 >> v2:172.21.15.156:3300/0 conn(0x7fcdd8071d20 0x7fcdd8072170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.399+0000 7fcdd77fe640 1 -- 172.21.15.116:0/2893538914 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd81403c0 con 0x7fcdd813cee0 2023-11-28T17:10:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.399+0000 7fcdd6ffd640 1 --2- 172.21.15.116:0/2893538914 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcdd813f640 0x7fcdd813fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.399+0000 7fcdd7fff640 1 --2- 172.21.15.116:0/2893538914 >> v2:172.21.15.156:3300/0 conn(0x7fcdd8071d20 0x7fcdd8072170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:21.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.400+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/2893538914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd813cee0 0x7fcdd813d600 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80cc910 tx=0x7fcdc802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f59ce0ba4672c49 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:21.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.400+0000 7fcdd67fc640 1 -- 172.21.15.116:0/2893538914 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8031750 con 0x7fcdd813cee0 2023-11-28T17:10:21.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.400+0000 7fcdd67fc640 1 -- 172.21.15.116:0/2893538914 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc80318b0 con 0x7fcdd813cee0 2023-11-28T17:10:21.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.400+0000 7fcdd67fc640 1 -- 172.21.15.116:0/2893538914 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8031c40 con 0x7fcdd813cee0 2023-11-28T17:10:21.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.401+0000 7fcdde23d640 1 -- 172.21.15.116:0/2893538914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd813cee0 msgr2=0x7fcdd813d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:21.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.401+0000 7fcdde23d640 1 --2- 172.21.15.116:0/2893538914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd813cee0 0x7fcdd813d600 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80cc910 tx=0x7fcdc802b7f0 comp rx=0 tx=0).stop 2023-11-28T17:10:21.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.401+0000 7fcdde23d640 1 -- 172.21.15.116:0/2893538914 shutdown_connections 2023-11-28T17:10:21.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.401+0000 7fcdde23d640 1 --2- 172.21.15.116:0/2893538914 >> v2:172.21.15.156:3300/0 conn(0x7fcdd8071d20 0x7fcdd8072170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:21.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.401+0000 7fcdde23d640 1 --2- 172.21.15.116:0/2893538914 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcdd813f640 0x7fcdd813fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:21.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.401+0000 7fcdde23d640 1 --2- 172.21.15.116:0/2893538914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd813cee0 0x7fcdd813d600 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:21.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.401+0000 7fcdde23d640 1 -- 172.21.15.116:0/2893538914 >> 172.21.15.116:0/2893538914 conn(0x7fcdd80b0cf0 msgr2=0x7fcdd80b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:21.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.401+0000 7fcdde23d640 1 -- 172.21.15.116:0/2893538914 shutdown_connections 2023-11-28T17:10:21.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.401+0000 7fcdde23d640 1 -- 172.21.15.116:0/2893538914 wait complete. 2023-11-28T17:10:21.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.403+0000 7fcdde23d640 1 Processor -- start 2023-11-28T17:10:21.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.403+0000 7fcdde23d640 1 -- start start 2023-11-28T17:10:21.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.404+0000 7fcdde23d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd8071d20 0x7fcdd8166310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:21.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.404+0000 7fcdde23d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdd813cee0 0x7fcdd8142d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:21.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.404+0000 7fcdde23d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcdd813f640 0x7fcdd81432b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:21.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.404+0000 7fcdde23d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80b0380 con 0x7fcdd8071d20 2023-11-28T17:10:21.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.404+0000 7fcdde23d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd813e040 con 0x7fcdd813cee0 2023-11-28T17:10:21.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.404+0000 7fcdde23d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd813dd60 con 0x7fcdd813f640 2023-11-28T17:10:21.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.404+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd8071d20 0x7fcdd8166310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.404+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd8071d20 0x7fcdd8166310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57534/0 (socket says 172.21.15.116:57534) 2023-11-28T17:10:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.404+0000 7fcdd77fe640 1 -- 172.21.15.116:0/182918363 learned_addr learned my addr 172.21.15.116:0/182918363 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.405+0000 7fcdd77fe640 1 -- 172.21.15.116:0/182918363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcdd813f640 msgr2=0x7fcdd81432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.405+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/182918363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcdd813f640 0x7fcdd81432b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.405+0000 7fcdd77fe640 1 -- 172.21.15.116:0/182918363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdd813cee0 msgr2=0x7fcdd8142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.405+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/182918363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdd813cee0 0x7fcdd8142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.405+0000 7fcdd77fe640 1 -- 172.21.15.116:0/182918363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc80094d0 con 0x7fcdd8071d20 2023-11-28T17:10:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.405+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/182918363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd8071d20 0x7fcdd8166310 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8000c00 tx=0x7fcdc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.405+0000 7fcdbffff640 1 -- 172.21.15.116:0/182918363 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8033c80 con 0x7fcdd8071d20 2023-11-28T17:10:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.405+0000 7fcdbffff640 1 -- 172.21.15.116:0/182918363 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc8033e00 con 0x7fcdd8071d20 2023-11-28T17:10:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.405+0000 7fcdbffff640 1 -- 172.21.15.116:0/182918363 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc803c640 con 0x7fcdd8071d20 2023-11-28T17:10:21.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.405+0000 7fcdde23d640 1 -- 172.21.15.116:0/182918363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd80b00a0 con 0x7fcdd8071d20 2023-11-28T17:10:21.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.405+0000 7fcdde23d640 1 -- 172.21.15.116:0/182918363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd8166850 con 0x7fcdd8071d20 2023-11-28T17:10:21.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.406+0000 7fcdde23d640 1 -- 172.21.15.116:0/182918363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd8136ee0 con 0x7fcdd8071d20 2023-11-28T17:10:21.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.407+0000 7fcdbffff640 1 -- 172.21.15.116:0/182918363 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fcdc803a040 con 0x7fcdd8071d20 2023-11-28T17:10:21.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.409+0000 7fcdbffff640 1 --2- 172.21.15.116:0/182918363 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcdc406b2a0 0x7fcdc406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:21.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.409+0000 7fcdbffff640 1 -- 172.21.15.116:0/182918363 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcdc80ab970 con 0x7fcdd8071d20 2023-11-28T17:10:21.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.409+0000 7fcdd6ffd640 1 --2- 172.21.15.116:0/182918363 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcdc406b2a0 0x7fcdc406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:21.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.410+0000 7fcdd6ffd640 1 --2- 172.21.15.116:0/182918363 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcdc406b2a0 0x7fcdc406d760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcdd81f6150 tx=0x7fcdcc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:21.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.414+0000 7fcdbffff640 1 -- 172.21.15.116:0/182918363 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcdd8136ee0 con 0x7fcdd8071d20 2023-11-28T17:10:21.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:21.644+0000 7fcdde23d640 1 -- 172.21.15.116:0/182918363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fcdd8136ee0 con 0x7fcdd8071d20 2023-11-28T17:10:22.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.322+0000 7fcdbffff640 1 -- 172.21.15.116:0/182918363 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) v1 ==== 118+0+0 (secure 0 0 0) 0x7fcdd8136ee0 con 0x7fcdd8071d20 2023-11-28T17:10:22.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.325+0000 7fcdde23d640 1 -- 172.21.15.116:0/182918363 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcdc406b2a0 msgr2=0x7fcdc406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:22.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.325+0000 7fcdde23d640 1 --2- 172.21.15.116:0/182918363 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcdc406b2a0 0x7fcdc406d760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcdd81f6150 tx=0x7fcdcc008040 comp rx=0 tx=0).stop 2023-11-28T17:10:22.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.325+0000 7fcdde23d640 1 -- 172.21.15.116:0/182918363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd8071d20 msgr2=0x7fcdd8166310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:22.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.325+0000 7fcdde23d640 1 --2- 172.21.15.116:0/182918363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd8071d20 0x7fcdd8166310 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8000c00 tx=0x7fcdc80046c0 comp rx=0 tx=0).stop 2023-11-28T17:10:22.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.325+0000 7fcdde23d640 1 -- 172.21.15.116:0/182918363 shutdown_connections 2023-11-28T17:10:22.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.325+0000 7fcdde23d640 1 --2- 172.21.15.116:0/182918363 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcdc406b2a0 0x7fcdc406d760 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:22.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.325+0000 7fcdde23d640 1 --2- 172.21.15.116:0/182918363 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcdd813f640 0x7fcdd81432b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:22.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.325+0000 7fcdde23d640 1 --2- 172.21.15.116:0/182918363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdd813cee0 0x7fcdd8142d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:22.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.325+0000 7fcdde23d640 1 --2- 172.21.15.116:0/182918363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd8071d20 0x7fcdd8166310 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:22.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.325+0000 7fcdde23d640 1 -- 172.21.15.116:0/182918363 >> 172.21.15.116:0/182918363 conn(0x7fcdd80b0cf0 msgr2=0x7fcdd80ce250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:22.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.326+0000 7fcdde23d640 1 -- 172.21.15.116:0/182918363 shutdown_connections 2023-11-28T17:10:22.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.326+0000 7fcdde23d640 1 -- 172.21.15.116:0/182918363 wait complete. 2023-11-28T17:10:22.349 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T17:10:22.349 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T17:10:22.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.560+0000 7fe7b5da9640 1 Processor -- start 2023-11-28T17:10:22.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.560+0000 7fe7b5da9640 1 -- start start 2023-11-28T17:10:22.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7b5da9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7b015b0c0 0x7fe7b015ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:22.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7b5da9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe7b00ce940 0x7fe7b00cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:22.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7b5da9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b00cd320 con 0x7fe7b0071b40 2023-11-28T17:10:22.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7b5da9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b00ced80 con 0x7fe7b00ce940 2023-11-28T17:10:22.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7b5da9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b00cbee0 con 0x7fe7b015b0c0 2023-11-28T17:10:22.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7a6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe7b00ce940 0x7fe7b00cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:22.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7affff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe7b0071b40 0x7fe7b015b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49058/0 (socket says 172.21.15.116:49058) 2023-11-28T17:10:22.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7affff640 1 -- 172.21.15.116:0/1617430265 learned_addr learned my addr 172.21.15.116:0/1617430265 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:22.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7af7fe640 1 --2- 172.21.15.116:0/1617430265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7b015b0c0 0x7fe7b015ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:22.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7aeffd640 1 -- 172.21.15.116:0/1617430265 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681690462 0 0) 0x7fe7b00cd320 con 0x7fe7b0071b40 2023-11-28T17:10:22.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7aeffd640 1 -- 172.21.15.116:0/1617430265 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe79c003520 con 0x7fe7b0071b40 2023-11-28T17:10:22.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7a6ffd640 1 -- 172.21.15.116:0/1617430265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7b015b0c0 msgr2=0x7fe7b015ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:22.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7a6ffd640 1 --2- 172.21.15.116:0/1617430265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7b015b0c0 0x7fe7b015ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:22.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7a6ffd640 1 -- 172.21.15.116:0/1617430265 >> v1:172.21.15.116:6789/0 conn(0x7fe7b0071b40 legacy=0x7fe7b015b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:22.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.562+0000 7fe7a6ffd640 1 -- 172.21.15.116:0/1617430265 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b00cc710 con 0x7fe7b00ce940 2023-11-28T17:10:22.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.563+0000 7fe7a6ffd640 1 --2- 172.21.15.116:0/1617430265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7b00ce940 0x7fe7b00cb9a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe798009d90 tx=0x7fe79802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff7d4c06f8566f5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:22.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.563+0000 7fe7aeffd640 1 -- 172.21.15.116:0/1617430265 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79802cbd0 con 0x7fe7b00ce940 2023-11-28T17:10:22.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.563+0000 7fe7aeffd640 1 -- 172.21.15.116:0/1617430265 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79802cd30 con 0x7fe7b00ce940 2023-11-28T17:10:22.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.563+0000 7fe7aeffd640 1 -- 172.21.15.116:0/1617430265 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79802cea0 con 0x7fe7b00ce940 2023-11-28T17:10:22.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.564+0000 7fe7b5da9640 1 -- 172.21.15.116:0/1617430265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7b00ce940 msgr2=0x7fe7b00cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:22.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.564+0000 7fe7b5da9640 1 --2- 172.21.15.116:0/1617430265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7b00ce940 0x7fe7b00cb9a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe798009d90 tx=0x7fe79802be20 comp rx=0 tx=0).stop 2023-11-28T17:10:22.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.564+0000 7fe7b5da9640 1 -- 172.21.15.116:0/1617430265 shutdown_connections 2023-11-28T17:10:22.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.564+0000 7fe7b5da9640 1 --2- 172.21.15.116:0/1617430265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7b00ce940 0x7fe7b00cb9a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:22.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.564+0000 7fe7b5da9640 1 --2- 172.21.15.116:0/1617430265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7b015b0c0 0x7fe7b015ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:22.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.564+0000 7fe7b5da9640 1 -- 172.21.15.116:0/1617430265 >> 172.21.15.116:0/1617430265 conn(0x7fe7b00b0cf0 msgr2=0x7fe7b00b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:22.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.564+0000 7fe7b5da9640 1 -- 172.21.15.116:0/1617430265 shutdown_connections 2023-11-28T17:10:22.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.564+0000 7fe7b5da9640 1 -- 172.21.15.116:0/1617430265 wait complete. 2023-11-28T17:10:22.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.566+0000 7fe7b5da9640 1 Processor -- start 2023-11-28T17:10:22.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.566+0000 7fe7b5da9640 1 -- start start 2023-11-28T17:10:22.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.567+0000 7fe7b5da9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7b0071b40 0x7fe7b00a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:22.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.567+0000 7fe7b5da9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7b00ce940 0x7fe7b0154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:22.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.567+0000 7fe7b5da9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7b015b0c0 0x7fe7b015c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:22.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.567+0000 7fe7b5da9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b00a8cb0 con 0x7fe7b015b0c0 2023-11-28T17:10:22.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.567+0000 7fe7b5da9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b00a8f90 con 0x7fe7b00ce940 2023-11-28T17:10:22.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.567+0000 7fe7b5da9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b00a9270 con 0x7fe7b0071b40 2023-11-28T17:10:22.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.567+0000 7fe7affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7b015b0c0 0x7fe7b015c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:22.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.567+0000 7fe7affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7b015b0c0 0x7fe7b015c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57550/0 (socket says 172.21.15.116:57550) 2023-11-28T17:10:22.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.567+0000 7fe7affff640 1 -- 172.21.15.116:0/2152019856 learned_addr learned my addr 172.21.15.116:0/2152019856 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:22.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.568+0000 7fe7af7fe640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7b0071b40 0x7fe7b00a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:22.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.568+0000 7fe7a6ffd640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7b00ce940 0x7fe7b0154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:22.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.568+0000 7fe7affff640 1 -- 172.21.15.116:0/2152019856 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7b0071b40 msgr2=0x7fe7b00a8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:22.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.568+0000 7fe7affff640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7b0071b40 0x7fe7b00a8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:22.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.568+0000 7fe7affff640 1 -- 172.21.15.116:0/2152019856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7b00ce940 msgr2=0x7fe7b0154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:22.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.568+0000 7fe7affff640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7b00ce940 0x7fe7b0154050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:22.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.568+0000 7fe7affff640 1 -- 172.21.15.116:0/2152019856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe798009a70 con 0x7fe7b015b0c0 2023-11-28T17:10:22.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.568+0000 7fe7af7fe640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7b0071b40 0x7fe7b00a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:22.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.568+0000 7fe7affff640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7b015b0c0 0x7fe7b015c850 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe7a800c430 tx=0x7fe7a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:22.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.568+0000 7fe7acff9640 1 -- 172.21.15.116:0/2152019856 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8005ce0 con 0x7fe7b015b0c0 2023-11-28T17:10:22.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.568+0000 7fe7acff9640 1 -- 172.21.15.116:0/2152019856 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a8005e40 con 0x7fe7b015b0c0 2023-11-28T17:10:22.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.568+0000 7fe7acff9640 1 -- 172.21.15.116:0/2152019856 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8007310 con 0x7fe7b015b0c0 2023-11-28T17:10:22.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.568+0000 7fe7b5da9640 1 -- 172.21.15.116:0/2152019856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b00aba60 con 0x7fe7b015b0c0 2023-11-28T17:10:22.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.569+0000 7fe7b5da9640 1 -- 172.21.15.116:0/2152019856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b0154590 con 0x7fe7b015b0c0 2023-11-28T17:10:22.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.570+0000 7fe7b5da9640 1 -- 172.21.15.116:0/2152019856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b00ab8c0 con 0x7fe7b015b0c0 2023-11-28T17:10:22.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.571+0000 7fe7acff9640 1 -- 172.21.15.116:0/2152019856 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe7a8007470 con 0x7fe7b015b0c0 2023-11-28T17:10:22.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.571+0000 7fe7acff9640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe79006b2a0 0x7fe79006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:22.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.572+0000 7fe7acff9640 1 -- 172.21.15.116:0/2152019856 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7a8089270 con 0x7fe7b015b0c0 2023-11-28T17:10:22.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.572+0000 7fe7af7fe640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe79006b2a0 0x7fe79006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:22.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.573+0000 7fe7af7fe640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe79006b2a0 0x7fe79006d760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0006fd0 tx=0x7fe7a0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:22.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.576+0000 7fe7acff9640 1 -- 172.21.15.116:0/2152019856 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7b00ab8c0 con 0x7fe7b015b0c0 2023-11-28T17:10:22.674 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:10:22.683 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:10:22.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:22.806+0000 7fe7b5da9640 1 -- 172.21.15.116:0/2152019856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe7b0073c80 con 0x7fe7b015b0c0 2023-11-28T17:10:23.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.344+0000 7fe7acff9640 1 -- 172.21.15.116:0/2152019856 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe7b0073c80 con 0x7fe7b015b0c0 2023-11-28T17:10:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.348+0000 7fe7b5da9640 1 -- 172.21.15.116:0/2152019856 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe79006b2a0 msgr2=0x7fe79006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:23.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.348+0000 7fe7b5da9640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe79006b2a0 0x7fe79006d760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0006fd0 tx=0x7fe7a0008040 comp rx=0 tx=0).stop 2023-11-28T17:10:23.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.348+0000 7fe7b5da9640 1 -- 172.21.15.116:0/2152019856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7b015b0c0 msgr2=0x7fe7b015c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:23.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.348+0000 7fe7b5da9640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7b015b0c0 0x7fe7b015c850 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe7a800c430 tx=0x7fe7a800c8f0 comp rx=0 tx=0).stop 2023-11-28T17:10:23.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.349+0000 7fe7b5da9640 1 -- 172.21.15.116:0/2152019856 shutdown_connections 2023-11-28T17:10:23.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.349+0000 7fe7b5da9640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe79006b2a0 0x7fe79006d760 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:23.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.349+0000 7fe7b5da9640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7b015b0c0 0x7fe7b015c850 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:23.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.349+0000 7fe7b5da9640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe7b00ce940 0x7fe7b0154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:23.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.349+0000 7fe7b5da9640 1 --2- 172.21.15.116:0/2152019856 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe7b0071b40 0x7fe7b00a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:23.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.349+0000 7fe7b5da9640 1 -- 172.21.15.116:0/2152019856 >> 172.21.15.116:0/2152019856 conn(0x7fe7b00b0cf0 msgr2=0x7fe7b00aeb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:23.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.349+0000 7fe7b5da9640 1 -- 172.21.15.116:0/2152019856 shutdown_connections 2023-11-28T17:10:23.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.349+0000 7fe7b5da9640 1 -- 172.21.15.116:0/2152019856 wait complete. 2023-11-28T17:10:23.373 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T17:10:23.374 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:10:23.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.633+0000 7ff6e565c640 1 Processor -- start 2023-11-28T17:10:23.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.633+0000 7ff6e565c640 1 -- start start 2023-11-28T17:10:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.635+0000 7ff6e565c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff6e015f7b0 0x7ff6e0071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.635+0000 7ff6e565c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e015fc10 con 0x7ff6e00721b0 2023-11-28T17:10:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.635+0000 7ff6e565c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e0070bd0 con 0x7ff6e0144430 2023-11-28T17:10:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.635+0000 7ff6e565c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e0144810 con 0x7ff6e015f7b0 2023-11-28T17:10:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.636+0000 7ff6d7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff6e015f7b0 0x7ff6e0071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.636+0000 7ff6df7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff6e00721b0 0x7ff6e006c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49072/0 (socket says 172.21.15.116:49072) 2023-11-28T17:10:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.636+0000 7ff6df7fe640 1 -- 172.21.15.116:0/2914451407 learned_addr learned my addr 172.21.15.116:0/2914451407 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.636+0000 7ff6d7fff640 1 -- 172.21.15.116:0/2914451407 >> v1:172.21.15.156:6789/0 conn(0x7ff6e0144430 legacy=0x7ff6e00703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.636+0000 7ff6d7fff640 1 -- 172.21.15.116:0/2914451407 >> v1:172.21.15.116:6789/0 conn(0x7ff6e00721b0 legacy=0x7ff6e006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.636+0000 7ff6d7fff640 1 -- 172.21.15.116:0/2914451407 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e0070eb0 con 0x7ff6e015f7b0 2023-11-28T17:10:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.636+0000 7ff6de7fc640 1 -- 172.21.15.116:0/2914451407 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234242526 0 0) 0x7ff6e015fc10 con 0x7ff6e00721b0 2023-11-28T17:10:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.637+0000 7ff6d7fff640 1 --2- 172.21.15.116:0/2914451407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e015f7b0 0x7ff6e0071c70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff6c80097a0 tx=0x7ff6c802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7ca34c179ace52 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.637+0000 7ff6de7fc640 1 -- 172.21.15.116:0/2914451407 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8030610 con 0x7ff6e015f7b0 2023-11-28T17:10:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.638+0000 7ff6de7fc640 1 -- 172.21.15.116:0/2914451407 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c8030770 con 0x7ff6e015f7b0 2023-11-28T17:10:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.638+0000 7ff6de7fc640 1 -- 172.21.15.116:0/2914451407 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8030b00 con 0x7ff6e015f7b0 2023-11-28T17:10:23.641 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.638+0000 7ff6e565c640 1 -- 172.21.15.116:0/2914451407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e015f7b0 msgr2=0x7ff6e0071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:23.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.638+0000 7ff6e565c640 1 --2- 172.21.15.116:0/2914451407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e015f7b0 0x7ff6e0071c70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff6c80097a0 tx=0x7ff6c802b960 comp rx=0 tx=0).stop 2023-11-28T17:10:23.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.638+0000 7ff6e565c640 1 -- 172.21.15.116:0/2914451407 shutdown_connections 2023-11-28T17:10:23.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.638+0000 7ff6e565c640 1 --2- 172.21.15.116:0/2914451407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e015f7b0 0x7ff6e0071c70 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:23.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.638+0000 7ff6e565c640 1 -- 172.21.15.116:0/2914451407 >> 172.21.15.116:0/2914451407 conn(0x7ff6e006a4c0 msgr2=0x7ff6e006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:23.642 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.638+0000 7ff6e565c640 1 -- 172.21.15.116:0/2914451407 shutdown_connections 2023-11-28T17:10:23.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.638+0000 7ff6e565c640 1 -- 172.21.15.116:0/2914451407 wait complete. 2023-11-28T17:10:23.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.640+0000 7ff6e565c640 1 Processor -- start 2023-11-28T17:10:23.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.640+0000 7ff6e565c640 1 -- start start 2023-11-28T17:10:23.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.641+0000 7ff6e565c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6e00721b0 0x7ff6e015e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:23.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.641+0000 7ff6e565c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e0144430 0x7ff6e0136610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:23.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.641+0000 7ff6e565c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6e015f7b0 0x7ff6e015cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:23.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.641+0000 7ff6e565c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e015e120 con 0x7ff6e0144430 2023-11-28T17:10:23.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.641+0000 7ff6e565c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e015d510 con 0x7ff6e00721b0 2023-11-28T17:10:23.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.641+0000 7ff6e565c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e015d7f0 con 0x7ff6e015f7b0 2023-11-28T17:10:23.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.641+0000 7ff6d7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e0144430 0x7ff6e0136610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:23.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.642+0000 7ff6d7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e0144430 0x7ff6e0136610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57562/0 (socket says 172.21.15.116:57562) 2023-11-28T17:10:23.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.642+0000 7ff6d7fff640 1 -- 172.21.15.116:0/2354381615 learned_addr learned my addr 172.21.15.116:0/2354381615 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:23.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.642+0000 7ff6df7fe640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6e015f7b0 0x7ff6e015cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:23.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.642+0000 7ff6deffd640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6e00721b0 0x7ff6e015e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:23.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.642+0000 7ff6d7fff640 1 -- 172.21.15.116:0/2354381615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6e015f7b0 msgr2=0x7ff6e015cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:23.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.642+0000 7ff6d7fff640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6e015f7b0 0x7ff6e015cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:23.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.642+0000 7ff6d7fff640 1 -- 172.21.15.116:0/2354381615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6e00721b0 msgr2=0x7ff6e015e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:23.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.642+0000 7ff6d7fff640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6e00721b0 0x7ff6e015e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:23.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.642+0000 7ff6d7fff640 1 -- 172.21.15.116:0/2354381615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c8009400 con 0x7ff6e0144430 2023-11-28T17:10:23.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.642+0000 7ff6deffd640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6e00721b0 0x7ff6e015e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:23.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.642+0000 7ff6d7fff640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e0144430 0x7ff6e0136610 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff6c80098c0 tx=0x7ff6c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:23.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.643+0000 7ff6d77fe640 1 -- 172.21.15.116:0/2354381615 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c803f070 con 0x7ff6e0144430 2023-11-28T17:10:23.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.643+0000 7ff6d77fe640 1 -- 172.21.15.116:0/2354381615 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c803aa20 con 0x7ff6e0144430 2023-11-28T17:10:23.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.643+0000 7ff6e565c640 1 -- 172.21.15.116:0/2354381615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e015dad0 con 0x7ff6e0144430 2023-11-28T17:10:23.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.643+0000 7ff6d77fe640 1 -- 172.21.15.116:0/2354381615 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c803add0 con 0x7ff6e0144430 2023-11-28T17:10:23.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.643+0000 7ff6e565c640 1 -- 172.21.15.116:0/2354381615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e0136b50 con 0x7ff6e0144430 2023-11-28T17:10:23.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.644+0000 7ff6e565c640 1 -- 172.21.15.116:0/2354381615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e015e2d0 con 0x7ff6e0144430 2023-11-28T17:10:23.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.645+0000 7ff6d77fe640 1 -- 172.21.15.116:0/2354381615 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff6c80433f0 con 0x7ff6e0144430 2023-11-28T17:10:23.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.646+0000 7ff6d77fe640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff6c006b320 0x7ff6c006d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:23.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.646+0000 7ff6d77fe640 1 -- 172.21.15.116:0/2354381615 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6c80abba0 con 0x7ff6e0144430 2023-11-28T17:10:23.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.646+0000 7ff6deffd640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff6c006b320 0x7ff6c006d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:23.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.647+0000 7ff6deffd640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff6c006b320 0x7ff6c006d7e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0000f60 tx=0x7ff6d0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:23.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.650+0000 7ff6d77fe640 1 -- 172.21.15.116:0/2354381615 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff6e015e2d0 con 0x7ff6e0144430 2023-11-28T17:10:23.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.874+0000 7ff6e565c640 1 -- 172.21.15.116:0/2354381615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6e0066fb0 con 0x7ff6e0144430 2023-11-28T17:10:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.875+0000 7ff6d77fe640 1 -- 172.21.15.116:0/2354381615 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4919 (secure 0 0 0) 0x7ff6c8049020 con 0x7ff6e0144430 2023-11-28T17:10:23.876 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:23.877 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/491158514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":491158514},{"type":"v1","addr":"172.21.15.156:6834","nonce":491158514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3927645596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3927645596},{"type":"v1","addr":"172.21.15.116:6835","nonce":3927645596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4738,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/141796020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":141796020},{"type":"v1","addr":"172.21.15.156:6835","nonce":141796020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:10:19.725913+0000","modified":"2023-11-28T17:10:22.808343+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6837/3177126349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3177126349},{"type":"v1","addr":"172.21.15.116:6837","nonce":3177126349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T17:10:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 15 2023-11-28T17:10:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.877+0000 7ff6e565c640 1 -- 172.21.15.116:0/2354381615 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff6c006b320 msgr2=0x7ff6c006d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.877+0000 7ff6e565c640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff6c006b320 0x7ff6c006d7e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0000f60 tx=0x7ff6d0009040 comp rx=0 tx=0).stop 2023-11-28T17:10:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.877+0000 7ff6e565c640 1 -- 172.21.15.116:0/2354381615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e0144430 msgr2=0x7ff6e0136610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.877+0000 7ff6e565c640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e0144430 0x7ff6e0136610 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff6c80098c0 tx=0x7ff6c80046c0 comp rx=0 tx=0).stop 2023-11-28T17:10:23.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.877+0000 7ff6e565c640 1 -- 172.21.15.116:0/2354381615 shutdown_connections 2023-11-28T17:10:23.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.877+0000 7ff6e565c640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff6c006b320 0x7ff6c006d7e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:23.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.877+0000 7ff6e565c640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6e015f7b0 0x7ff6e015cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:23.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.877+0000 7ff6e565c640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6e0144430 0x7ff6e0136610 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:23.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.877+0000 7ff6e565c640 1 --2- 172.21.15.116:0/2354381615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6e00721b0 0x7ff6e015e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:23.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.878+0000 7ff6e565c640 1 -- 172.21.15.116:0/2354381615 >> 172.21.15.116:0/2354381615 conn(0x7ff6e006a4c0 msgr2=0x7ff6e006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:23.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.878+0000 7ff6e565c640 1 -- 172.21.15.116:0/2354381615 shutdown_connections 2023-11-28T17:10:23.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:23.878+0000 7ff6e565c640 1 -- 172.21.15.116:0/2354381615 wait complete. 2023-11-28T17:10:23.896 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:10:24.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.105+0000 7f0394ebb640 1 Processor -- start 2023-11-28T17:10:24.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.105+0000 7f0394ebb640 1 -- start start 2023-11-28T17:10:24.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.106+0000 7f0394ebb640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f03900b4620 0x7f03900cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:24.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.107+0000 7f0394ebb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03900d0db0 0x7f03900cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:24.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.107+0000 7f0394ebb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03900ce670 con 0x7f03900ce140 2023-11-28T17:10:24.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.107+0000 7f0394ebb640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03900ce950 con 0x7f03900b4620 2023-11-28T17:10:24.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.107+0000 7f0394ebb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03900cec30 con 0x7f03900d0db0 2023-11-28T17:10:24.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.107+0000 7f038f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f03900b4620 0x7f03900cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:24.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.107+0000 7f038f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f03900b4620 0x7f03900cdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57578/0 (socket says 172.21.15.116:57578) 2023-11-28T17:10:24.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.107+0000 7f038f7fe640 1 -- 172.21.15.116:0/1559498152 learned_addr learned my addr 172.21.15.116:0/1559498152 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:24.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.107+0000 7f0387fff640 1 --2- 172.21.15.116:0/1559498152 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03900d0db0 0x7f03900cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:24.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.107+0000 7f038f7fe640 1 -- 172.21.15.116:0/1559498152 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03900d0db0 msgr2=0x7f03900cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:24.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.108+0000 7f038f7fe640 1 --2- 172.21.15.116:0/1559498152 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03900d0db0 0x7f03900cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.108+0000 7f038f7fe640 1 -- 172.21.15.116:0/1559498152 >> v1:172.21.15.116:6789/0 conn(0x7f03900ce140 legacy=0x7f039015cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:24.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.108+0000 7f038f7fe640 1 -- 172.21.15.116:0/1559498152 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03900cef10 con 0x7f03900b4620 2023-11-28T17:10:24.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.108+0000 7f0387fff640 1 --2- 172.21.15.116:0/1559498152 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03900d0db0 0x7f03900cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:24.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.109+0000 7f038f7fe640 1 --2- 172.21.15.116:0/1559498152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03900b4620 0x7f03900cdc00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0380009870 tx=0x7f038002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9028dc7a94dd350 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:24.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.109+0000 7f038effd640 1 -- 172.21.15.116:0/1559498152 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0380030840 con 0x7f03900b4620 2023-11-28T17:10:24.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.109+0000 7f038effd640 1 -- 172.21.15.116:0/1559498152 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03800309a0 con 0x7f03900b4620 2023-11-28T17:10:24.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.109+0000 7f038effd640 1 -- 172.21.15.116:0/1559498152 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0380030d30 con 0x7f03900b4620 2023-11-28T17:10:24.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.109+0000 7f0394ebb640 1 -- 172.21.15.116:0/1559498152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03900b4620 msgr2=0x7f03900cdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:24.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.109+0000 7f0394ebb640 1 --2- 172.21.15.116:0/1559498152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03900b4620 0x7f03900cdc00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0380009870 tx=0x7f038002b980 comp rx=0 tx=0).stop 2023-11-28T17:10:24.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.110+0000 7f0394ebb640 1 -- 172.21.15.116:0/1559498152 shutdown_connections 2023-11-28T17:10:24.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.110+0000 7f0394ebb640 1 --2- 172.21.15.116:0/1559498152 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03900d0db0 0x7f03900cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.110+0000 7f0394ebb640 1 --2- 172.21.15.116:0/1559498152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03900b4620 0x7f03900cdc00 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.110+0000 7f0394ebb640 1 -- 172.21.15.116:0/1559498152 >> 172.21.15.116:0/1559498152 conn(0x7f03900d2f40 msgr2=0x7f03900d3350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:24.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.110+0000 7f0394ebb640 1 -- 172.21.15.116:0/1559498152 shutdown_connections 2023-11-28T17:10:24.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.110+0000 7f0394ebb640 1 -- 172.21.15.116:0/1559498152 wait complete. 2023-11-28T17:10:24.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.111+0000 7f0394ebb640 1 Processor -- start 2023-11-28T17:10:24.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.111+0000 7f0394ebb640 1 -- start start 2023-11-28T17:10:24.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.112+0000 7f0394ebb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03900b4620 0x7f039014c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:24.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.112+0000 7f0394ebb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03900ce140 0x7f0390148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:24.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.112+0000 7f0394ebb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03900d0db0 0x7f03900c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:24.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.112+0000 7f0394ebb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0390149430 con 0x7f03900ce140 2023-11-28T17:10:24.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.112+0000 7f0394ebb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03900c9270 con 0x7f03900b4620 2023-11-28T17:10:24.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.112+0000 7f0394ebb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03900c9550 con 0x7f03900d0db0 2023-11-28T17:10:24.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.113+0000 7f0387fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03900ce140 0x7f0390148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:24.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.113+0000 7f0387fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03900ce140 0x7f0390148ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57588/0 (socket says 172.21.15.116:57588) 2023-11-28T17:10:24.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.113+0000 7f0387fff640 1 -- 172.21.15.116:0/1271571133 learned_addr learned my addr 172.21.15.116:0/1271571133 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:24.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.113+0000 7f038ffff640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03900d0db0 0x7f03900c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:24.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.113+0000 7f038f7fe640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03900b4620 0x7f039014c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:24.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.113+0000 7f0387fff640 1 -- 172.21.15.116:0/1271571133 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03900d0db0 msgr2=0x7f03900c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:24.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.113+0000 7f0387fff640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03900d0db0 0x7f03900c89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.113+0000 7f0387fff640 1 -- 172.21.15.116:0/1271571133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03900b4620 msgr2=0x7f039014c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:24.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.113+0000 7f0387fff640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03900b4620 0x7f039014c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.113+0000 7f0387fff640 1 -- 172.21.15.116:0/1271571133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03800094d0 con 0x7f03900ce140 2023-11-28T17:10:24.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.113+0000 7f038ffff640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03900d0db0 0x7f03900c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:24.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.113+0000 7f0387fff640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03900ce140 0x7f0390148ef0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f037800c7c0 tx=0x7f037800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:24.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.114+0000 7f038cff9640 1 -- 172.21.15.116:0/1271571133 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0378007820 con 0x7f03900ce140 2023-11-28T17:10:24.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.114+0000 7f038cff9640 1 -- 172.21.15.116:0/1271571133 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0378007980 con 0x7f03900ce140 2023-11-28T17:10:24.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.114+0000 7f038cff9640 1 -- 172.21.15.116:0/1271571133 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0378007d10 con 0x7f03900ce140 2023-11-28T17:10:24.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.114+0000 7f0394ebb640 1 -- 172.21.15.116:0/1271571133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03900c9830 con 0x7f03900ce140 2023-11-28T17:10:24.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.114+0000 7f0394ebb640 1 -- 172.21.15.116:0/1271571133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03900c8ef0 con 0x7f03900ce140 2023-11-28T17:10:24.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.115+0000 7f0394ebb640 1 -- 172.21.15.116:0/1271571133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f039015e6e0 con 0x7f03900ce140 2023-11-28T17:10:24.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.116+0000 7f038cff9640 1 -- 172.21.15.116:0/1271571133 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f03780113f0 con 0x7f03900ce140 2023-11-28T17:10:24.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.117+0000 7f038cff9640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f037006b2a0 0x7f037006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:24.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.117+0000 7f038cff9640 1 -- 172.21.15.116:0/1271571133 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0378089ce0 con 0x7f03900ce140 2023-11-28T17:10:24.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.117+0000 7f038f7fe640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f037006b2a0 0x7f037006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:24.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.118+0000 7f038f7fe640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f037006b2a0 0x7f037006d760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0380036000 tx=0x7f038003b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:24.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.121+0000 7f038cff9640 1 -- 172.21.15.116:0/1271571133 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f039015e6e0 con 0x7f03900ce140 2023-11-28T17:10:24.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.301+0000 7f0394ebb640 1 -- 172.21.15.116:0/1271571133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f039015e6e0 con 0x7f03900ce140 2023-11-28T17:10:24.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.303+0000 7f038cff9640 1 -- 172.21.15.116:0/1271571133 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16521 (secure 0 0 0) 0x7f039015e6e0 con 0x7f03900ce140 2023-11-28T17:10:24.305 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:24.305 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":34,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:10:21.316243+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:10:17.338444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-28T17:10:18.766338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:10:24.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.306+0000 7f0394ebb640 1 -- 172.21.15.116:0/1271571133 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f037006b2a0 msgr2=0x7f037006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:24.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.306+0000 7f0394ebb640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f037006b2a0 0x7f037006d760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0380036000 tx=0x7f038003b040 comp rx=0 tx=0).stop 2023-11-28T17:10:24.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.306+0000 7f0394ebb640 1 -- 172.21.15.116:0/1271571133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03900ce140 msgr2=0x7f0390148ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:24.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.306+0000 7f0394ebb640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03900ce140 0x7f0390148ef0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f037800c7c0 tx=0x7f037800cc80 comp rx=0 tx=0).stop 2023-11-28T17:10:24.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.306+0000 7f0394ebb640 1 -- 172.21.15.116:0/1271571133 shutdown_connections 2023-11-28T17:10:24.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.306+0000 7f0394ebb640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f037006b2a0 0x7f037006d760 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.306+0000 7f0394ebb640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03900d0db0 0x7f03900c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.306+0000 7f0394ebb640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03900ce140 0x7f0390148ef0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.306+0000 7f0394ebb640 1 --2- 172.21.15.116:0/1271571133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03900b4620 0x7f039014c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.306+0000 7f0394ebb640 1 -- 172.21.15.116:0/1271571133 >> 172.21.15.116:0/1271571133 conn(0x7f03900d2f40 msgr2=0x7f03900afb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:24.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.306+0000 7f0394ebb640 1 -- 172.21.15.116:0/1271571133 shutdown_connections 2023-11-28T17:10:24.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.306+0000 7f0394ebb640 1 -- 172.21.15.116:0/1271571133 wait complete. 2023-11-28T17:10:24.326 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T17:10:24.327 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:10:24.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.535+0000 7fc7df064640 1 Processor -- start 2023-11-28T17:10:24.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.535+0000 7fc7df064640 1 -- start start 2023-11-28T17:10:24.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.537+0000 7fc7df064640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc7d8144430 0x7fc7d815f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:24.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.537+0000 7fc7df064640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d8070bd0 con 0x7fc7d806c250 2023-11-28T17:10:24.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.537+0000 7fc7df064640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d8070eb0 con 0x7fc7d80703f0 2023-11-28T17:10:24.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.537+0000 7fc7df064640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d8144810 con 0x7fc7d8144430 2023-11-28T17:10:24.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.537+0000 7fc7dd601640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc7d806c250 0x7fc7d815b640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49076/0 (socket says 172.21.15.116:49076) 2023-11-28T17:10:24.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.537+0000 7fc7dd601640 1 -- 172.21.15.116:0/1848649515 learned_addr learned my addr 172.21.15.116:0/1848649515 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:24.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.538+0000 7fc7dce00640 1 --2- 172.21.15.116:0/1848649515 >> v2:172.21.15.156:3300/0 conn(0x7fc7d8144430 0x7fc7d815f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:24.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.538+0000 7fc7d77fe640 1 -- 172.21.15.116:0/1848649515 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254182300 0 0) 0x7fc7d8070bd0 con 0x7fc7d806c250 2023-11-28T17:10:24.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.538+0000 7fc7d77fe640 1 -- 172.21.15.116:0/1848649515 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c8003520 con 0x7fc7d806c250 2023-11-28T17:10:24.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.538+0000 7fc7dce00640 1 -- 172.21.15.116:0/1848649515 >> v1:172.21.15.156:6789/0 conn(0x7fc7d80703f0 legacy=0x7fc7d8071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:24.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.538+0000 7fc7dce00640 1 -- 172.21.15.116:0/1848649515 >> v1:172.21.15.116:6789/0 conn(0x7fc7d806c250 legacy=0x7fc7d815b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:24.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.538+0000 7fc7dce00640 1 -- 172.21.15.116:0/1848649515 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d8071190 con 0x7fc7d8144430 2023-11-28T17:10:24.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.539+0000 7fc7dce00640 1 --2- 172.21.15.116:0/1848649515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7d8144430 0x7fc7d815f7b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0097a0 tx=0x7fc7cc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a43abfa7892ab91 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:24.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.540+0000 7fc7d77fe640 1 -- 172.21.15.116:0/1848649515 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc030610 con 0x7fc7d8144430 2023-11-28T17:10:24.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.540+0000 7fc7d77fe640 1 -- 172.21.15.116:0/1848649515 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc030770 con 0x7fc7d8144430 2023-11-28T17:10:24.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.540+0000 7fc7d77fe640 1 -- 172.21.15.116:0/1848649515 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc030b00 con 0x7fc7d8144430 2023-11-28T17:10:24.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.540+0000 7fc7df064640 1 -- 172.21.15.116:0/1848649515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7d8144430 msgr2=0x7fc7d815f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:24.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.540+0000 7fc7df064640 1 --2- 172.21.15.116:0/1848649515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7d8144430 0x7fc7d815f7b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0097a0 tx=0x7fc7cc02b960 comp rx=0 tx=0).stop 2023-11-28T17:10:24.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.541+0000 7fc7df064640 1 -- 172.21.15.116:0/1848649515 shutdown_connections 2023-11-28T17:10:24.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.541+0000 7fc7df064640 1 --2- 172.21.15.116:0/1848649515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7d8144430 0x7fc7d815f7b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.541+0000 7fc7df064640 1 -- 172.21.15.116:0/1848649515 >> 172.21.15.116:0/1848649515 conn(0x7fc7d806a4c0 msgr2=0x7fc7d806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:24.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.541+0000 7fc7df064640 1 -- 172.21.15.116:0/1848649515 shutdown_connections 2023-11-28T17:10:24.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.541+0000 7fc7df064640 1 -- 172.21.15.116:0/1848649515 wait complete. 2023-11-28T17:10:24.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.543+0000 7fc7df064640 1 Processor -- start 2023-11-28T17:10:24.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.543+0000 7fc7df064640 1 -- start start 2023-11-28T17:10:24.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.544+0000 7fc7df064640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d806c250 0x7fc7d81e79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.544+0000 7fc7df064640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7d80703f0 0x7fc7d8132d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.544+0000 7fc7df064640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc7d8144430 0x7fc7d815e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.544+0000 7fc7df064640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d81e8d10 con 0x7fc7d806c250 2023-11-28T17:10:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.544+0000 7fc7df064640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d8133260 con 0x7fc7d80703f0 2023-11-28T17:10:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.544+0000 7fc7df064640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d81e8090 con 0x7fc7d8144430 2023-11-28T17:10:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.545+0000 7fc7dce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d806c250 0x7fc7d81e79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.545+0000 7fc7dce00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d806c250 0x7fc7d81e79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57592/0 (socket says 172.21.15.116:57592) 2023-11-28T17:10:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.545+0000 7fc7dce00640 1 -- 172.21.15.116:0/3402468203 learned_addr learned my addr 172.21.15.116:0/3402468203 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.545+0000 7fc7d7fff640 1 --2- 172.21.15.116:0/3402468203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7d80703f0 0x7fc7d8132d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.545+0000 7fc7dd601640 1 --2- 172.21.15.116:0/3402468203 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc7d8144430 0x7fc7d815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.545+0000 7fc7dce00640 1 -- 172.21.15.116:0/3402468203 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc7d8144430 msgr2=0x7fc7d815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.545+0000 7fc7dce00640 1 --2- 172.21.15.116:0/3402468203 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc7d8144430 0x7fc7d815e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.545+0000 7fc7dce00640 1 -- 172.21.15.116:0/3402468203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7d80703f0 msgr2=0x7fc7d8132d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:24.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.545+0000 7fc7dce00640 1 --2- 172.21.15.116:0/3402468203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7d80703f0 0x7fc7d8132d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.545+0000 7fc7dce00640 1 -- 172.21.15.116:0/3402468203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc009400 con 0x7fc7d806c250 2023-11-28T17:10:24.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.545+0000 7fc7dce00640 1 --2- 172.21.15.116:0/3402468203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d806c250 0x7fc7d81e79d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc02bf70 tx=0x7fc7cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:24.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.546+0000 7fc7d57fa640 1 -- 172.21.15.116:0/3402468203 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc03e040 con 0x7fc7d806c250 2023-11-28T17:10:24.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.546+0000 7fc7d57fa640 1 -- 172.21.15.116:0/3402468203 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc039cb0 con 0x7fc7d806c250 2023-11-28T17:10:24.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.546+0000 7fc7d57fa640 1 -- 172.21.15.116:0/3402468203 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc044630 con 0x7fc7d806c250 2023-11-28T17:10:24.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.546+0000 7fc7df064640 1 -- 172.21.15.116:0/3402468203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d81e8370 con 0x7fc7d806c250 2023-11-28T17:10:24.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.546+0000 7fc7df064640 1 -- 172.21.15.116:0/3402468203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d81e8930 con 0x7fc7d806c250 2023-11-28T17:10:24.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.547+0000 7fc7df064640 1 -- 172.21.15.116:0/3402468203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d81e8e50 con 0x7fc7d806c250 2023-11-28T17:10:24.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.548+0000 7fc7d57fa640 1 -- 172.21.15.116:0/3402468203 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fc7cc043390 con 0x7fc7d806c250 2023-11-28T17:10:24.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.549+0000 7fc7d57fa640 1 --2- 172.21.15.116:0/3402468203 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc7bc06b2a0 0x7fc7bc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:24.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.549+0000 7fc7d57fa640 1 -- 172.21.15.116:0/3402468203 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7cc0abc00 con 0x7fc7d806c250 2023-11-28T17:10:24.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.550+0000 7fc7d7fff640 1 --2- 172.21.15.116:0/3402468203 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc7bc06b2a0 0x7fc7bc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:24.552 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.550+0000 7fc7d7fff640 1 --2- 172.21.15.116:0/3402468203 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc7bc06b2a0 0x7fc7bc06d760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc7d81f6360 tx=0x7fc7c4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:24.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.554+0000 7fc7d57fa640 1 -- 172.21.15.116:0/3402468203 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7d81e8e50 con 0x7fc7d806c250 2023-11-28T17:10:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.728+0000 7fc7df064640 1 -- 172.21.15.116:0/3402468203 --> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc7d8072380 con 0x7fc7bc06b2a0 2023-11-28T17:10:24.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.741+0000 7fc7d57fa640 1 -- 172.21.15.116:0/3402468203 <== mgr.4098 v2:172.21.15.116:6832/1077400178 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281501 (secure 0 0 0) 0x7fc7c4001d10 con 0x7fc7bc06b2a0 2023-11-28T17:10:24.746 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:24.748 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-28T17:10:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.746+0000 7fc7df064640 1 -- 172.21.15.116:0/3402468203 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc7bc06b2a0 msgr2=0x7fc7bc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.746+0000 7fc7df064640 1 --2- 172.21.15.116:0/3402468203 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc7bc06b2a0 0x7fc7bc06d760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc7d81f6360 tx=0x7fc7c4009040 comp rx=0 tx=0).stop 2023-11-28T17:10:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.746+0000 7fc7df064640 1 -- 172.21.15.116:0/3402468203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d806c250 msgr2=0x7fc7d81e79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.746+0000 7fc7df064640 1 --2- 172.21.15.116:0/3402468203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d806c250 0x7fc7d81e79d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc02bf70 tx=0x7fc7cc004730 comp rx=0 tx=0).stop 2023-11-28T17:10:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.746+0000 7fc7df064640 1 -- 172.21.15.116:0/3402468203 shutdown_connections 2023-11-28T17:10:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.746+0000 7fc7df064640 1 --2- 172.21.15.116:0/3402468203 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc7bc06b2a0 0x7fc7bc06d760 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.747+0000 7fc7df064640 1 --2- 172.21.15.116:0/3402468203 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc7d8144430 0x7fc7d815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.747+0000 7fc7df064640 1 --2- 172.21.15.116:0/3402468203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc7d80703f0 0x7fc7d8132d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.747+0000 7fc7df064640 1 --2- 172.21.15.116:0/3402468203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d806c250 0x7fc7d81e79d0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.747+0000 7fc7df064640 1 -- 172.21.15.116:0/3402468203 >> 172.21.15.116:0/3402468203 conn(0x7fc7d806a4c0 msgr2=0x7fc7d80b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.747+0000 7fc7df064640 1 -- 172.21.15.116:0/3402468203 shutdown_connections 2023-11-28T17:10:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:24.747+0000 7fc7df064640 1 -- 172.21.15.116:0/3402468203 wait complete. 2023-11-28T17:10:24.765 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2023-11-28T17:10:23.064571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236680,"kb_used_data":3880,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738040,"statfs":{"total":773094113280,"available":772851752960,"internally_reserved":0,"allocated":3973120,"data_stored":2237240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.806213"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.283066+0000","last_change":"2023-11-28T17:10:19.283066+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.283066+0000","last_undegraded":"2023-11-28T17:10:19.283066+0000","last_fullsized":"2023-11-28T17:10:19.283066+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.281423+0000","last_change":"2023-11-28T17:10:19.281423+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.281423+0000","last_undegraded":"2023-11-28T17:10:19.281423+0000","last_fullsized":"2023-11-28T17:10:19.281423+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.273834+0000","last_change":"2023-11-28T17:10:19.273834+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.273834+0000","last_undegraded":"2023-11-28T17:10:19.273834+0000","last_fullsized":"2023-11-28T17:10:19.273834+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.291453+0000","last_change":"2023-11-28T17:10:19.291453+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.291453+0000","last_undegraded":"2023-11-28T17:10:19.291453+0000","last_fullsized":"2023-11-28T17:10:19.291453+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.279393+0000","last_change":"2023-11-28T17:10:19.279393+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.279393+0000","last_undegraded":"2023-11-28T17:10:19.279393+0000","last_fullsized":"2023-11-28T17:10:19.279393+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.283970+0000","last_change":"2023-11-28T17:10:19.283970+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.283970+0000","last_undegraded":"2023-11-28T17:10:19.283970+0000","last_fullsized":"2023-11-28T17:10:19.283970+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.276093+0000","last_change":"2023-11-28T17:10:19.276093+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.276093+0000","last_undegraded":"2023-11-28T17:10:19.276093+0000","last_fullsized":"2023-11-28T17:10:19.276093+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.276646+0000","last_change":"2023-11-28T17:10:19.276646+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.276646+0000","last_undegraded":"2023-11-28T17:10:19.276646+0000","last_fullsized":"2023-11-28T17:10:19.276646+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.273615+0000","last_change":"2023-11-28T17:10:19.273615+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.273615+0000","last_undegraded":"2023-11-28T17:10:19.273615+0000","last_fullsized":"2023-11-28T17:10:19.273615+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.298663+0000","last_change":"2023-11-28T17:10:19.298663+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.298663+0000","last_undegraded":"2023-11-28T17:10:19.298663+0000","last_fullsized":"2023-11-28T17:10:19.298663+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.283837+0000","last_change":"2023-11-28T17:10:19.283837+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.283837+0000","last_undegraded":"2023-11-28T17:10:19.283837+0000","last_fullsized":"2023-11-28T17:10:19.283837+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.273422+0000","last_change":"2023-11-28T17:10:19.273422+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.273422+0000","last_undegraded":"2023-11-28T17:10:19.273422+0000","last_fullsized":"2023-11-28T17:10:19.273422+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.276667+0000","last_change":"2023-11-28T17:10:19.276667+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.276667+0000","last_undegraded":"2023-11-28T17:10:19.276667+0000","last_fullsized":"2023-11-28T17:10:19.276667+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.284341+0000","last_change":"2023-11-28T17:10:19.284341+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.284341+0000","last_undegraded":"2023-11-28T17:10:19.284341+0000","last_fullsized":"2023-11-28T17:10:19.284341+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.277153+0000","last_change":"2023-11-28T17:10:19.277153+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.277153+0000","last_undegraded":"2023-11-28T17:10:19.277153+0000","last_fullsized":"2023-11-28T17:10:19.277153+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.277106+0000","last_change":"2023-11-28T17:10:19.277106+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.277106+0000","last_undegraded":"2023-11-28T17:10:19.277106+0000","last_fullsized":"2023-11-28T17:10:19.277106+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.284129+0000","last_change":"2023-11-28T17:10:19.284129+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.284129+0000","last_undegraded":"2023-11-28T17:10:19.284129+0000","last_fullsized":"2023-11-28T17:10:19.284129+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.287536+0000","last_change":"2023-11-28T17:10:19.287536+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.287536+0000","last_undegraded":"2023-11-28T17:10:19.287536+0000","last_fullsized":"2023-11-28T17:10:19.287536+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.288754+0000","last_change":"2023-11-28T17:10:19.288754+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.288754+0000","last_undegraded":"2023-11-28T17:10:19.288754+0000","last_fullsized":"2023-11-28T17:10:19.288754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.274408+0000","last_change":"2023-11-28T17:10:19.274408+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.274408+0000","last_undegraded":"2023-11-28T17:10:19.274408+0000","last_fullsized":"2023-11-28T17:10:19.274408+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.278233+0000","last_change":"2023-11-28T17:10:19.278233+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.278233+0000","last_undegraded":"2023-11-28T17:10:19.278233+0000","last_fullsized":"2023-11-28T17:10:19.278233+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.276543+0000","last_change":"2023-11-28T17:10:19.276543+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.276543+0000","last_undegraded":"2023-11-28T17:10:19.276543+0000","last_fullsized":"2023-11-28T17:10:19.276543+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.275670+0000","last_change":"2023-11-28T17:10:19.275670+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.275670+0000","last_undegraded":"2023-11-28T17:10:19.275670+0000","last_fullsized":"2023-11-28T17:10:19.275670+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.276068+0000","last_change":"2023-11-28T17:10:19.276068+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.276068+0000","last_undegraded":"2023-11-28T17:10:19.276068+0000","last_fullsized":"2023-11-28T17:10:19.276068+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.295347+0000","last_change":"2023-11-28T17:10:19.295347+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.295347+0000","last_undegraded":"2023-11-28T17:10:19.295347+0000","last_fullsized":"2023-11-28T17:10:19.295347+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.298097+0000","last_change":"2023-11-28T17:10:19.298097+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.298097+0000","last_undegraded":"2023-11-28T17:10:19.298097+0000","last_fullsized":"2023-11-28T17:10:19.298097+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.283389+0000","last_change":"2023-11-28T17:10:19.283389+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.283389+0000","last_undegraded":"2023-11-28T17:10:19.283389+0000","last_fullsized":"2023-11-28T17:10:19.283389+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.282147+0000","last_change":"2023-11-28T17:10:19.282147+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.282147+0000","last_undegraded":"2023-11-28T17:10:19.282147+0000","last_fullsized":"2023-11-28T17:10:19.282147+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.278607+0000","last_change":"2023-11-28T17:10:19.278607+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.278607+0000","last_undegraded":"2023-11-28T17:10:19.278607+0000","last_fullsized":"2023-11-28T17:10:19.278607+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.279464+0000","last_change":"2023-11-28T17:10:19.279464+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.279464+0000","last_undegraded":"2023-11-28T17:10:19.279464+0000","last_fullsized":"2023-11-28T17:10:19.279464+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.275439+0000","last_change":"2023-11-28T17:10:19.275439+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.275439+0000","last_undegraded":"2023-11-28T17:10:19.275439+0000","last_fullsized":"2023-11-28T17:10:19.275439+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.281475+0000","last_change":"2023-11-28T17:10:19.281475+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.281475+0000","last_undegraded":"2023-11-28T17:10:19.281475+0000","last_fullsized":"2023-11-28T17:10:19.281475+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.273779+0000","last_change":"2023-11-28T17:10:19.273779+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.273779+0000","last_undegraded":"2023-11-28T17:10:19.273779+0000","last_fullsized":"2023-11-28T17:10:19.273779+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.298464+0000","last_change":"2023-11-28T17:10:19.295420+0000","last_active":"2023-11-28T17:10:19.298464+0000","last_peered":"2023-11-28T17:10:19.298464+0000","last_clean":"2023-11-28T17:10:19.298464+0000","last_became_active":"2023-11-28T17:10:19.294612+0000","last_became_peered":"2023-11-28T17:10:19.294612+0000","last_unstale":"2023-11-28T17:10:19.298464+0000","last_undegraded":"2023-11-28T17:10:19.298464+0000","last_fullsized":"2023-11-28T17:10:19.298464+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:56:23.402303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.282533+0000","last_change":"2023-11-28T17:10:19.282533+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.282533+0000","last_undegraded":"2023-11-28T17:10:19.282533+0000","last_fullsized":"2023-11-28T17:10:19.282533+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.307742+0000","last_change":"2023-11-28T17:10:19.306768+0000","last_active":"2023-11-28T17:10:19.307742+0000","last_peered":"2023-11-28T17:10:19.307742+0000","last_clean":"2023-11-28T17:10:19.307742+0000","last_became_active":"2023-11-28T17:10:19.306492+0000","last_became_peered":"2023-11-28T17:10:19.306492+0000","last_unstale":"2023-11-28T17:10:19.307742+0000","last_undegraded":"2023-11-28T17:10:19.307742+0000","last_fullsized":"2023-11-28T17:10:19.307742+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:59:44.868450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.280570+0000","last_change":"2023-11-28T17:10:19.280570+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.280570+0000","last_undegraded":"2023-11-28T17:10:19.280570+0000","last_fullsized":"2023-11-28T17:10:19.280570+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.301614+0000","last_change":"2023-11-28T17:10:19.301614+0000","last_active":"2023-11-28T17:10:19.301614+0000","last_peered":"2023-11-28T17:10:19.301614+0000","last_clean":"2023-11-28T17:10:19.301614+0000","last_became_active":"2023-11-28T17:10:19.301307+0000","last_became_peered":"2023-11-28T17:10:19.301307+0000","last_unstale":"2023-11-28T17:10:19.301614+0000","last_undegraded":"2023-11-28T17:10:19.301614+0000","last_fullsized":"2023-11-28T17:10:19.301614+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:29:07.449087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.292118+0000","last_change":"2023-11-28T17:10:19.292118+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.292118+0000","last_undegraded":"2023-11-28T17:10:19.292118+0000","last_fullsized":"2023-11-28T17:10:19.292118+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.305158+0000","last_change":"2023-11-28T17:10:19.301144+0000","last_active":"2023-11-28T17:10:19.305158+0000","last_peered":"2023-11-28T17:10:19.305158+0000","last_clean":"2023-11-28T17:10:19.305158+0000","last_became_active":"2023-11-28T17:10:19.300801+0000","last_became_peered":"2023-11-28T17:10:19.300801+0000","last_unstale":"2023-11-28T17:10:19.305158+0000","last_undegraded":"2023-11-28T17:10:19.305158+0000","last_fullsized":"2023-11-28T17:10:19.305158+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:54:19.482117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.272951+0000","last_change":"2023-11-28T17:10:19.272951+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.272951+0000","last_undegraded":"2023-11-28T17:10:19.272951+0000","last_fullsized":"2023-11-28T17:10:19.272951+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.297931+0000","last_change":"2023-11-28T17:10:19.295030+0000","last_active":"2023-11-28T17:10:19.297931+0000","last_peered":"2023-11-28T17:10:19.297931+0000","last_clean":"2023-11-28T17:10:19.297931+0000","last_became_active":"2023-11-28T17:10:19.294679+0000","last_became_peered":"2023-11-28T17:10:19.294679+0000","last_unstale":"2023-11-28T17:10:19.297931+0000","last_undegraded":"2023-11-28T17:10:19.297931+0000","last_fullsized":"2023-11-28T17:10:19.297931+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:10:51.416766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.281053+0000","last_change":"2023-11-28T17:10:19.281053+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.281053+0000","last_undegraded":"2023-11-28T17:10:19.281053+0000","last_fullsized":"2023-11-28T17:10:19.281053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.302819+0000","last_change":"2023-11-28T17:10:19.298673+0000","last_active":"2023-11-28T17:10:19.302819+0000","last_peered":"2023-11-28T17:10:19.302819+0000","last_clean":"2023-11-28T17:10:19.302819+0000","last_became_active":"2023-11-28T17:10:19.297133+0000","last_became_peered":"2023-11-28T17:10:19.297133+0000","last_unstale":"2023-11-28T17:10:19.302819+0000","last_undegraded":"2023-11-28T17:10:19.302819+0000","last_fullsized":"2023-11-28T17:10:19.302819+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:31.526535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.273120+0000","last_change":"2023-11-28T17:10:19.273120+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.273120+0000","last_undegraded":"2023-11-28T17:10:19.273120+0000","last_fullsized":"2023-11-28T17:10:19.273120+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.304071+0000","last_change":"2023-11-28T17:10:19.304071+0000","last_active":"2023-11-28T17:10:19.304071+0000","last_peered":"2023-11-28T17:10:19.304071+0000","last_clean":"2023-11-28T17:10:19.304071+0000","last_became_active":"2023-11-28T17:10:19.303800+0000","last_became_peered":"2023-11-28T17:10:19.303800+0000","last_unstale":"2023-11-28T17:10:19.304071+0000","last_undegraded":"2023-11-28T17:10:19.304071+0000","last_fullsized":"2023-11-28T17:10:19.304071+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:47:19.431073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.283327+0000","last_change":"2023-11-28T17:10:19.283327+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.283327+0000","last_undegraded":"2023-11-28T17:10:19.283327+0000","last_fullsized":"2023-11-28T17:10:19.283327+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.297864+0000","last_change":"2023-11-28T17:10:19.295229+0000","last_active":"2023-11-28T17:10:19.297864+0000","last_peered":"2023-11-28T17:10:19.297864+0000","last_clean":"2023-11-28T17:10:19.297864+0000","last_became_active":"2023-11-28T17:10:19.294927+0000","last_became_peered":"2023-11-28T17:10:19.294927+0000","last_unstale":"2023-11-28T17:10:19.297864+0000","last_undegraded":"2023-11-28T17:10:19.297864+0000","last_fullsized":"2023-11-28T17:10:19.297864+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:47:19.431073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.287579+0000","last_change":"2023-11-28T17:10:19.287579+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.287579+0000","last_undegraded":"2023-11-28T17:10:19.287579+0000","last_fullsized":"2023-11-28T17:10:19.287579+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.302698+0000","last_change":"2023-11-28T17:10:19.300456+0000","last_active":"2023-11-28T17:10:19.302698+0000","last_peered":"2023-11-28T17:10:19.302698+0000","last_clean":"2023-11-28T17:10:19.302698+0000","last_became_active":"2023-11-28T17:10:19.300191+0000","last_became_peered":"2023-11-28T17:10:19.300191+0000","last_unstale":"2023-11-28T17:10:19.302698+0000","last_undegraded":"2023-11-28T17:10:19.302698+0000","last_fullsized":"2023-11-28T17:10:19.302698+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:10:24.406709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.293607+0000","last_change":"2023-11-28T17:10:19.293607+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.293607+0000","last_undegraded":"2023-11-28T17:10:19.293607+0000","last_fullsized":"2023-11-28T17:10:19.293607+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.303793+0000","last_change":"2023-11-28T17:10:19.302405+0000","last_active":"2023-11-28T17:10:19.303793+0000","last_peered":"2023-11-28T17:10:19.303793+0000","last_clean":"2023-11-28T17:10:19.303793+0000","last_became_active":"2023-11-28T17:10:19.301927+0000","last_became_peered":"2023-11-28T17:10:19.301927+0000","last_unstale":"2023-11-28T17:10:19.303793+0000","last_undegraded":"2023-11-28T17:10:19.303793+0000","last_fullsized":"2023-11-28T17:10:19.303793+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:16.378585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.290991+0000","last_change":"2023-11-28T17:10:19.290991+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.290991+0000","last_undegraded":"2023-11-28T17:10:19.290991+0000","last_fullsized":"2023-11-28T17:10:19.290991+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.292528+0000","last_change":"2023-11-28T17:10:19.291826+0000","last_active":"2023-11-28T17:10:19.292528+0000","last_peered":"2023-11-28T17:10:19.292528+0000","last_clean":"2023-11-28T17:10:19.292528+0000","last_became_active":"2023-11-28T17:10:19.291533+0000","last_became_peered":"2023-11-28T17:10:19.291533+0000","last_unstale":"2023-11-28T17:10:19.292528+0000","last_undegraded":"2023-11-28T17:10:19.292528+0000","last_fullsized":"2023-11-28T17:10:19.292528+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:38.203130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.285756+0000","last_change":"2023-11-28T17:10:19.285756+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.285756+0000","last_undegraded":"2023-11-28T17:10:19.285756+0000","last_fullsized":"2023-11-28T17:10:19.285756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.307878+0000","last_change":"2023-11-28T17:10:19.305127+0000","last_active":"2023-11-28T17:10:19.307878+0000","last_peered":"2023-11-28T17:10:19.307878+0000","last_clean":"2023-11-28T17:10:19.307878+0000","last_became_active":"2023-11-28T17:10:19.304819+0000","last_became_peered":"2023-11-28T17:10:19.304819+0000","last_unstale":"2023-11-28T17:10:19.307878+0000","last_undegraded":"2023-11-28T17:10:19.307878+0000","last_fullsized":"2023-11-28T17:10:19.307878+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:10:51.416766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.278204+0000","last_change":"2023-11-28T17:10:19.278204+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.278204+0000","last_undegraded":"2023-11-28T17:10:19.278204+0000","last_fullsized":"2023-11-28T17:10:19.278204+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.295477+0000","last_change":"2023-11-28T17:10:19.289570+0000","last_active":"2023-11-28T17:10:19.295477+0000","last_peered":"2023-11-28T17:10:19.295477+0000","last_clean":"2023-11-28T17:10:19.295477+0000","last_became_active":"2023-11-28T17:10:19.289310+0000","last_became_peered":"2023-11-28T17:10:19.289310+0000","last_unstale":"2023-11-28T17:10:19.295477+0000","last_undegraded":"2023-11-28T17:10:19.295477+0000","last_fullsized":"2023-11-28T17:10:19.295477+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:33:39.917238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.285572+0000","last_change":"2023-11-28T17:10:19.285572+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.285572+0000","last_undegraded":"2023-11-28T17:10:19.285572+0000","last_fullsized":"2023-11-28T17:10:19.285572+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.305335+0000","last_change":"2023-11-28T17:10:19.303515+0000","last_active":"2023-11-28T17:10:19.305335+0000","last_peered":"2023-11-28T17:10:19.305335+0000","last_clean":"2023-11-28T17:10:19.305335+0000","last_became_active":"2023-11-28T17:10:19.303217+0000","last_became_peered":"2023-11-28T17:10:19.303217+0000","last_unstale":"2023-11-28T17:10:19.305335+0000","last_undegraded":"2023-11-28T17:10:19.305335+0000","last_fullsized":"2023-11-28T17:10:19.305335+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:56:15.992690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.275322+0000","last_change":"2023-11-28T17:10:19.275322+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.275322+0000","last_undegraded":"2023-11-28T17:10:19.275322+0000","last_fullsized":"2023-11-28T17:10:19.275322+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.300712+0000","last_change":"2023-11-28T17:10:19.294783+0000","last_active":"2023-11-28T17:10:19.300712+0000","last_peered":"2023-11-28T17:10:19.300712+0000","last_clean":"2023-11-28T17:10:19.300712+0000","last_became_active":"2023-11-28T17:10:19.294480+0000","last_became_peered":"2023-11-28T17:10:19.294480+0000","last_unstale":"2023-11-28T17:10:19.300712+0000","last_undegraded":"2023-11-28T17:10:19.300712+0000","last_fullsized":"2023-11-28T17:10:19.300712+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:13:10.313820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.282382+0000","last_change":"2023-11-28T17:10:19.282382+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.282382+0000","last_undegraded":"2023-11-28T17:10:19.282382+0000","last_fullsized":"2023-11-28T17:10:19.282382+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.302973+0000","last_change":"2023-11-28T17:10:19.302973+0000","last_active":"2023-11-28T17:10:19.302973+0000","last_peered":"2023-11-28T17:10:19.302973+0000","last_clean":"2023-11-28T17:10:19.302973+0000","last_became_active":"2023-11-28T17:10:19.302711+0000","last_became_peered":"2023-11-28T17:10:19.302711+0000","last_unstale":"2023-11-28T17:10:19.302973+0000","last_undegraded":"2023-11-28T17:10:19.302973+0000","last_fullsized":"2023-11-28T17:10:19.302973+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:11:08.566273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.286820+0000","last_change":"2023-11-28T17:10:19.286820+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.286820+0000","last_undegraded":"2023-11-28T17:10:19.286820+0000","last_fullsized":"2023-11-28T17:10:19.286820+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2023-11-28T17:10:19.294619+0000","last_change":"2023-11-28T17:10:19.294619+0000","last_active":"2023-11-28T17:10:18.250875+0000","last_peered":"2023-11-28T17:10:18.250875+0000","last_clean":"2023-11-28T17:10:18.250875+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.294619+0000","last_undegraded":"2023-11-28T17:10:19.294619+0000","last_fullsized":"2023-11-28T17:10:19.294619+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:12:08.768260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.272760+0000","last_change":"2023-11-28T17:10:19.272760+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.272760+0000","last_undegraded":"2023-11-28T17:10:19.272760+0000","last_fullsized":"2023-11-28T17:10:19.272760+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.297485+0000","last_change":"2023-11-28T17:10:19.290789+0000","last_active":"2023-11-28T17:10:19.297485+0000","last_peered":"2023-11-28T17:10:19.297485+0000","last_clean":"2023-11-28T17:10:19.297485+0000","last_became_active":"2023-11-28T17:10:19.290545+0000","last_became_peered":"2023-11-28T17:10:19.290545+0000","last_unstale":"2023-11-28T17:10:19.297485+0000","last_undegraded":"2023-11-28T17:10:19.297485+0000","last_fullsized":"2023-11-28T17:10:19.297485+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:12:17.157009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.275418+0000","last_change":"2023-11-28T17:10:19.275418+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.275418+0000","last_undegraded":"2023-11-28T17:10:19.275418+0000","last_fullsized":"2023-11-28T17:10:19.275418+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.298696+0000","last_change":"2023-11-28T17:10:19.297053+0000","last_active":"2023-11-28T17:10:19.298696+0000","last_peered":"2023-11-28T17:10:19.298696+0000","last_clean":"2023-11-28T17:10:19.298696+0000","last_became_active":"2023-11-28T17:10:19.296490+0000","last_became_peered":"2023-11-28T17:10:19.296490+0000","last_unstale":"2023-11-28T17:10:19.298696+0000","last_undegraded":"2023-11-28T17:10:19.298696+0000","last_fullsized":"2023-11-28T17:10:19.298696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:13:41.044511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.287918+0000","last_change":"2023-11-28T17:10:19.287918+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.287918+0000","last_undegraded":"2023-11-28T17:10:19.287918+0000","last_fullsized":"2023-11-28T17:10:19.287918+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.307370+0000","last_change":"2023-11-28T17:10:19.305800+0000","last_active":"2023-11-28T17:10:19.307370+0000","last_peered":"2023-11-28T17:10:19.307370+0000","last_clean":"2023-11-28T17:10:19.307370+0000","last_became_active":"2023-11-28T17:10:19.305482+0000","last_became_peered":"2023-11-28T17:10:19.305482+0000","last_unstale":"2023-11-28T17:10:19.307370+0000","last_undegraded":"2023-11-28T17:10:19.307370+0000","last_fullsized":"2023-11-28T17:10:19.307370+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:03:36.600341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.274362+0000","last_change":"2023-11-28T17:10:19.274362+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.274362+0000","last_undegraded":"2023-11-28T17:10:19.274362+0000","last_fullsized":"2023-11-28T17:10:19.274362+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.306504+0000","last_change":"2023-11-28T17:10:19.303455+0000","last_active":"2023-11-28T17:10:19.306504+0000","last_peered":"2023-11-28T17:10:19.306504+0000","last_clean":"2023-11-28T17:10:19.306504+0000","last_became_active":"2023-11-28T17:10:19.303189+0000","last_became_peered":"2023-11-28T17:10:19.303189+0000","last_unstale":"2023-11-28T17:10:19.306504+0000","last_undegraded":"2023-11-28T17:10:19.306504+0000","last_fullsized":"2023-11-28T17:10:19.306504+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:57:52.242903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.278157+0000","last_change":"2023-11-28T17:10:19.278157+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.278157+0000","last_undegraded":"2023-11-28T17:10:19.278157+0000","last_fullsized":"2023-11-28T17:10:19.278157+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.297428+0000","last_change":"2023-11-28T17:10:19.291224+0000","last_active":"2023-11-28T17:10:19.297428+0000","last_peered":"2023-11-28T17:10:19.297428+0000","last_clean":"2023-11-28T17:10:19.297428+0000","last_became_active":"2023-11-28T17:10:19.289292+0000","last_became_peered":"2023-11-28T17:10:19.289292+0000","last_unstale":"2023-11-28T17:10:19.297428+0000","last_undegraded":"2023-11-28T17:10:19.297428+0000","last_fullsized":"2023-11-28T17:10:19.297428+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:26:44.987761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.275059+0000","last_change":"2023-11-28T17:10:19.275059+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.275059+0000","last_undegraded":"2023-11-28T17:10:19.275059+0000","last_fullsized":"2023-11-28T17:10:19.275059+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.294035+0000","last_change":"2023-11-28T17:10:19.288811+0000","last_active":"2023-11-28T17:10:19.294035+0000","last_peered":"2023-11-28T17:10:19.294035+0000","last_clean":"2023-11-28T17:10:19.294035+0000","last_became_active":"2023-11-28T17:10:19.288560+0000","last_became_peered":"2023-11-28T17:10:19.288560+0000","last_unstale":"2023-11-28T17:10:19.294035+0000","last_undegraded":"2023-11-28T17:10:19.294035+0000","last_fullsized":"2023-11-28T17:10:19.294035+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:12:08.768260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.272668+0000","last_change":"2023-11-28T17:10:19.272668+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.272668+0000","last_undegraded":"2023-11-28T17:10:19.272668+0000","last_fullsized":"2023-11-28T17:10:19.272668+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.295522+0000","last_change":"2023-11-28T17:10:19.292580+0000","last_active":"2023-11-28T17:10:19.295522+0000","last_peered":"2023-11-28T17:10:19.295522+0000","last_clean":"2023-11-28T17:10:19.295522+0000","last_became_active":"2023-11-28T17:10:19.292338+0000","last_became_peered":"2023-11-28T17:10:19.292338+0000","last_unstale":"2023-11-28T17:10:19.295522+0000","last_undegraded":"2023-11-28T17:10:19.295522+0000","last_fullsized":"2023-11-28T17:10:19.295522+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:36:39.864313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.286948+0000","last_change":"2023-11-28T17:10:19.286948+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.286948+0000","last_undegraded":"2023-11-28T17:10:19.286948+0000","last_fullsized":"2023-11-28T17:10:19.286948+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.296671+0000","last_change":"2023-11-28T17:10:19.291370+0000","last_active":"2023-11-28T17:10:19.296671+0000","last_peered":"2023-11-28T17:10:19.296671+0000","last_clean":"2023-11-28T17:10:19.296671+0000","last_became_active":"2023-11-28T17:10:19.291024+0000","last_became_peered":"2023-11-28T17:10:19.291024+0000","last_unstale":"2023-11-28T17:10:19.296671+0000","last_undegraded":"2023-11-28T17:10:19.296671+0000","last_fullsized":"2023-11-28T17:10:19.296671+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:12:17.157009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.283656+0000","last_change":"2023-11-28T17:10:19.283656+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.283656+0000","last_undegraded":"2023-11-28T17:10:19.283656+0000","last_fullsized":"2023-11-28T17:10:19.283656+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.299855+0000","last_change":"2023-11-28T17:10:19.298073+0000","last_active":"2023-11-28T17:10:19.299855+0000","last_peered":"2023-11-28T17:10:19.299855+0000","last_clean":"2023-11-28T17:10:19.299855+0000","last_became_active":"2023-11-28T17:10:19.297817+0000","last_became_peered":"2023-11-28T17:10:19.297817+0000","last_unstale":"2023-11-28T17:10:19.299855+0000","last_undegraded":"2023-11-28T17:10:19.299855+0000","last_fullsized":"2023-11-28T17:10:19.299855+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:03:36.600341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.287035+0000","last_change":"2023-11-28T17:10:19.287035+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.287035+0000","last_undegraded":"2023-11-28T17:10:19.287035+0000","last_fullsized":"2023-11-28T17:10:19.287035+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.302167+0000","last_change":"2023-11-28T17:10:19.298953+0000","last_active":"2023-11-28T17:10:19.302167+0000","last_peered":"2023-11-28T17:10:19.302167+0000","last_clean":"2023-11-28T17:10:19.302167+0000","last_became_active":"2023-11-28T17:10:19.298664+0000","last_became_peered":"2023-11-28T17:10:19.298664+0000","last_unstale":"2023-11-28T17:10:19.302167+0000","last_undegraded":"2023-11-28T17:10:19.302167+0000","last_fullsized":"2023-11-28T17:10:19.302167+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:33:39.917238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.279091+0000","last_change":"2023-11-28T17:10:19.279091+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.279091+0000","last_undegraded":"2023-11-28T17:10:19.279091+0000","last_fullsized":"2023-11-28T17:10:19.279091+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.300715+0000","last_change":"2023-11-28T17:10:19.294746+0000","last_active":"2023-11-28T17:10:19.300715+0000","last_peered":"2023-11-28T17:10:19.300715+0000","last_clean":"2023-11-28T17:10:19.300715+0000","last_became_active":"2023-11-28T17:10:19.294468+0000","last_became_peered":"2023-11-28T17:10:19.294468+0000","last_unstale":"2023-11-28T17:10:19.300715+0000","last_undegraded":"2023-11-28T17:10:19.300715+0000","last_fullsized":"2023-11-28T17:10:19.300715+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:02:05.023289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.284593+0000","last_change":"2023-11-28T17:10:19.284593+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.284593+0000","last_undegraded":"2023-11-28T17:10:19.284593+0000","last_fullsized":"2023-11-28T17:10:19.284593+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.301529+0000","last_change":"2023-11-28T17:10:19.295931+0000","last_active":"2023-11-28T17:10:19.301529+0000","last_peered":"2023-11-28T17:10:19.301529+0000","last_clean":"2023-11-28T17:10:19.301529+0000","last_became_active":"2023-11-28T17:10:19.291579+0000","last_became_peered":"2023-11-28T17:10:19.291579+0000","last_unstale":"2023-11-28T17:10:19.301529+0000","last_undegraded":"2023-11-28T17:10:19.301529+0000","last_fullsized":"2023-11-28T17:10:19.301529+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:13:10.313820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.285749+0000","last_change":"2023-11-28T17:10:19.285749+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.285749+0000","last_undegraded":"2023-11-28T17:10:19.285749+0000","last_fullsized":"2023-11-28T17:10:19.285749+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.293009+0000","last_change":"2023-11-28T17:10:19.287226+0000","last_active":"2023-11-28T17:10:19.293009+0000","last_peered":"2023-11-28T17:10:19.293009+0000","last_clean":"2023-11-28T17:10:19.293009+0000","last_became_active":"2023-11-28T17:10:19.286939+0000","last_became_peered":"2023-11-28T17:10:19.286939+0000","last_unstale":"2023-11-28T17:10:19.293009+0000","last_undegraded":"2023-11-28T17:10:19.293009+0000","last_fullsized":"2023-11-28T17:10:19.293009+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:11:08.566273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.281901+0000","last_change":"2023-11-28T17:10:19.281901+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.281901+0000","last_undegraded":"2023-11-28T17:10:19.281901+0000","last_fullsized":"2023-11-28T17:10:19.281901+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.299783+0000","last_change":"2023-11-28T17:10:19.296746+0000","last_active":"2023-11-28T17:10:19.299783+0000","last_peered":"2023-11-28T17:10:19.299783+0000","last_clean":"2023-11-28T17:10:19.299783+0000","last_became_active":"2023-11-28T17:10:19.296442+0000","last_became_peered":"2023-11-28T17:10:19.296442+0000","last_unstale":"2023-11-28T17:10:19.299783+0000","last_undegraded":"2023-11-28T17:10:19.299783+0000","last_fullsized":"2023-11-28T17:10:19.299783+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:53:51.233176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-11-28T17:10:19.279146+0000","last_change":"2023-11-28T17:10:19.279146+0000","last_active":"2023-11-28T17:10:19.256392+0000","last_peered":"2023-11-28T17:10:19.256392+0000","last_clean":"2023-11-28T17:10:19.256392+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-28T17:10:19.279146+0000","last_undegraded":"2023-11-28T17:10:19.279146+0000","last_fullsized":"2023-11-28T17:10:19.279146+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-11-28T17:10:19.298869+0000","last_change":"2023-11-28T17:10:19.297204+0000","last_active":"2023-11-28T17:10:19.298869+0000","last_peered":"2023-11-28T17:10:19.298869+0000","last_clean":"2023-11-28T17:10:19.298869+0000","last_became_active":"2023-11-28T17:10:19.296821+0000","last_became_peered":"2023-11-28T17:10:19.296821+0000","last_unstale":"2023-11-28T17:10:19.298869+0000","last_undegraded":"2023-11-28T17:10:19.298869+0000","last_fullsized":"2023-11-28T17:10:19.298869+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:01:47.887732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":176,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-11-28T17:10:18.261365+0000","last_change":"2023-11-28T17:08:51.392782+0000","last_active":"2023-11-28T17:10:18.261365+0000","last_peered":"2023-11-28T17:10:18.261365+0000","last_clean":"2023-11-28T17:10:18.261365+0000","last_became_active":"2023-11-28T17:08:51.390415+0000","last_became_peered":"2023-11-28T17:08:51.390415+0000","last_unstale":"2023-11-28T17:10:18.261365+0000","last_undegraded":"2023-11-28T17:10:18.261365+0000","last_fullsized":"2023-11-28T17:10:18.261365+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_clean_scrub_stamp":"2023-11-28T17:08:49.967682+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:03:32.638685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":132063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":1.111},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":1.6539999999999999}]},{"osd":1,"last update":"Tue Nov 28 17:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.3440000000000001}]},{"osd":2,"last update":"Tue Nov 28 17:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.351,"5min":1.351,"15min":1.351},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.7029999999999998,"5min":2.7029999999999998,"15min":2.7029999999999998},"last":1.294}]},{"osd":3,"last update":"Tue Nov 28 17:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":1.054},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":1.0189999999999999}]},{"osd":4,"last update":"Tue Nov 28 17:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.327,"15min":1.327},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":2.202,"5min":2.202,"15min":2.202},"last":1.798},{"interface":"front","average":{"1min":1.377,"5min":1.377,"15min":1.377},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":2.069,"5min":2.069,"15min":2.069},"last":1.4930000000000001}]},{"osd":5,"last update":"Tue Nov 28 17:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.476,"15min":1.476},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.4420000000000002,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.7850000000000001,"5min":2.7850000000000001,"15min":2.7850000000000001},"last":1.891}]},{"osd":6,"last update":"Tue Nov 28 17:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.742,"15min":1.742},"min":{"1min":1.276,"5min":1.276,"15min":1.276},"max":{"1min":3.8010000000000002,"5min":3.8010000000000002,"15min":3.8010000000000002},"last":1.9590000000000001},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":3.4049999999999998,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.742}]}]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":132063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":4.3579999999999997,"5min":4.3579999999999997,"15min":4.3579999999999997},"last":0.872},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.841,"5min":1.841,"15min":1.841},"last":0.92900000000000005}]},{"osd":1,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":4.1580000000000004,"5min":4.1580000000000004,"15min":4.1580000000000004},"last":1.522},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":1.6259999999999999}]},{"osd":2,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":1.079},{"interface":"front","average":{"1min":1.534,"5min":1.534,"15min":1.534},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":4.2750000000000004,"5min":4.2750000000000004,"15min":4.2750000000000004},"last":1.798}]},{"osd":3,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.74},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":1.9399999999999999}]},{"osd":4,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.363,"15min":1.363},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":2.1299999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.8460000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.4279999999999999}]},{"osd":5,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.522,"15min":1.522},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":1.895},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":3.0030000000000001,"5min":3.0030000000000001,"15min":3.0030000000000001},"last":2.0209999999999999}]},{"osd":7,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.514,"15min":1.514},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.524,"5min":2.524,"15min":2.524},"last":1.982},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.5660000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":722431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":1.2569999999999999}]},{"osd":1,"last update":"Tue Nov 28 17:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":0.94499999999999995}]},{"osd":2,"last update":"Tue Nov 28 17:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":1.6100000000000001},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.1679999999999999}]},{"osd":3,"last update":"Tue Nov 28 17:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":1.373},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.4180000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.488}]},{"osd":4,"last update":"Tue Nov 28 17:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.6179999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.554},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":1.9099999999999999},"last":1.8220000000000001}]},{"osd":6,"last update":"Tue Nov 28 17:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.415,"15min":1.415},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.873},{"interface":"front","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.7759999999999998,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.6579999999999999}]},{"osd":7,"last update":"Tue Nov 28 17:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.036,"5min":2.036,"15min":2.036},"last":1.716},{"interface":"front","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":1.7669999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":132063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.048,"15min":1.048},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":1.0680000000000001}]},{"osd":1,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":0.88700000000000001}]},{"osd":2,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.745},{"interface":"front","average":{"1min":1.361,"5min":1.361,"15min":1.361},"min":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.1679999999999999}]},{"osd":3,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.3109999999999999,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":1.256}]},{"osd":5,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.476,"15min":1.476},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":3.1760000000000002,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.512},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.4220000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.97599999999999998}]},{"osd":6,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.1520000000000001},"last":1.55},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":3.2799999999999998,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":1.579}]},{"osd":7,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.669,"5min":2.669,"15min":2.669},"last":1.133},{"interface":"front","average":{"1min":1.673,"5min":1.673,"15min":1.673},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":2.387,"5min":2.387,"15min":2.387},"last":1.458}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":2.444,"5min":2.444,"15min":2.444},"last":1.363}]},{"osd":1,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":1.071}]},{"osd":2,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":2.319,"5min":2.319,"15min":2.319},"last":1.296}]},{"osd":4,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.292,"15min":1.292},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.5699999999999998,"5min":2.5699999999999998,"15min":2.5699999999999998},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.5059999999999998,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":1.5720000000000001}]},{"osd":5,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.389},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.2330000000000001}]},{"osd":6,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"max":{"1min":2.6000000000000001,"5min":2.6000000000000001,"15min":2.6000000000000001},"last":1.446}]},{"osd":7,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":1.609},{"interface":"front","average":{"1min":1.496,"5min":1.496,"15min":1.496},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.5419999999999998,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":1.4610000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.6560000000000001,"5min":2.6560000000000001,"15min":2.6560000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.39400000000000002}]},{"osd":1,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":0.51000000000000001}]},{"osd":3,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":2.5590000000000002,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.56000000000000005}]},{"osd":4,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.88300000000000001}]},{"osd":5,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.302,"5min":1.302,"15min":1.302},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.71199999999999997}]},{"osd":6,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.7429999999999999,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":0.90900000000000003}]},{"osd":7,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":0.90000000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":722431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.4940000000000002,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":4.0819999999999999,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.35499999999999998}]},{"osd":2,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":4.0090000000000003,"5min":4.0090000000000003,"15min":4.0090000000000003},"last":0.496}]},{"osd":3,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":3.9569999999999999,"5min":3.9569999999999999,"15min":3.9569999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.6389999999999998,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":0.40799999999999997}]},{"osd":4,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":4.29,"5min":4.29,"15min":4.29},"last":1.1279999999999999}]},{"osd":5,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":4.1589999999999998,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.601,"5min":1.601,"15min":1.601},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":4.2460000000000004,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":0.89800000000000002}]},{"osd":6,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":1.7809999999999999},"min":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":4.3419999999999996,"5min":4.3419999999999996,"15min":4.3419999999999996},"last":0.873},{"interface":"front","average":{"1min":1.766,"5min":1.766,"15min":1.766},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":4.1239999999999997,"5min":4.1239999999999997,"15min":4.1239999999999997},"last":0.82399999999999995}]},{"osd":7,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.804,"5min":1.804,"15min":1.804},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.4220000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.85099999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 17:09:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.40600000000000003}]},{"osd":2,"last update":"Tue Nov 28 17:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.601,"5min":1.601,"15min":1.601},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.46600000000000003}]},{"osd":3,"last update":"Tue Nov 28 17:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.42899999999999999}]},{"osd":4,"last update":"Tue Nov 28 17:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.464,"5min":1.464,"15min":1.464},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.746}]},{"osd":5,"last update":"Tue Nov 28 17:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.2570000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.71199999999999997}]},{"osd":6,"last update":"Tue Nov 28 17:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.423,"15min":1.423},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.0780000000000001}]},{"osd":7,"last update":"Tue Nov 28 17:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.76200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T17:10:24.772 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:10:25.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.014+0000 7f21f202d640 1 Processor -- start 2023-11-28T17:10:25.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.014+0000 7f21f202d640 1 -- start start 2023-11-28T17:10:25.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.015+0000 7f21f202d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21ec13cee0 0x7f21ec06c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:25.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.015+0000 7f21f202d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21ec0aea20 con 0x7f21ec15aba0 2023-11-28T17:10:25.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.015+0000 7f21f202d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21ec13d2c0 con 0x7f21ec140200 2023-11-28T17:10:25.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.015+0000 7f21f202d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec15b3a0 con 0x7f21ec13cee0 2023-11-28T17:10:25.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.015+0000 7f21ebfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f21ec15aba0 0x7f21ec0cd8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49082/0 (socket says 172.21.15.116:49082) 2023-11-28T17:10:25.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21ebfff640 1 -- 172.21.15.116:0/138170528 learned_addr learned my addr 172.21.15.116:0/138170528 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:25.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21eb7fe640 1 --2- 172.21.15.116:0/138170528 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21ec13cee0 0x7f21ec06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:25.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21eaffd640 1 -- 172.21.15.116:0/138170528 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940279470 0 0) 0x7f21ec0aea20 con 0x7f21ec15aba0 2023-11-28T17:10:25.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21eaffd640 1 -- 172.21.15.116:0/138170528 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21d8003520 con 0x7f21ec15aba0 2023-11-28T17:10:25.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21eaffd640 1 -- 172.21.15.116:0/138170528 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1177173078 0 0) 0x7f21d8003520 con 0x7f21ec15aba0 2023-11-28T17:10:25.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21eaffd640 1 -- 172.21.15.116:0/138170528 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21ec13cee0 msgr2=0x7f21ec06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:25.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21eaffd640 1 --2- 172.21.15.116:0/138170528 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21ec13cee0 0x7f21ec06c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21eaffd640 1 -- 172.21.15.116:0/138170528 >> v1:172.21.15.156:6789/0 conn(0x7f21ec140200 legacy=0x7f21ec15a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:25.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21eaffd640 1 -- 172.21.15.116:0/138170528 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ec15b0c0 con 0x7f21ec15aba0 2023-11-28T17:10:25.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21eaffd640 1 -- 172.21.15.116:0/138170528 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f21e40019f0 con 0x7f21ec15aba0 2023-11-28T17:10:25.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21eaffd640 1 -- 172.21.15.116:0/138170528 >> v1:172.21.15.116:6789/0 conn(0x7f21ec15aba0 legacy=0x7f21ec0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:25.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21eaffd640 1 --2- 172.21.15.116:0/138170528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21d80039a0 0x7f21d8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:25.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21eaffd640 1 --2- 172.21.15.116:0/138170528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21d80047c0 0x7f21d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:25.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21eaffd640 1 --2- 172.21.15.116:0/138170528 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21ec13cee0 0x7f21d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:25.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.016+0000 7f21eaffd640 1 -- 172.21.15.116:0/138170528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec15b3a0 con 0x7f21d80047c0 2023-11-28T17:10:25.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.017+0000 7f21eaffd640 1 -- 172.21.15.116:0/138170528 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d800bab0 con 0x7f21d80039a0 2023-11-28T17:10:25.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.017+0000 7f21eaffd640 1 -- 172.21.15.116:0/138170528 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d800bc70 con 0x7f21ec13cee0 2023-11-28T17:10:25.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.017+0000 7f21e3fff640 1 --2- 172.21.15.116:0/138170528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21d80047c0 0x7f21d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:25.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.017+0000 7f21eb7fe640 1 --2- 172.21.15.116:0/138170528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21d80039a0 0x7f21d8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:25.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.017+0000 7f21ebfff640 1 --2- 172.21.15.116:0/138170528 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21ec13cee0 0x7f21d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:25.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.017+0000 7f21e3fff640 1 -- 172.21.15.116:0/138170528 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21ec13cee0 msgr2=0x7f21d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:25.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.017+0000 7f21e3fff640 1 --2- 172.21.15.116:0/138170528 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21ec13cee0 0x7f21d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.017+0000 7f21e3fff640 1 -- 172.21.15.116:0/138170528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21d80039a0 msgr2=0x7f21d8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:25.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.017+0000 7f21e3fff640 1 --2- 172.21.15.116:0/138170528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21d80039a0 0x7f21d8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.017+0000 7f21e3fff640 1 -- 172.21.15.116:0/138170528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d8003520 con 0x7f21d80047c0 2023-11-28T17:10:25.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.017+0000 7f21ebfff640 1 --2- 172.21.15.116:0/138170528 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21ec13cee0 0x7f21d80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:25.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.017+0000 7f21eb7fe640 1 --2- 172.21.15.116:0/138170528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21d80039a0 0x7f21d8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:25.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.018+0000 7f21e3fff640 1 --2- 172.21.15.116:0/138170528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21d80047c0 0x7f21d8004c40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f21d40015b0 tx=0x7f21d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e00e2c7404f1b9a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:25.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.018+0000 7f21eaffd640 1 -- 172.21.15.116:0/138170528 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d40052d0 con 0x7f21d80047c0 2023-11-28T17:10:25.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.018+0000 7f21eaffd640 1 -- 172.21.15.116:0/138170528 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d402e050 con 0x7f21d80047c0 2023-11-28T17:10:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.018+0000 7f21eaffd640 1 -- 172.21.15.116:0/138170528 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d40035f0 con 0x7f21d80047c0 2023-11-28T17:10:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.019+0000 7f21f202d640 1 -- 172.21.15.116:0/138170528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21d80047c0 msgr2=0x7f21d8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.019+0000 7f21f202d640 1 --2- 172.21.15.116:0/138170528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21d80047c0 0x7f21d8004c40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f21d40015b0 tx=0x7f21d80057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.019+0000 7f21ebfff640 1 -- 172.21.15.116:0/138170528 reap_dead start 2023-11-28T17:10:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.019+0000 7f21f202d640 1 -- 172.21.15.116:0/138170528 shutdown_connections 2023-11-28T17:10:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.019+0000 7f21f202d640 1 -- 172.21.15.116:0/138170528 >> 172.21.15.116:0/138170528 conn(0x7f21ec06a4c0 msgr2=0x7f21ec06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.019+0000 7f21f202d640 1 -- 172.21.15.116:0/138170528 shutdown_connections 2023-11-28T17:10:25.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.019+0000 7f21f202d640 1 -- 172.21.15.116:0/138170528 wait complete. 2023-11-28T17:10:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.021+0000 7f21f202d640 1 Processor -- start 2023-11-28T17:10:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.021+0000 7f21f202d640 1 -- start start 2023-11-28T17:10:25.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.022+0000 7f21f202d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21d80047c0 0x7f21ec1ecad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.022+0000 7f21f202d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec158020 0x7f21ec1584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.022+0000 7f21f202d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21ec13b060 0x7f21ec13b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.022+0000 7f21f202d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec13d460 con 0x7f21ec158020 2023-11-28T17:10:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.022+0000 7f21f202d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec13dd00 con 0x7f21d80047c0 2023-11-28T17:10:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.022+0000 7f21f202d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec1589e0 con 0x7f21ec13b060 2023-11-28T17:10:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.022+0000 7f21e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec158020 0x7f21ec1584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.022+0000 7f21e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec158020 0x7f21ec1584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57614/0 (socket says 172.21.15.116:57614) 2023-11-28T17:10:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.022+0000 7f21e3fff640 1 -- 172.21.15.116:0/3930978672 learned_addr learned my addr 172.21.15.116:0/3930978672 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.023+0000 7f21e3fff640 1 -- 172.21.15.116:0/3930978672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21ec13b060 msgr2=0x7f21ec13b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.023+0000 7f21e3fff640 1 --2- 172.21.15.116:0/3930978672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21ec13b060 0x7f21ec13b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.023+0000 7f21eb7fe640 1 --2- 172.21.15.116:0/3930978672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21d80047c0 0x7f21ec1ecad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:25.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.023+0000 7f21e3fff640 1 -- 172.21.15.116:0/3930978672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21d80047c0 msgr2=0x7f21ec1ecad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.023+0000 7f21e3fff640 1 --2- 172.21.15.116:0/3930978672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21d80047c0 0x7f21ec1ecad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.023+0000 7f21e3fff640 1 -- 172.21.15.116:0/3930978672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ec0b17b0 con 0x7f21ec158020 2023-11-28T17:10:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.023+0000 7f21e3fff640 1 --2- 172.21.15.116:0/3930978672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec158020 0x7f21ec1584a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f21d4005040 tx=0x7f21d4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.023+0000 7f21e8ff9640 1 -- 172.21.15.116:0/3930978672 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d40076c0 con 0x7f21ec158020 2023-11-28T17:10:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.023+0000 7f21e8ff9640 1 -- 172.21.15.116:0/3930978672 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d4007a30 con 0x7f21ec158020 2023-11-28T17:10:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.023+0000 7f21e8ff9640 1 -- 172.21.15.116:0/3930978672 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d4006be0 con 0x7f21ec158020 2023-11-28T17:10:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.023+0000 7f21f202d640 1 -- 172.21.15.116:0/3930978672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ec1ed010 con 0x7f21ec158020 2023-11-28T17:10:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.023+0000 7f21f202d640 1 -- 172.21.15.116:0/3930978672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ec13d640 con 0x7f21ec158020 2023-11-28T17:10:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.025+0000 7f21f202d640 1 -- 172.21.15.116:0/3930978672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ec13d460 con 0x7f21ec158020 2023-11-28T17:10:25.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.025+0000 7f21e8ff9640 1 -- 172.21.15.116:0/3930978672 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f21d4006d40 con 0x7f21ec158020 2023-11-28T17:10:25.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.026+0000 7f21e8ff9640 1 --2- 172.21.15.116:0/3930978672 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f21cc06b2a0 0x7f21cc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:25.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.026+0000 7f21e8ff9640 1 -- 172.21.15.116:0/3930978672 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21d40ab8e0 con 0x7f21ec158020 2023-11-28T17:10:25.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.026+0000 7f21eb7fe640 1 --2- 172.21.15.116:0/3930978672 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f21cc06b2a0 0x7f21cc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:25.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.027+0000 7f21eb7fe640 1 --2- 172.21.15.116:0/3930978672 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f21cc06b2a0 0x7f21cc06d760 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f21ec13c070 tx=0x7f21dc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:25.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.031+0000 7f21e8ff9640 1 -- 172.21.15.116:0/3930978672 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f21d4076900 con 0x7f21ec158020 2023-11-28T17:10:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.203+0000 7f21f202d640 1 -- 172.21.15.116:0/3930978672 --> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f21ec13ba50 con 0x7f21cc06b2a0 2023-11-28T17:10:25.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.226+0000 7f21e8ff9640 1 -- 172.21.15.116:0/3930978672 <== mgr.4098 v2:172.21.15.116:6832/1077400178 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286850 (secure 0 0 0) 0x7f21ec13ba50 con 0x7f21cc06b2a0 2023-11-28T17:10:25.230 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:25.232 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-28T17:10:25.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.232+0000 7f21f202d640 1 -- 172.21.15.116:0/3930978672 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f21cc06b2a0 msgr2=0x7f21cc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:25.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.232+0000 7f21f202d640 1 --2- 172.21.15.116:0/3930978672 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f21cc06b2a0 0x7f21cc06d760 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f21ec13c070 tx=0x7f21dc008040 comp rx=0 tx=0).stop 2023-11-28T17:10:25.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.232+0000 7f21f202d640 1 -- 172.21.15.116:0/3930978672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec158020 msgr2=0x7f21ec1584a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:25.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.232+0000 7f21f202d640 1 --2- 172.21.15.116:0/3930978672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec158020 0x7f21ec1584a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f21d4005040 tx=0x7f21d4001120 comp rx=0 tx=0).stop 2023-11-28T17:10:25.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.232+0000 7f21f202d640 1 -- 172.21.15.116:0/3930978672 shutdown_connections 2023-11-28T17:10:25.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.232+0000 7f21f202d640 1 --2- 172.21.15.116:0/3930978672 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f21cc06b2a0 0x7f21cc06d760 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.232+0000 7f21f202d640 1 --2- 172.21.15.116:0/3930978672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21ec13b060 0x7f21ec13b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.232+0000 7f21f202d640 1 --2- 172.21.15.116:0/3930978672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ec158020 0x7f21ec1584a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.232+0000 7f21f202d640 1 --2- 172.21.15.116:0/3930978672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21d80047c0 0x7f21ec1ecad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.232+0000 7f21f202d640 1 -- 172.21.15.116:0/3930978672 >> 172.21.15.116:0/3930978672 conn(0x7f21ec06a4c0 msgr2=0x7f21ec15b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:25.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.232+0000 7f21f202d640 1 -- 172.21.15.116:0/3930978672 shutdown_connections 2023-11-28T17:10:25.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.233+0000 7f21f202d640 1 -- 172.21.15.116:0/3930978672 wait complete. 2023-11-28T17:10:25.252 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2023-11-28T17:10:25.065158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237036,"kb_used_data":4232,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737684,"statfs":{"total":773094113280,"available":772851388416,"internally_reserved":0,"allocated":4333568,"data_stored":2396860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001792"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.331270+0000","last_change":"2023-11-28T17:10:20.348463+0000","last_active":"2023-11-28T17:10:21.331270+0000","last_peered":"2023-11-28T17:10:21.331270+0000","last_clean":"2023-11-28T17:10:21.331270+0000","last_became_active":"2023-11-28T17:10:20.348151+0000","last_became_peered":"2023-11-28T17:10:20.348151+0000","last_unstale":"2023-11-28T17:10:21.331270+0000","last_undegraded":"2023-11-28T17:10:21.331270+0000","last_fullsized":"2023-11-28T17:10:21.331270+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:26:00.875011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.542891+0000","last_change":"2023-11-28T17:10:20.341566+0000","last_active":"2023-11-28T17:10:21.542891+0000","last_peered":"2023-11-28T17:10:21.542891+0000","last_clean":"2023-11-28T17:10:21.542891+0000","last_became_active":"2023-11-28T17:10:20.341251+0000","last_became_peered":"2023-11-28T17:10:20.341251+0000","last_unstale":"2023-11-28T17:10:21.542891+0000","last_undegraded":"2023-11-28T17:10:21.542891+0000","last_fullsized":"2023-11-28T17:10:21.542891+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:12:51.144707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.908840+0000","last_change":"2023-11-28T17:10:20.350810+0000","last_active":"2023-11-28T17:10:22.908840+0000","last_peered":"2023-11-28T17:10:22.908840+0000","last_clean":"2023-11-28T17:10:22.908840+0000","last_became_active":"2023-11-28T17:10:20.350548+0000","last_became_peered":"2023-11-28T17:10:20.350548+0000","last_unstale":"2023-11-28T17:10:22.908840+0000","last_undegraded":"2023-11-28T17:10:22.908840+0000","last_fullsized":"2023-11-28T17:10:22.908840+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:10.586414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.908561+0000","last_change":"2023-11-28T17:10:20.355425+0000","last_active":"2023-11-28T17:10:22.908561+0000","last_peered":"2023-11-28T17:10:22.908561+0000","last_clean":"2023-11-28T17:10:22.908561+0000","last_became_active":"2023-11-28T17:10:20.354551+0000","last_became_peered":"2023-11-28T17:10:20.354551+0000","last_unstale":"2023-11-28T17:10:22.908561+0000","last_undegraded":"2023-11-28T17:10:22.908561+0000","last_fullsized":"2023-11-28T17:10:22.908561+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:48:55.325689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.543041+0000","last_change":"2023-11-28T17:10:20.352258+0000","last_active":"2023-11-28T17:10:21.543041+0000","last_peered":"2023-11-28T17:10:21.543041+0000","last_clean":"2023-11-28T17:10:21.543041+0000","last_became_active":"2023-11-28T17:10:20.351778+0000","last_became_peered":"2023-11-28T17:10:20.351778+0000","last_unstale":"2023-11-28T17:10:21.543041+0000","last_undegraded":"2023-11-28T17:10:21.543041+0000","last_fullsized":"2023-11-28T17:10:21.543041+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:48:55.325689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.326320+0000","last_change":"2023-11-28T17:10:20.352605+0000","last_active":"2023-11-28T17:10:21.326320+0000","last_peered":"2023-11-28T17:10:21.326320+0000","last_clean":"2023-11-28T17:10:21.326320+0000","last_became_active":"2023-11-28T17:10:20.352300+0000","last_became_peered":"2023-11-28T17:10:20.352300+0000","last_unstale":"2023-11-28T17:10:21.326320+0000","last_undegraded":"2023-11-28T17:10:21.326320+0000","last_fullsized":"2023-11-28T17:10:21.326320+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:01.430445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.330200+0000","last_change":"2023-11-28T17:10:20.348697+0000","last_active":"2023-11-28T17:10:21.330200+0000","last_peered":"2023-11-28T17:10:21.330200+0000","last_clean":"2023-11-28T17:10:21.330200+0000","last_became_active":"2023-11-28T17:10:20.348431+0000","last_became_peered":"2023-11-28T17:10:20.348431+0000","last_unstale":"2023-11-28T17:10:21.330200+0000","last_undegraded":"2023-11-28T17:10:21.330200+0000","last_fullsized":"2023-11-28T17:10:21.330200+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:01:48.893249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.329804+0000","last_change":"2023-11-28T17:10:20.349660+0000","last_active":"2023-11-28T17:10:21.329804+0000","last_peered":"2023-11-28T17:10:21.329804+0000","last_clean":"2023-11-28T17:10:21.329804+0000","last_became_active":"2023-11-28T17:10:20.348741+0000","last_became_peered":"2023-11-28T17:10:20.348741+0000","last_unstale":"2023-11-28T17:10:21.329804+0000","last_undegraded":"2023-11-28T17:10:21.329804+0000","last_fullsized":"2023-11-28T17:10:21.329804+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:42:59.695037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.909920+0000","last_change":"2023-11-28T17:10:20.336539+0000","last_active":"2023-11-28T17:10:22.909920+0000","last_peered":"2023-11-28T17:10:22.909920+0000","last_clean":"2023-11-28T17:10:22.909920+0000","last_became_active":"2023-11-28T17:10:20.335567+0000","last_became_peered":"2023-11-28T17:10:20.335567+0000","last_unstale":"2023-11-28T17:10:22.909920+0000","last_undegraded":"2023-11-28T17:10:22.909920+0000","last_fullsized":"2023-11-28T17:10:22.909920+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:57:53.248420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.909253+0000","last_change":"2023-11-28T17:10:20.346760+0000","last_active":"2023-11-28T17:10:22.909253+0000","last_peered":"2023-11-28T17:10:22.909253+0000","last_clean":"2023-11-28T17:10:22.909253+0000","last_became_active":"2023-11-28T17:10:20.346504+0000","last_became_peered":"2023-11-28T17:10:20.346504+0000","last_unstale":"2023-11-28T17:10:22.909253+0000","last_undegraded":"2023-11-28T17:10:22.909253+0000","last_fullsized":"2023-11-28T17:10:22.909253+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:03:45.389221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.327328+0000","last_change":"2023-11-28T17:10:20.317810+0000","last_active":"2023-11-28T17:10:21.327328+0000","last_peered":"2023-11-28T17:10:21.327328+0000","last_clean":"2023-11-28T17:10:21.327328+0000","last_became_active":"2023-11-28T17:10:20.317396+0000","last_became_peered":"2023-11-28T17:10:20.317396+0000","last_unstale":"2023-11-28T17:10:21.327328+0000","last_undegraded":"2023-11-28T17:10:21.327328+0000","last_fullsized":"2023-11-28T17:10:21.327328+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:58.105898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.908437+0000","last_change":"2023-11-28T17:10:20.344958+0000","last_active":"2023-11-28T17:10:22.908437+0000","last_peered":"2023-11-28T17:10:22.908437+0000","last_clean":"2023-11-28T17:10:22.908437+0000","last_became_active":"2023-11-28T17:10:20.344657+0000","last_became_peered":"2023-11-28T17:10:20.344657+0000","last_unstale":"2023-11-28T17:10:22.908437+0000","last_undegraded":"2023-11-28T17:10:22.908437+0000","last_fullsized":"2023-11-28T17:10:22.908437+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:01.430445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.733855+0000","last_change":"2023-11-28T17:10:20.344228+0000","last_active":"2023-11-28T17:10:21.733855+0000","last_peered":"2023-11-28T17:10:21.733855+0000","last_clean":"2023-11-28T17:10:21.733855+0000","last_became_active":"2023-11-28T17:10:20.343990+0000","last_became_peered":"2023-11-28T17:10:20.343990+0000","last_unstale":"2023-11-28T17:10:21.733855+0000","last_undegraded":"2023-11-28T17:10:21.733855+0000","last_fullsized":"2023-11-28T17:10:21.733855+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:54:33.890435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.908769+0000","last_change":"2023-11-28T17:10:20.345449+0000","last_active":"2023-11-28T17:10:22.908769+0000","last_peered":"2023-11-28T17:10:22.908769+0000","last_clean":"2023-11-28T17:10:22.908769+0000","last_became_active":"2023-11-28T17:10:20.345190+0000","last_became_peered":"2023-11-28T17:10:20.345190+0000","last_unstale":"2023-11-28T17:10:22.908769+0000","last_undegraded":"2023-11-28T17:10:22.908769+0000","last_fullsized":"2023-11-28T17:10:22.908769+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:49:28.491600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.907414+0000","last_change":"2023-11-28T17:10:20.348961+0000","last_active":"2023-11-28T17:10:22.907414+0000","last_peered":"2023-11-28T17:10:22.907414+0000","last_clean":"2023-11-28T17:10:22.907414+0000","last_became_active":"2023-11-28T17:10:20.348707+0000","last_became_peered":"2023-11-28T17:10:20.348707+0000","last_unstale":"2023-11-28T17:10:22.907414+0000","last_undegraded":"2023-11-28T17:10:22.907414+0000","last_fullsized":"2023-11-28T17:10:22.907414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:01:48.893249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.907204+0000","last_change":"2023-11-28T17:10:20.333457+0000","last_active":"2023-11-28T17:10:22.907204+0000","last_peered":"2023-11-28T17:10:22.907204+0000","last_clean":"2023-11-28T17:10:22.907204+0000","last_became_active":"2023-11-28T17:10:20.333164+0000","last_became_peered":"2023-11-28T17:10:20.333164+0000","last_unstale":"2023-11-28T17:10:22.907204+0000","last_undegraded":"2023-11-28T17:10:22.907204+0000","last_fullsized":"2023-11-28T17:10:22.907204+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:36:40.869830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.328311+0000","last_change":"2023-11-28T17:10:20.347607+0000","last_active":"2023-11-28T17:10:21.328311+0000","last_peered":"2023-11-28T17:10:21.328311+0000","last_clean":"2023-11-28T17:10:21.328311+0000","last_became_active":"2023-11-28T17:10:20.347324+0000","last_became_peered":"2023-11-28T17:10:20.347324+0000","last_unstale":"2023-11-28T17:10:21.328311+0000","last_undegraded":"2023-11-28T17:10:21.328311+0000","last_fullsized":"2023-11-28T17:10:21.328311+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:42:01.288021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.326536+0000","last_change":"2023-11-28T17:10:20.354614+0000","last_active":"2023-11-28T17:10:21.326536+0000","last_peered":"2023-11-28T17:10:21.326536+0000","last_clean":"2023-11-28T17:10:21.326536+0000","last_became_active":"2023-11-28T17:10:20.354311+0000","last_became_peered":"2023-11-28T17:10:20.354311+0000","last_unstale":"2023-11-28T17:10:21.326536+0000","last_undegraded":"2023-11-28T17:10:21.326536+0000","last_fullsized":"2023-11-28T17:10:21.326536+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:57:53.248420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.326490+0000","last_change":"2023-11-28T17:10:20.342005+0000","last_active":"2023-11-28T17:10:21.326490+0000","last_peered":"2023-11-28T17:10:21.326490+0000","last_clean":"2023-11-28T17:10:21.326490+0000","last_became_active":"2023-11-28T17:10:20.341521+0000","last_became_peered":"2023-11-28T17:10:20.341521+0000","last_unstale":"2023-11-28T17:10:21.326490+0000","last_undegraded":"2023-11-28T17:10:21.326490+0000","last_fullsized":"2023-11-28T17:10:21.326490+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:49:10.586414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.734236+0000","last_change":"2023-11-28T17:10:20.351477+0000","last_active":"2023-11-28T17:10:21.734236+0000","last_peered":"2023-11-28T17:10:21.734236+0000","last_clean":"2023-11-28T17:10:21.734236+0000","last_became_active":"2023-11-28T17:10:20.351170+0000","last_became_peered":"2023-11-28T17:10:20.351170+0000","last_unstale":"2023-11-28T17:10:21.734236+0000","last_undegraded":"2023-11-28T17:10:21.734236+0000","last_fullsized":"2023-11-28T17:10:21.734236+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:01:48.893249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.908793+0000","last_change":"2023-11-28T17:10:20.344664+0000","last_active":"2023-11-28T17:10:22.908793+0000","last_peered":"2023-11-28T17:10:22.908793+0000","last_clean":"2023-11-28T17:10:22.908793+0000","last_became_active":"2023-11-28T17:10:20.344409+0000","last_became_peered":"2023-11-28T17:10:20.344409+0000","last_unstale":"2023-11-28T17:10:22.908793+0000","last_undegraded":"2023-11-28T17:10:22.908793+0000","last_fullsized":"2023-11-28T17:10:22.908793+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:12:51.144707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.330352+0000","last_change":"2023-11-28T17:10:20.347550+0000","last_active":"2023-11-28T17:10:21.330352+0000","last_peered":"2023-11-28T17:10:21.330352+0000","last_clean":"2023-11-28T17:10:21.330352+0000","last_became_active":"2023-11-28T17:10:20.346923+0000","last_became_peered":"2023-11-28T17:10:20.346923+0000","last_unstale":"2023-11-28T17:10:21.330352+0000","last_undegraded":"2023-11-28T17:10:21.330352+0000","last_fullsized":"2023-11-28T17:10:21.330352+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:13:42.050028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.733371+0000","last_change":"2023-11-28T17:10:20.344372+0000","last_active":"2023-11-28T17:10:21.733371+0000","last_peered":"2023-11-28T17:10:21.733371+0000","last_clean":"2023-11-28T17:10:21.733371+0000","last_became_active":"2023-11-28T17:10:20.344110+0000","last_became_peered":"2023-11-28T17:10:20.344110+0000","last_unstale":"2023-11-28T17:10:21.733371+0000","last_undegraded":"2023-11-28T17:10:21.733371+0000","last_fullsized":"2023-11-28T17:10:21.733371+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:39:13.984416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.329949+0000","last_change":"2023-11-28T17:10:20.350006+0000","last_active":"2023-11-28T17:10:21.329949+0000","last_peered":"2023-11-28T17:10:21.329949+0000","last_clean":"2023-11-28T17:10:21.329949+0000","last_became_active":"2023-11-28T17:10:20.349704+0000","last_became_peered":"2023-11-28T17:10:20.349704+0000","last_unstale":"2023-11-28T17:10:21.329949+0000","last_undegraded":"2023-11-28T17:10:21.329949+0000","last_fullsized":"2023-11-28T17:10:21.329949+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:54:33.890435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.908586+0000","last_change":"2023-11-28T17:10:20.329453+0000","last_active":"2023-11-28T17:10:22.908586+0000","last_peered":"2023-11-28T17:10:22.908586+0000","last_clean":"2023-11-28T17:10:22.908586+0000","last_became_active":"2023-11-28T17:10:20.320413+0000","last_became_peered":"2023-11-28T17:10:20.320413+0000","last_unstale":"2023-11-28T17:10:22.908586+0000","last_undegraded":"2023-11-28T17:10:22.908586+0000","last_fullsized":"2023-11-28T17:10:22.908586+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:26:00.875011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.909209+0000","last_change":"2023-11-28T17:10:20.336990+0000","last_active":"2023-11-28T17:10:22.909209+0000","last_peered":"2023-11-28T17:10:22.909209+0000","last_clean":"2023-11-28T17:10:22.909209+0000","last_became_active":"2023-11-28T17:10:20.336835+0000","last_became_peered":"2023-11-28T17:10:20.336835+0000","last_unstale":"2023-11-28T17:10:22.909209+0000","last_undegraded":"2023-11-28T17:10:22.909209+0000","last_fullsized":"2023-11-28T17:10:22.909209+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:01.473881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.542648+0000","last_change":"2023-11-28T17:10:20.346142+0000","last_active":"2023-11-28T17:10:21.542648+0000","last_peered":"2023-11-28T17:10:21.542648+0000","last_clean":"2023-11-28T17:10:21.542648+0000","last_became_active":"2023-11-28T17:10:20.345406+0000","last_became_peered":"2023-11-28T17:10:20.345406+0000","last_unstale":"2023-11-28T17:10:21.542648+0000","last_undegraded":"2023-11-28T17:10:21.542648+0000","last_fullsized":"2023-11-28T17:10:21.542648+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:39.208647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.330996+0000","last_change":"2023-11-28T17:10:20.355193+0000","last_active":"2023-11-28T17:10:21.330996+0000","last_peered":"2023-11-28T17:10:21.330996+0000","last_clean":"2023-11-28T17:10:21.330996+0000","last_became_active":"2023-11-28T17:10:20.354863+0000","last_became_peered":"2023-11-28T17:10:20.354863+0000","last_unstale":"2023-11-28T17:10:21.330996+0000","last_undegraded":"2023-11-28T17:10:21.330996+0000","last_fullsized":"2023-11-28T17:10:21.330996+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:48:55.325689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.908097+0000","last_change":"2023-11-28T17:10:20.330513+0000","last_active":"2023-11-28T17:10:22.908097+0000","last_peered":"2023-11-28T17:10:22.908097+0000","last_clean":"2023-11-28T17:10:22.908097+0000","last_became_active":"2023-11-28T17:10:20.329815+0000","last_became_peered":"2023-11-28T17:10:20.329815+0000","last_unstale":"2023-11-28T17:10:22.908097+0000","last_undegraded":"2023-11-28T17:10:22.908097+0000","last_fullsized":"2023-11-28T17:10:22.908097+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:39.208647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.907550+0000","last_change":"2023-11-28T17:10:20.321789+0000","last_active":"2023-11-28T17:10:22.907550+0000","last_peered":"2023-11-28T17:10:22.907550+0000","last_clean":"2023-11-28T17:10:22.907550+0000","last_became_active":"2023-11-28T17:10:20.321488+0000","last_became_peered":"2023-11-28T17:10:20.321488+0000","last_unstale":"2023-11-28T17:10:22.907550+0000","last_undegraded":"2023-11-28T17:10:22.907550+0000","last_fullsized":"2023-11-28T17:10:22.907550+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:56:24.407820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.330580+0000","last_change":"2023-11-28T17:10:20.349037+0000","last_active":"2023-11-28T17:10:21.330580+0000","last_peered":"2023-11-28T17:10:21.330580+0000","last_clean":"2023-11-28T17:10:21.330580+0000","last_became_active":"2023-11-28T17:10:20.348746+0000","last_became_peered":"2023-11-28T17:10:20.348746+0000","last_unstale":"2023-11-28T17:10:21.330580+0000","last_undegraded":"2023-11-28T17:10:21.330580+0000","last_fullsized":"2023-11-28T17:10:21.330580+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:39.208647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.326179+0000","last_change":"2023-11-28T17:10:20.342172+0000","last_active":"2023-11-28T17:10:21.326179+0000","last_peered":"2023-11-28T17:10:21.326179+0000","last_clean":"2023-11-28T17:10:21.326179+0000","last_became_active":"2023-11-28T17:10:20.341747+0000","last_became_peered":"2023-11-28T17:10:20.341747+0000","last_unstale":"2023-11-28T17:10:21.326179+0000","last_undegraded":"2023-11-28T17:10:21.326179+0000","last_fullsized":"2023-11-28T17:10:21.326179+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:53.024497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.734273+0000","last_change":"2023-11-28T17:10:20.343917+0000","last_active":"2023-11-28T17:10:21.734273+0000","last_peered":"2023-11-28T17:10:21.734273+0000","last_clean":"2023-11-28T17:10:21.734273+0000","last_became_active":"2023-11-28T17:10:20.343622+0000","last_became_peered":"2023-11-28T17:10:20.343622+0000","last_unstale":"2023-11-28T17:10:21.734273+0000","last_undegraded":"2023-11-28T17:10:21.734273+0000","last_fullsized":"2023-11-28T17:10:21.734273+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:36:40.869830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.345118+0000","last_change":"2023-11-28T17:10:19.295420+0000","last_active":"2023-11-28T17:10:20.345118+0000","last_peered":"2023-11-28T17:10:20.345118+0000","last_clean":"2023-11-28T17:10:20.345118+0000","last_became_active":"2023-11-28T17:10:19.294612+0000","last_became_peered":"2023-11-28T17:10:19.294612+0000","last_unstale":"2023-11-28T17:10:20.345118+0000","last_undegraded":"2023-11-28T17:10:20.345118+0000","last_fullsized":"2023-11-28T17:10:20.345118+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:40:33.669514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.331418+0000","last_change":"2023-11-28T17:10:20.344521+0000","last_active":"2023-11-28T17:10:21.331418+0000","last_peered":"2023-11-28T17:10:21.331418+0000","last_clean":"2023-11-28T17:10:21.331418+0000","last_became_active":"2023-11-28T17:10:20.344176+0000","last_became_peered":"2023-11-28T17:10:20.344176+0000","last_unstale":"2023-11-28T17:10:21.331418+0000","last_undegraded":"2023-11-28T17:10:21.331418+0000","last_fullsized":"2023-11-28T17:10:21.331418+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:14:50.050841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.352249+0000","last_change":"2023-11-28T17:10:19.306768+0000","last_active":"2023-11-28T17:10:20.352249+0000","last_peered":"2023-11-28T17:10:20.352249+0000","last_clean":"2023-11-28T17:10:20.352249+0000","last_became_active":"2023-11-28T17:10:19.306492+0000","last_became_peered":"2023-11-28T17:10:19.306492+0000","last_unstale":"2023-11-28T17:10:20.352249+0000","last_undegraded":"2023-11-28T17:10:20.352249+0000","last_fullsized":"2023-11-28T17:10:20.352249+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:10:03.281902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.330939+0000","last_change":"2023-11-28T17:10:20.347354+0000","last_active":"2023-11-28T17:10:21.330939+0000","last_peered":"2023-11-28T17:10:21.330939+0000","last_clean":"2023-11-28T17:10:21.330939+0000","last_became_active":"2023-11-28T17:10:20.347057+0000","last_became_peered":"2023-11-28T17:10:20.347057+0000","last_unstale":"2023-11-28T17:10:21.330939+0000","last_undegraded":"2023-11-28T17:10:21.330939+0000","last_fullsized":"2023-11-28T17:10:21.330939+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:12:51.144707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.347118+0000","last_change":"2023-11-28T17:10:19.301614+0000","last_active":"2023-11-28T17:10:20.347118+0000","last_peered":"2023-11-28T17:10:20.347118+0000","last_clean":"2023-11-28T17:10:20.347118+0000","last_became_active":"2023-11-28T17:10:19.301307+0000","last_became_peered":"2023-11-28T17:10:19.301307+0000","last_unstale":"2023-11-28T17:10:20.347118+0000","last_undegraded":"2023-11-28T17:10:20.347118+0000","last_fullsized":"2023-11-28T17:10:20.347118+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:54:19.482117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.908665+0000","last_change":"2023-11-28T17:10:20.349618+0000","last_active":"2023-11-28T17:10:22.908665+0000","last_peered":"2023-11-28T17:10:22.908665+0000","last_clean":"2023-11-28T17:10:22.908665+0000","last_became_active":"2023-11-28T17:10:20.349352+0000","last_became_peered":"2023-11-28T17:10:20.349352+0000","last_unstale":"2023-11-28T17:10:22.908665+0000","last_undegraded":"2023-11-28T17:10:22.908665+0000","last_fullsized":"2023-11-28T17:10:22.908665+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:14:50.050841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.355263+0000","last_change":"2023-11-28T17:10:19.301144+0000","last_active":"2023-11-28T17:10:20.355263+0000","last_peered":"2023-11-28T17:10:20.355263+0000","last_clean":"2023-11-28T17:10:20.355263+0000","last_became_active":"2023-11-28T17:10:19.300801+0000","last_became_peered":"2023-11-28T17:10:19.300801+0000","last_unstale":"2023-11-28T17:10:20.355263+0000","last_undegraded":"2023-11-28T17:10:20.355263+0000","last_fullsized":"2023-11-28T17:10:20.355263+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:01:47.887732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.908330+0000","last_change":"2023-11-28T17:10:20.336507+0000","last_active":"2023-11-28T17:10:22.908330+0000","last_peered":"2023-11-28T17:10:22.908330+0000","last_clean":"2023-11-28T17:10:22.908330+0000","last_became_active":"2023-11-28T17:10:20.335510+0000","last_became_peered":"2023-11-28T17:10:20.335510+0000","last_unstale":"2023-11-28T17:10:22.908330+0000","last_undegraded":"2023-11-28T17:10:22.908330+0000","last_fullsized":"2023-11-28T17:10:22.908330+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:59:45.873967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.353902+0000","last_change":"2023-11-28T17:10:19.295030+0000","last_active":"2023-11-28T17:10:20.353902+0000","last_peered":"2023-11-28T17:10:20.353902+0000","last_clean":"2023-11-28T17:10:20.353902+0000","last_became_active":"2023-11-28T17:10:19.294679+0000","last_became_peered":"2023-11-28T17:10:19.294679+0000","last_unstale":"2023-11-28T17:10:20.353902+0000","last_undegraded":"2023-11-28T17:10:20.353902+0000","last_fullsized":"2023-11-28T17:10:20.353902+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:58:45.955545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.330341+0000","last_change":"2023-11-28T17:10:20.348452+0000","last_active":"2023-11-28T17:10:21.330341+0000","last_peered":"2023-11-28T17:10:21.330341+0000","last_clean":"2023-11-28T17:10:21.330341+0000","last_became_active":"2023-11-28T17:10:20.348164+0000","last_became_peered":"2023-11-28T17:10:20.348164+0000","last_unstale":"2023-11-28T17:10:21.330341+0000","last_undegraded":"2023-11-28T17:10:21.330341+0000","last_fullsized":"2023-11-28T17:10:21.330341+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:56:24.407820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.352935+0000","last_change":"2023-11-28T17:10:19.298673+0000","last_active":"2023-11-28T17:10:20.352935+0000","last_peered":"2023-11-28T17:10:20.352935+0000","last_clean":"2023-11-28T17:10:20.352935+0000","last_became_active":"2023-11-28T17:10:19.297133+0000","last_became_peered":"2023-11-28T17:10:19.297133+0000","last_unstale":"2023-11-28T17:10:20.352935+0000","last_undegraded":"2023-11-28T17:10:20.352935+0000","last_fullsized":"2023-11-28T17:10:20.352935+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:08:23.387134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.907552+0000","last_change":"2023-11-28T17:10:20.336848+0000","last_active":"2023-11-28T17:10:22.907552+0000","last_peered":"2023-11-28T17:10:22.907552+0000","last_clean":"2023-11-28T17:10:22.907552+0000","last_became_active":"2023-11-28T17:10:20.335659+0000","last_became_peered":"2023-11-28T17:10:20.335659+0000","last_unstale":"2023-11-28T17:10:22.907552+0000","last_undegraded":"2023-11-28T17:10:22.907552+0000","last_fullsized":"2023-11-28T17:10:22.907552+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:56:16.998208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.322363+0000","last_change":"2023-11-28T17:10:19.304071+0000","last_active":"2023-11-28T17:10:20.322363+0000","last_peered":"2023-11-28T17:10:20.322363+0000","last_clean":"2023-11-28T17:10:20.322363+0000","last_became_active":"2023-11-28T17:10:19.303800+0000","last_became_peered":"2023-11-28T17:10:19.303800+0000","last_unstale":"2023-11-28T17:10:20.322363+0000","last_undegraded":"2023-11-28T17:10:20.322363+0000","last_fullsized":"2023-11-28T17:10:20.322363+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:26:44.987761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.907746+0000","last_change":"2023-11-28T17:10:20.328251+0000","last_active":"2023-11-28T17:10:22.907746+0000","last_peered":"2023-11-28T17:10:22.907746+0000","last_clean":"2023-11-28T17:10:22.907746+0000","last_became_active":"2023-11-28T17:10:20.323497+0000","last_became_peered":"2023-11-28T17:10:20.323497+0000","last_unstale":"2023-11-28T17:10:22.907746+0000","last_undegraded":"2023-11-28T17:10:22.907746+0000","last_fullsized":"2023-11-28T17:10:22.907746+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:56:24.407820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.332787+0000","last_change":"2023-11-28T17:10:19.295229+0000","last_active":"2023-11-28T17:10:20.332787+0000","last_peered":"2023-11-28T17:10:20.332787+0000","last_clean":"2023-11-28T17:10:20.332787+0000","last_became_active":"2023-11-28T17:10:19.294927+0000","last_became_peered":"2023-11-28T17:10:19.294927+0000","last_unstale":"2023-11-28T17:10:20.332787+0000","last_undegraded":"2023-11-28T17:10:20.332787+0000","last_fullsized":"2023-11-28T17:10:20.332787+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:16.378585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.327484+0000","last_change":"2023-11-28T17:10:20.354523+0000","last_active":"2023-11-28T17:10:21.327484+0000","last_peered":"2023-11-28T17:10:21.327484+0000","last_clean":"2023-11-28T17:10:21.327484+0000","last_became_active":"2023-11-28T17:10:20.354166+0000","last_became_peered":"2023-11-28T17:10:20.354166+0000","last_unstale":"2023-11-28T17:10:21.327484+0000","last_undegraded":"2023-11-28T17:10:21.327484+0000","last_fullsized":"2023-11-28T17:10:21.327484+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:13:06.560017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.325784+0000","last_change":"2023-11-28T17:10:19.300456+0000","last_active":"2023-11-28T17:10:20.325784+0000","last_peered":"2023-11-28T17:10:20.325784+0000","last_clean":"2023-11-28T17:10:20.325784+0000","last_became_active":"2023-11-28T17:10:19.300191+0000","last_became_peered":"2023-11-28T17:10:19.300191+0000","last_unstale":"2023-11-28T17:10:20.325784+0000","last_undegraded":"2023-11-28T17:10:20.325784+0000","last_fullsized":"2023-11-28T17:10:20.325784+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:54:32.884918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.908918+0000","last_change":"2023-11-28T17:10:20.341469+0000","last_active":"2023-11-28T17:10:22.908918+0000","last_peered":"2023-11-28T17:10:22.908918+0000","last_clean":"2023-11-28T17:10:22.908918+0000","last_became_active":"2023-11-28T17:10:20.340178+0000","last_became_peered":"2023-11-28T17:10:20.340178+0000","last_unstale":"2023-11-28T17:10:22.908918+0000","last_undegraded":"2023-11-28T17:10:22.908918+0000","last_fullsized":"2023-11-28T17:10:22.908918+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:17:42.113807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.324247+0000","last_change":"2023-11-28T17:10:19.302405+0000","last_active":"2023-11-28T17:10:20.324247+0000","last_peered":"2023-11-28T17:10:20.324247+0000","last_clean":"2023-11-28T17:10:20.324247+0000","last_became_active":"2023-11-28T17:10:19.301927+0000","last_became_peered":"2023-11-28T17:10:19.301927+0000","last_unstale":"2023-11-28T17:10:20.324247+0000","last_undegraded":"2023-11-28T17:10:20.324247+0000","last_fullsized":"2023-11-28T17:10:20.324247+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:39:12.978898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.908855+0000","last_change":"2023-11-28T17:10:20.355399+0000","last_active":"2023-11-28T17:10:22.908855+0000","last_peered":"2023-11-28T17:10:22.908855+0000","last_clean":"2023-11-28T17:10:22.908855+0000","last_became_active":"2023-11-28T17:10:20.354885+0000","last_became_peered":"2023-11-28T17:10:20.354885+0000","last_unstale":"2023-11-28T17:10:22.908855+0000","last_undegraded":"2023-11-28T17:10:22.908855+0000","last_fullsized":"2023-11-28T17:10:22.908855+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:17:47.647425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.347403+0000","last_change":"2023-11-28T17:10:19.291826+0000","last_active":"2023-11-28T17:10:20.347403+0000","last_peered":"2023-11-28T17:10:20.347403+0000","last_clean":"2023-11-28T17:10:20.347403+0000","last_became_active":"2023-11-28T17:10:19.291533+0000","last_became_peered":"2023-11-28T17:10:19.291533+0000","last_unstale":"2023-11-28T17:10:20.347403+0000","last_undegraded":"2023-11-28T17:10:20.347403+0000","last_fullsized":"2023-11-28T17:10:20.347403+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:22:27.085940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.326864+0000","last_change":"2023-11-28T17:10:20.333788+0000","last_active":"2023-11-28T17:10:21.326864+0000","last_peered":"2023-11-28T17:10:21.326864+0000","last_clean":"2023-11-28T17:10:21.326864+0000","last_became_active":"2023-11-28T17:10:20.331714+0000","last_became_peered":"2023-11-28T17:10:20.331714+0000","last_unstale":"2023-11-28T17:10:21.326864+0000","last_undegraded":"2023-11-28T17:10:21.326864+0000","last_fullsized":"2023-11-28T17:10:21.326864+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:03.379242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.358258+0000","last_change":"2023-11-28T17:10:19.305127+0000","last_active":"2023-11-28T17:10:20.358258+0000","last_peered":"2023-11-28T17:10:20.358258+0000","last_clean":"2023-11-28T17:10:20.358258+0000","last_became_active":"2023-11-28T17:10:19.304819+0000","last_became_peered":"2023-11-28T17:10:19.304819+0000","last_unstale":"2023-11-28T17:10:20.358258+0000","last_undegraded":"2023-11-28T17:10:20.358258+0000","last_fullsized":"2023-11-28T17:10:20.358258+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:03:36.600341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.733369+0000","last_change":"2023-11-28T17:10:20.338253+0000","last_active":"2023-11-28T17:10:21.733369+0000","last_peered":"2023-11-28T17:10:21.733369+0000","last_clean":"2023-11-28T17:10:21.733369+0000","last_became_active":"2023-11-28T17:10:20.337998+0000","last_became_peered":"2023-11-28T17:10:20.337998+0000","last_unstale":"2023-11-28T17:10:21.733369+0000","last_undegraded":"2023-11-28T17:10:21.733369+0000","last_fullsized":"2023-11-28T17:10:21.733369+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:15:02.896529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.353848+0000","last_change":"2023-11-28T17:10:19.289570+0000","last_active":"2023-11-28T17:10:20.353848+0000","last_peered":"2023-11-28T17:10:20.353848+0000","last_clean":"2023-11-28T17:10:20.353848+0000","last_became_active":"2023-11-28T17:10:19.289310+0000","last_became_peered":"2023-11-28T17:10:19.289310+0000","last_unstale":"2023-11-28T17:10:20.353848+0000","last_undegraded":"2023-11-28T17:10:20.353848+0000","last_fullsized":"2023-11-28T17:10:20.353848+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:10:03.281902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.908167+0000","last_change":"2023-11-28T17:10:20.354627+0000","last_active":"2023-11-28T17:10:22.908167+0000","last_peered":"2023-11-28T17:10:22.908167+0000","last_clean":"2023-11-28T17:10:22.908167+0000","last_became_active":"2023-11-28T17:10:20.354182+0000","last_became_peered":"2023-11-28T17:10:20.354182+0000","last_unstale":"2023-11-28T17:10:22.908167+0000","last_undegraded":"2023-11-28T17:10:22.908167+0000","last_fullsized":"2023-11-28T17:10:22.908167+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:02:39.208647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.305964+0000","last_change":"2023-11-28T17:10:19.303515+0000","last_active":"2023-11-28T17:10:20.305964+0000","last_peered":"2023-11-28T17:10:20.305964+0000","last_clean":"2023-11-28T17:10:20.305964+0000","last_became_active":"2023-11-28T17:10:19.303217+0000","last_became_peered":"2023-11-28T17:10:19.303217+0000","last_unstale":"2023-11-28T17:10:20.305964+0000","last_undegraded":"2023-11-28T17:10:20.305964+0000","last_fullsized":"2023-11-28T17:10:20.305964+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:10:24.406709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.330232+0000","last_change":"2023-11-28T17:10:20.349875+0000","last_active":"2023-11-28T17:10:21.330232+0000","last_peered":"2023-11-28T17:10:21.330232+0000","last_clean":"2023-11-28T17:10:21.330232+0000","last_became_active":"2023-11-28T17:10:20.349389+0000","last_became_peered":"2023-11-28T17:10:20.349389+0000","last_unstale":"2023-11-28T17:10:21.330232+0000","last_undegraded":"2023-11-28T17:10:21.330232+0000","last_fullsized":"2023-11-28T17:10:21.330232+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:36:40.869830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.306563+0000","last_change":"2023-11-28T17:10:19.294783+0000","last_active":"2023-11-28T17:10:20.306563+0000","last_peered":"2023-11-28T17:10:20.306563+0000","last_clean":"2023-11-28T17:10:20.306563+0000","last_became_active":"2023-11-28T17:10:19.294480+0000","last_became_peered":"2023-11-28T17:10:19.294480+0000","last_unstale":"2023-11-28T17:10:20.306563+0000","last_undegraded":"2023-11-28T17:10:20.306563+0000","last_fullsized":"2023-11-28T17:10:20.306563+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:16.378585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.326255+0000","last_change":"2023-11-28T17:10:20.352309+0000","last_active":"2023-11-28T17:10:21.326255+0000","last_peered":"2023-11-28T17:10:21.326255+0000","last_clean":"2023-11-28T17:10:21.326255+0000","last_became_active":"2023-11-28T17:10:20.352143+0000","last_became_peered":"2023-11-28T17:10:20.352143+0000","last_unstale":"2023-11-28T17:10:21.326255+0000","last_undegraded":"2023-11-28T17:10:21.326255+0000","last_fullsized":"2023-11-28T17:10:21.326255+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:11:00.211175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.348749+0000","last_change":"2023-11-28T17:10:19.302973+0000","last_active":"2023-11-28T17:10:20.348749+0000","last_peered":"2023-11-28T17:10:20.348749+0000","last_clean":"2023-11-28T17:10:20.348749+0000","last_became_active":"2023-11-28T17:10:19.302711+0000","last_became_peered":"2023-11-28T17:10:19.302711+0000","last_unstale":"2023-11-28T17:10:20.348749+0000","last_undegraded":"2023-11-28T17:10:20.348749+0000","last_fullsized":"2023-11-28T17:10:20.348749+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:45:31.526535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.327896+0000","last_change":"2023-11-28T17:10:20.332013+0000","last_active":"2023-11-28T17:10:21.327896+0000","last_peered":"2023-11-28T17:10:21.327896+0000","last_clean":"2023-11-28T17:10:21.327896+0000","last_became_active":"2023-11-28T17:10:20.331669+0000","last_became_peered":"2023-11-28T17:10:20.331669+0000","last_unstale":"2023-11-28T17:10:21.327896+0000","last_undegraded":"2023-11-28T17:10:21.327896+0000","last_fullsized":"2023-11-28T17:10:21.327896+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:32:48.410249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.348885+0000","last_change":"2023-11-28T17:10:19.306286+0000","last_active":"2023-11-28T17:10:20.348885+0000","last_peered":"2023-11-28T17:10:20.348885+0000","last_clean":"2023-11-28T17:10:20.348885+0000","last_became_active":"2023-11-28T17:10:19.305927+0000","last_became_peered":"2023-11-28T17:10:19.305927+0000","last_unstale":"2023-11-28T17:10:20.348885+0000","last_undegraded":"2023-11-28T17:10:20.348885+0000","last_fullsized":"2023-11-28T17:10:20.348885+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:13:10.313820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.909938+0000","last_change":"2023-11-28T17:10:20.346485+0000","last_active":"2023-11-28T17:10:22.909938+0000","last_peered":"2023-11-28T17:10:22.909938+0000","last_clean":"2023-11-28T17:10:22.909938+0000","last_became_active":"2023-11-28T17:10:20.346208+0000","last_became_peered":"2023-11-28T17:10:20.346208+0000","last_unstale":"2023-11-28T17:10:22.909938+0000","last_undegraded":"2023-11-28T17:10:22.909938+0000","last_fullsized":"2023-11-28T17:10:22.909938+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:10:25.412226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.349866+0000","last_change":"2023-11-28T17:10:19.290789+0000","last_active":"2023-11-28T17:10:20.349866+0000","last_peered":"2023-11-28T17:10:20.349866+0000","last_clean":"2023-11-28T17:10:20.349866+0000","last_became_active":"2023-11-28T17:10:19.290545+0000","last_became_peered":"2023-11-28T17:10:19.290545+0000","last_unstale":"2023-11-28T17:10:20.349866+0000","last_undegraded":"2023-11-28T17:10:20.349866+0000","last_fullsized":"2023-11-28T17:10:20.349866+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:15:01.891011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.542500+0000","last_change":"2023-11-28T17:10:20.339174+0000","last_active":"2023-11-28T17:10:21.542500+0000","last_peered":"2023-11-28T17:10:21.542500+0000","last_clean":"2023-11-28T17:10:21.542500+0000","last_became_active":"2023-11-28T17:10:20.338867+0000","last_became_peered":"2023-11-28T17:10:20.338867+0000","last_unstale":"2023-11-28T17:10:21.542500+0000","last_undegraded":"2023-11-28T17:10:21.542500+0000","last_fullsized":"2023-11-28T17:10:21.542500+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:56:24.407820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.336223+0000","last_change":"2023-11-28T17:10:19.297053+0000","last_active":"2023-11-28T17:10:20.336223+0000","last_peered":"2023-11-28T17:10:20.336223+0000","last_clean":"2023-11-28T17:10:20.336223+0000","last_became_active":"2023-11-28T17:10:19.296490+0000","last_became_peered":"2023-11-28T17:10:19.296490+0000","last_unstale":"2023-11-28T17:10:20.336223+0000","last_undegraded":"2023-11-28T17:10:20.336223+0000","last_fullsized":"2023-11-28T17:10:20.336223+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:01:42.339512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.326833+0000","last_change":"2023-11-28T17:10:20.351638+0000","last_active":"2023-11-28T17:10:21.326833+0000","last_peered":"2023-11-28T17:10:21.326833+0000","last_clean":"2023-11-28T17:10:21.326833+0000","last_became_active":"2023-11-28T17:10:20.350952+0000","last_became_peered":"2023-11-28T17:10:20.350952+0000","last_unstale":"2023-11-28T17:10:21.326833+0000","last_undegraded":"2023-11-28T17:10:21.326833+0000","last_fullsized":"2023-11-28T17:10:21.326833+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:17.384103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.315232+0000","last_change":"2023-11-28T17:10:19.305800+0000","last_active":"2023-11-28T17:10:20.315232+0000","last_peered":"2023-11-28T17:10:20.315232+0000","last_clean":"2023-11-28T17:10:20.315232+0000","last_became_active":"2023-11-28T17:10:19.305482+0000","last_became_peered":"2023-11-28T17:10:19.305482+0000","last_unstale":"2023-11-28T17:10:20.315232+0000","last_undegraded":"2023-11-28T17:10:20.315232+0000","last_fullsized":"2023-11-28T17:10:20.315232+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:54:32.884918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.733568+0000","last_change":"2023-11-28T17:10:20.349022+0000","last_active":"2023-11-28T17:10:21.733568+0000","last_peered":"2023-11-28T17:10:21.733568+0000","last_clean":"2023-11-28T17:10:21.733568+0000","last_became_active":"2023-11-28T17:10:20.348627+0000","last_became_peered":"2023-11-28T17:10:20.348627+0000","last_unstale":"2023-11-28T17:10:21.733568+0000","last_undegraded":"2023-11-28T17:10:21.733568+0000","last_fullsized":"2023-11-28T17:10:21.733568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:42:59.695037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.345940+0000","last_change":"2023-11-28T17:10:19.303455+0000","last_active":"2023-11-28T17:10:20.345940+0000","last_peered":"2023-11-28T17:10:20.345940+0000","last_clean":"2023-11-28T17:10:20.345940+0000","last_became_active":"2023-11-28T17:10:19.303189+0000","last_became_peered":"2023-11-28T17:10:19.303189+0000","last_unstale":"2023-11-28T17:10:20.345940+0000","last_undegraded":"2023-11-28T17:10:20.345940+0000","last_fullsized":"2023-11-28T17:10:20.345940+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:58:45.955545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.542601+0000","last_change":"2023-11-28T17:10:20.340082+0000","last_active":"2023-11-28T17:10:21.542601+0000","last_peered":"2023-11-28T17:10:21.542601+0000","last_clean":"2023-11-28T17:10:21.542601+0000","last_became_active":"2023-11-28T17:10:20.339647+0000","last_became_peered":"2023-11-28T17:10:20.339647+0000","last_unstale":"2023-11-28T17:10:21.542601+0000","last_undegraded":"2023-11-28T17:10:21.542601+0000","last_fullsized":"2023-11-28T17:10:21.542601+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:13:42.050028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.332895+0000","last_change":"2023-11-28T17:10:19.291224+0000","last_active":"2023-11-28T17:10:20.332895+0000","last_peered":"2023-11-28T17:10:20.332895+0000","last_clean":"2023-11-28T17:10:20.332895+0000","last_became_active":"2023-11-28T17:10:19.289292+0000","last_became_peered":"2023-11-28T17:10:19.289292+0000","last_unstale":"2023-11-28T17:10:20.332895+0000","last_undegraded":"2023-11-28T17:10:20.332895+0000","last_fullsized":"2023-11-28T17:10:20.332895+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:09:28.605640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.733317+0000","last_change":"2023-11-28T17:10:20.335913+0000","last_active":"2023-11-28T17:10:21.733317+0000","last_peered":"2023-11-28T17:10:21.733317+0000","last_clean":"2023-11-28T17:10:21.733317+0000","last_became_active":"2023-11-28T17:10:20.335451+0000","last_became_peered":"2023-11-28T17:10:20.335451+0000","last_unstale":"2023-11-28T17:10:21.733317+0000","last_undegraded":"2023-11-28T17:10:21.733317+0000","last_fullsized":"2023-11-28T17:10:21.733317+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:10:04.287419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.331348+0000","last_change":"2023-11-28T17:10:19.288811+0000","last_active":"2023-11-28T17:10:20.331348+0000","last_peered":"2023-11-28T17:10:20.331348+0000","last_clean":"2023-11-28T17:10:20.331348+0000","last_became_active":"2023-11-28T17:10:19.288560+0000","last_became_peered":"2023-11-28T17:10:19.288560+0000","last_unstale":"2023-11-28T17:10:20.331348+0000","last_undegraded":"2023-11-28T17:10:20.331348+0000","last_fullsized":"2023-11-28T17:10:20.331348+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:13:10.313820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.907844+0000","last_change":"2023-11-28T17:10:20.335858+0000","last_active":"2023-11-28T17:10:22.907844+0000","last_peered":"2023-11-28T17:10:22.907844+0000","last_clean":"2023-11-28T17:10:22.907844+0000","last_became_active":"2023-11-28T17:10:20.334075+0000","last_became_peered":"2023-11-28T17:10:20.334075+0000","last_unstale":"2023-11-28T17:10:22.907844+0000","last_undegraded":"2023-11-28T17:10:22.907844+0000","last_fullsized":"2023-11-28T17:10:22.907844+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:54:20.487635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.316902+0000","last_change":"2023-11-28T17:10:19.292580+0000","last_active":"2023-11-28T17:10:20.316902+0000","last_peered":"2023-11-28T17:10:20.316902+0000","last_clean":"2023-11-28T17:10:20.316902+0000","last_became_active":"2023-11-28T17:10:19.292338+0000","last_became_peered":"2023-11-28T17:10:19.292338+0000","last_unstale":"2023-11-28T17:10:20.316902+0000","last_undegraded":"2023-11-28T17:10:20.316902+0000","last_fullsized":"2023-11-28T17:10:20.316902+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:10:01.077968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.326769+0000","last_change":"2023-11-28T17:10:20.346077+0000","last_active":"2023-11-28T17:10:21.326769+0000","last_peered":"2023-11-28T17:10:21.326769+0000","last_clean":"2023-11-28T17:10:21.326769+0000","last_became_active":"2023-11-28T17:10:20.345735+0000","last_became_peered":"2023-11-28T17:10:20.345735+0000","last_unstale":"2023-11-28T17:10:21.326769+0000","last_undegraded":"2023-11-28T17:10:21.326769+0000","last_fullsized":"2023-11-28T17:10:21.326769+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:10:25.412226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.353040+0000","last_change":"2023-11-28T17:10:19.291370+0000","last_active":"2023-11-28T17:10:20.353040+0000","last_peered":"2023-11-28T17:10:20.353040+0000","last_clean":"2023-11-28T17:10:20.353040+0000","last_became_active":"2023-11-28T17:10:19.291024+0000","last_became_peered":"2023-11-28T17:10:19.291024+0000","last_unstale":"2023-11-28T17:10:20.353040+0000","last_undegraded":"2023-11-28T17:10:20.353040+0000","last_fullsized":"2023-11-28T17:10:20.353040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:33:00.424928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.328304+0000","last_change":"2023-11-28T17:10:20.332462+0000","last_active":"2023-11-28T17:10:21.328304+0000","last_peered":"2023-11-28T17:10:21.328304+0000","last_clean":"2023-11-28T17:10:21.328304+0000","last_became_active":"2023-11-28T17:10:20.332177+0000","last_became_peered":"2023-11-28T17:10:20.332177+0000","last_unstale":"2023-11-28T17:10:21.328304+0000","last_undegraded":"2023-11-28T17:10:21.328304+0000","last_fullsized":"2023-11-28T17:10:21.328304+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:39:41.746035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.324422+0000","last_change":"2023-11-28T17:10:19.298073+0000","last_active":"2023-11-28T17:10:20.324422+0000","last_peered":"2023-11-28T17:10:20.324422+0000","last_clean":"2023-11-28T17:10:20.324422+0000","last_became_active":"2023-11-28T17:10:19.297817+0000","last_became_peered":"2023-11-28T17:10:19.297817+0000","last_unstale":"2023-11-28T17:10:20.324422+0000","last_undegraded":"2023-11-28T17:10:20.324422+0000","last_fullsized":"2023-11-28T17:10:20.324422+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:42:58.689520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.326773+0000","last_change":"2023-11-28T17:10:20.339315+0000","last_active":"2023-11-28T17:10:21.326773+0000","last_peered":"2023-11-28T17:10:21.326773+0000","last_clean":"2023-11-28T17:10:21.326773+0000","last_became_active":"2023-11-28T17:10:20.338982+0000","last_became_peered":"2023-11-28T17:10:20.338982+0000","last_unstale":"2023-11-28T17:10:21.326773+0000","last_undegraded":"2023-11-28T17:10:21.326773+0000","last_fullsized":"2023-11-28T17:10:21.326773+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:03:37.605858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.331297+0000","last_change":"2023-11-28T17:10:19.298953+0000","last_active":"2023-11-28T17:10:20.331297+0000","last_peered":"2023-11-28T17:10:20.331297+0000","last_clean":"2023-11-28T17:10:20.331297+0000","last_became_active":"2023-11-28T17:10:19.298664+0000","last_became_peered":"2023-11-28T17:10:19.298664+0000","last_unstale":"2023-11-28T17:10:20.331297+0000","last_undegraded":"2023-11-28T17:10:20.331297+0000","last_fullsized":"2023-11-28T17:10:20.331297+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:16.378585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.908028+0000","last_change":"2023-11-28T17:10:20.334628+0000","last_active":"2023-11-28T17:10:22.908028+0000","last_peered":"2023-11-28T17:10:22.908028+0000","last_clean":"2023-11-28T17:10:22.908028+0000","last_became_active":"2023-11-28T17:10:20.334334+0000","last_became_peered":"2023-11-28T17:10:20.334334+0000","last_unstale":"2023-11-28T17:10:22.908028+0000","last_undegraded":"2023-11-28T17:10:22.908028+0000","last_fullsized":"2023-11-28T17:10:22.908028+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:13:42.050028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.331663+0000","last_change":"2023-11-28T17:10:19.294746+0000","last_active":"2023-11-28T17:10:20.331663+0000","last_peered":"2023-11-28T17:10:20.331663+0000","last_clean":"2023-11-28T17:10:20.331663+0000","last_became_active":"2023-11-28T17:10:19.294468+0000","last_became_peered":"2023-11-28T17:10:19.294468+0000","last_unstale":"2023-11-28T17:10:20.331663+0000","last_undegraded":"2023-11-28T17:10:20.331663+0000","last_fullsized":"2023-11-28T17:10:20.331663+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:08:23.387134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.908234+0000","last_change":"2023-11-28T17:10:20.345274+0000","last_active":"2023-11-28T17:10:22.908234+0000","last_peered":"2023-11-28T17:10:22.908234+0000","last_clean":"2023-11-28T17:10:22.908234+0000","last_became_active":"2023-11-28T17:10:20.345005+0000","last_became_peered":"2023-11-28T17:10:20.345005+0000","last_unstale":"2023-11-28T17:10:22.908234+0000","last_undegraded":"2023-11-28T17:10:22.908234+0000","last_fullsized":"2023-11-28T17:10:22.908234+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:12:51.144707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.324465+0000","last_change":"2023-11-28T17:10:19.295931+0000","last_active":"2023-11-28T17:10:20.324465+0000","last_peered":"2023-11-28T17:10:20.324465+0000","last_clean":"2023-11-28T17:10:20.324465+0000","last_became_active":"2023-11-28T17:10:19.291579+0000","last_became_peered":"2023-11-28T17:10:19.291579+0000","last_unstale":"2023-11-28T17:10:20.324465+0000","last_undegraded":"2023-11-28T17:10:20.324465+0000","last_fullsized":"2023-11-28T17:10:20.324465+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:59:44.868450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.907927+0000","last_change":"2023-11-28T17:10:20.343084+0000","last_active":"2023-11-28T17:10:22.907927+0000","last_peered":"2023-11-28T17:10:22.907927+0000","last_clean":"2023-11-28T17:10:22.907927+0000","last_became_active":"2023-11-28T17:10:20.342809+0000","last_became_peered":"2023-11-28T17:10:20.342809+0000","last_unstale":"2023-11-28T17:10:22.907927+0000","last_undegraded":"2023-11-28T17:10:22.907927+0000","last_fullsized":"2023-11-28T17:10:22.907927+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:13:42.050028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.337367+0000","last_change":"2023-11-28T17:10:19.287226+0000","last_active":"2023-11-28T17:10:20.337367+0000","last_peered":"2023-11-28T17:10:20.337367+0000","last_clean":"2023-11-28T17:10:20.337367+0000","last_became_active":"2023-11-28T17:10:19.286939+0000","last_became_peered":"2023-11-28T17:10:19.286939+0000","last_unstale":"2023-11-28T17:10:20.337367+0000","last_undegraded":"2023-11-28T17:10:20.337367+0000","last_fullsized":"2023-11-28T17:10:20.337367+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:34:16.145618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:21.326443+0000","last_change":"2023-11-28T17:10:20.328074+0000","last_active":"2023-11-28T17:10:21.326443+0000","last_peered":"2023-11-28T17:10:21.326443+0000","last_clean":"2023-11-28T17:10:21.326443+0000","last_became_active":"2023-11-28T17:10:20.327399+0000","last_became_peered":"2023-11-28T17:10:20.327399+0000","last_unstale":"2023-11-28T17:10:21.326443+0000","last_undegraded":"2023-11-28T17:10:21.326443+0000","last_fullsized":"2023-11-28T17:10:21.326443+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:24:38.855347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.306520+0000","last_change":"2023-11-28T17:10:19.296746+0000","last_active":"2023-11-28T17:10:20.306520+0000","last_peered":"2023-11-28T17:10:20.306520+0000","last_clean":"2023-11-28T17:10:20.306520+0000","last_became_active":"2023-11-28T17:10:19.296442+0000","last_became_peered":"2023-11-28T17:10:19.296442+0000","last_unstale":"2023-11-28T17:10:20.306520+0000","last_undegraded":"2023-11-28T17:10:20.306520+0000","last_fullsized":"2023-11-28T17:10:20.306520+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:03:36.600341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-28T17:10:22.909471+0000","last_change":"2023-11-28T17:10:20.333808+0000","last_active":"2023-11-28T17:10:22.909471+0000","last_peered":"2023-11-28T17:10:22.909471+0000","last_clean":"2023-11-28T17:10:22.909471+0000","last_became_active":"2023-11-28T17:10:20.333453+0000","last_became_peered":"2023-11-28T17:10:20.333453+0000","last_unstale":"2023-11-28T17:10:22.909471+0000","last_undegraded":"2023-11-28T17:10:22.909471+0000","last_fullsized":"2023-11-28T17:10:22.909471+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:19.256392+0000","last_clean_scrub_stamp":"2023-11-28T17:10:19.256392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:48:55.325689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.314720+0000","last_change":"2023-11-28T17:10:19.297204+0000","last_active":"2023-11-28T17:10:20.314720+0000","last_peered":"2023-11-28T17:10:20.314720+0000","last_clean":"2023-11-28T17:10:20.314720+0000","last_became_active":"2023-11-28T17:10:19.296821+0000","last_became_peered":"2023-11-28T17:10:19.296821+0000","last_unstale":"2023-11-28T17:10:20.314720+0000","last_undegraded":"2023-11-28T17:10:20.314720+0000","last_fullsized":"2023-11-28T17:10:20.314720+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:10:18.250875+0000","last_clean_scrub_stamp":"2023-11-28T17:10:18.250875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:26:26.557777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-28T17:10:20.329471+0000","last_change":"2023-11-28T17:08:51.392782+0000","last_active":"2023-11-28T17:10:20.329471+0000","last_peered":"2023-11-28T17:10:20.329471+0000","last_clean":"2023-11-28T17:10:20.329471+0000","last_became_active":"2023-11-28T17:08:51.390415+0000","last_became_peered":"2023-11-28T17:08:51.390415+0000","last_unstale":"2023-11-28T17:10:20.329471+0000","last_undegraded":"2023-11-28T17:10:20.329471+0000","last_fullsized":"2023-11-28T17:10:20.329471+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_clean_scrub_stamp":"2023-11-28T17:08:49.967682+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:03:32.638685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":389120,"data_stored":151435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.105,"5min":1.105,"15min":1.105},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":1.3220000000000001}]},{"osd":1,"last update":"Tue Nov 28 17:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.369}]},{"osd":2,"last update":"Tue Nov 28 17:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.212,"15min":1.212},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.999},{"interface":"front","average":{"1min":1.351,"5min":1.351,"15min":1.351},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":2.7029999999999998,"5min":2.7029999999999998,"15min":2.7029999999999998},"last":1.099}]},{"osd":3,"last update":"Tue Nov 28 17:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":1.659},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":1.51}]},{"osd":4,"last update":"Tue Nov 28 17:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.327,"15min":1.327},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":2.202,"5min":2.202,"15min":2.202},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.377,"5min":1.377,"15min":1.377},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":2.069,"5min":2.069,"15min":2.069},"last":1.8939999999999999}]},{"osd":5,"last update":"Tue Nov 28 17:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.476,"15min":1.476},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.4420000000000002,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":1.724},{"interface":"front","average":{"1min":1.573,"5min":1.573,"15min":1.573},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.7850000000000001,"5min":2.7850000000000001,"15min":2.7850000000000001},"last":1.7609999999999999}]},{"osd":6,"last update":"Tue Nov 28 17:09:48 2023","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.742,"15min":1.742},"min":{"1min":1.276,"5min":1.276,"15min":1.276},"max":{"1min":3.8010000000000002,"5min":3.8010000000000002,"15min":3.8010000000000002},"last":1.839},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":3.4049999999999998,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.9570000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":397312,"data_stored":153025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.228,"15min":1.228},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":4.3579999999999997,"5min":4.3579999999999997,"15min":4.3579999999999997},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.841,"5min":1.841,"15min":1.841},"last":0.95999999999999996}]},{"osd":1,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":4.1580000000000004,"5min":4.1580000000000004,"15min":4.1580000000000004},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":4.202,"5min":4.202,"15min":4.202},"last":1.3600000000000001}]},{"osd":2,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":1.194},{"interface":"front","average":{"1min":1.534,"5min":1.534,"15min":1.534},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":4.2750000000000004,"5min":4.2750000000000004,"15min":4.2750000000000004},"last":1.7090000000000001}]},{"osd":3,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.6339999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":1.8680000000000001}]},{"osd":4,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.363,"15min":1.363},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":2.1299999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.8200000000000001}]},{"osd":5,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.522,"15min":1.522},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":2.0289999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":3.0030000000000001,"5min":3.0030000000000001,"15min":3.0030000000000001},"last":1.9399999999999999}]},{"osd":7,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.514,"15min":1.514},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.524,"5min":2.524,"15min":2.524},"last":1.7789999999999999},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":2.4369999999999998,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.97}]}]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":742363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":1.044}]},{"osd":1,"last update":"Tue Nov 28 17:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.925,"5min":1.925,"15min":1.925},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.758,"5min":1.758,"15min":1.758},"last":1.6479999999999999}]},{"osd":2,"last update":"Tue Nov 28 17:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.774,"5min":1.774,"15min":1.774},"last":1.825},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.2070000000000001}]},{"osd":3,"last update":"Tue Nov 28 17:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.4180000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.504}]},{"osd":4,"last update":"Tue Nov 28 17:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.6179999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.107},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":1.9099999999999999},"last":1.458}]},{"osd":6,"last update":"Tue Nov 28 17:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.415,"15min":1.415},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.411},{"interface":"front","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":2.7759999999999998,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.6950000000000001}]},{"osd":7,"last update":"Tue Nov 28 17:09:51 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.036,"5min":2.036,"15min":2.036},"last":1.782},{"interface":"front","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.46,"5min":2.46,"15min":2.46},"last":1.75}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":153047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":1.048,"15min":1.048},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":1.45}]},{"osd":1,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.3120000000000001}]},{"osd":2,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.361,"15min":1.361},"min":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.1839999999999999}]},{"osd":3,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.347,"15min":1.347},"min":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":2.3109999999999999,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":1.3999999999999999}]},{"osd":5,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.476,"15min":1.476},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":3.1760000000000002,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.4220000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.876}]},{"osd":6,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.1520000000000001},"last":1.26},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":3.2799999999999998,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":1.157}]},{"osd":7,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.669,"5min":2.669,"15min":2.669},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.673,"5min":1.673,"15min":1.673},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":2.387,"5min":2.387,"15min":2.387},"last":1.5}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":152189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.2650000000000001,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":1.034},{"interface":"front","average":{"1min":1.004,"5min":1.004,"15min":1.004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":2.444,"5min":2.444,"15min":2.444},"last":1.097}]},{"osd":1,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":2.3900000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":1.4710000000000001}]},{"osd":2,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":2.319,"5min":2.319,"15min":2.319},"last":1.3540000000000001}]},{"osd":4,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.292,"15min":1.292},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":2.5699999999999998,"5min":2.5699999999999998,"15min":2.5699999999999998},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.5059999999999998,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":1.7110000000000001}]},{"osd":5,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"max":{"1min":2.6280000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":1.659}]},{"osd":6,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":1.754},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"min":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"max":{"1min":2.6000000000000001,"5min":2.6000000000000001,"15min":2.6000000000000001},"last":1.825}]},{"osd":7,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":1.7869999999999999},{"interface":"front","average":{"1min":1.496,"5min":1.496,"15min":1.496},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":2.5419999999999998,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":1.8580000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":151447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.6560000000000001,"5min":2.6560000000000001,"15min":2.6560000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.85399999999999998}]},{"osd":1,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":0.98399999999999999}]},{"osd":3,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":2.5590000000000002,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":0.72499999999999998}]},{"osd":4,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.88,"5min":0.88,"15min":0.88},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.92500000000000004}]},{"osd":5,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":1.155},{"interface":"front","average":{"1min":1.302,"5min":1.302,"15min":1.302},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.044}]},{"osd":6,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":2.7429999999999999,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":1.0960000000000001}]},{"osd":7,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.413,"15min":1.413},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":0.95699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":983040,"data_stored":741851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.4940000000000002,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":4.0819999999999999,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.56399999999999995}]},{"osd":2,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":4.0090000000000003,"5min":4.0090000000000003,"15min":4.0090000000000003},"last":0.96399999999999997}]},{"osd":3,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":3.9569999999999999,"5min":3.9569999999999999,"15min":3.9569999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.288,"5min":1.288,"15min":1.288},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.6389999999999998,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":0.84299999999999997}]},{"osd":4,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.6759999999999999,"15min":1.6759999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":4.29,"5min":4.29,"15min":4.29},"last":1.4450000000000001}]},{"osd":5,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":4.1589999999999998,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.601,"5min":1.601,"15min":1.601},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":4.2460000000000004,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":1.2050000000000001}]},{"osd":6,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":1.7809999999999999},"min":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":4.3419999999999996,"5min":4.3419999999999996,"15min":4.3419999999999996},"last":1.518},{"interface":"front","average":{"1min":1.766,"5min":1.766,"15min":1.766},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":4.1239999999999997,"5min":4.1239999999999997,"15min":4.1239999999999997},"last":1.2789999999999999}]},{"osd":7,"last update":"Tue Nov 28 17:09:50 2023","interfaces":[{"interface":"back","average":{"1min":1.804,"5min":1.804,"15min":1.804},"min":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.4220000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":1.123}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":151503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 17:09:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.82199999999999995}]},{"osd":2,"last update":"Tue Nov 28 17:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.601,"5min":1.601,"15min":1.601},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.63200000000000001}]},{"osd":3,"last update":"Tue Nov 28 17:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.181,"15min":1.181},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":1.208},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.76400000000000001}]},{"osd":4,"last update":"Tue Nov 28 17:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.464,"15min":1.464},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.1279999999999999}]},{"osd":5,"last update":"Tue Nov 28 17:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":2.2570000000000001,"5min":2.2570000000000001,"15min":2.2570000000000001},"last":1.296},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.3999999999999999}]},{"osd":6,"last update":"Tue Nov 28 17:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.423,"15min":1.423},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.3069999999999999,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.4470000000000001}]},{"osd":7,"last update":"Tue Nov 28 17:09:52 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.3340000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T17:10:25.258 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T17:10:25.258 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-28T17:10:25.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.505+0000 7f378893c640 1 Processor -- start 2023-11-28T17:10:25.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.505+0000 7f378893c640 1 -- start start 2023-11-28T17:10:25.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.507+0000 7f378893c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f378413d170 0x7f378413f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:25.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.507+0000 7f378893c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f37840ce460 0x7f37840ce910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:25.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.507+0000 7f378893c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37840cdfd0 con 0x7f378413fd40 2023-11-28T17:10:25.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.507+0000 7f378893c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37841402a0 con 0x7f378413d170 2023-11-28T17:10:25.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.507+0000 7f378893c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3784140580 con 0x7f37840ce460 2023-11-28T17:10:25.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.507+0000 7f378259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f378413d170 0x7f378413f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:25.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.507+0000 7f378259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f378413d170 0x7f378413f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57620/0 (socket says 172.21.15.116:57620) 2023-11-28T17:10:25.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.507+0000 7f378259c640 1 -- 172.21.15.116:0/2171787355 learned_addr learned my addr 172.21.15.116:0/2171787355 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:25.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.507+0000 7f378259c640 1 -- 172.21.15.116:0/2171787355 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f37840ce460 msgr2=0x7f37840ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:25.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.507+0000 7f378259c640 1 --2- 172.21.15.116:0/2171787355 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f37840ce460 0x7f37840ce910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.507+0000 7f378259c640 1 -- 172.21.15.116:0/2171787355 >> v1:172.21.15.156:6789/0 conn(0x7f378413fd40 legacy=0x7f378413d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:25.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.508+0000 7f378259c640 1 -- 172.21.15.116:0/2171787355 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37840cdc60 con 0x7f378413d170 2023-11-28T17:10:25.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.509+0000 7f378259c640 1 --2- 172.21.15.116:0/2171787355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f378413d170 0x7f378413f800 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f377400bb40 tx=0x7f377402e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba04d948409c86a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:25.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.509+0000 7f378159a640 1 -- 172.21.15.116:0/2171787355 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377402ede0 con 0x7f378413d170 2023-11-28T17:10:25.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.509+0000 7f378159a640 1 -- 172.21.15.116:0/2171787355 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3774036440 con 0x7f378413d170 2023-11-28T17:10:25.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.509+0000 7f378159a640 1 -- 172.21.15.116:0/2171787355 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37740367d0 con 0x7f378413d170 2023-11-28T17:10:25.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.509+0000 7f378893c640 1 -- 172.21.15.116:0/2171787355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f378413d170 msgr2=0x7f378413f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:25.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.509+0000 7f378893c640 1 --2- 172.21.15.116:0/2171787355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f378413d170 0x7f378413f800 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f377400bb40 tx=0x7f377402e030 comp rx=0 tx=0).stop 2023-11-28T17:10:25.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.509+0000 7f378893c640 1 -- 172.21.15.116:0/2171787355 shutdown_connections 2023-11-28T17:10:25.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.509+0000 7f378893c640 1 --2- 172.21.15.116:0/2171787355 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f37840ce460 0x7f37840ce910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.509+0000 7f378893c640 1 --2- 172.21.15.116:0/2171787355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f378413d170 0x7f378413f800 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.509+0000 7f378893c640 1 -- 172.21.15.116:0/2171787355 >> 172.21.15.116:0/2171787355 conn(0x7f37840b0cf0 msgr2=0x7f37840b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:25.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.510+0000 7f378893c640 1 -- 172.21.15.116:0/2171787355 shutdown_connections 2023-11-28T17:10:25.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.510+0000 7f378893c640 1 -- 172.21.15.116:0/2171787355 wait complete. 2023-11-28T17:10:25.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.512+0000 7f378893c640 1 Processor -- start 2023-11-28T17:10:25.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.512+0000 7f378893c640 1 -- start start 2023-11-28T17:10:25.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.514+0000 7f378893c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37840ce460 0x7f3784141300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:25.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.514+0000 7f378893c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f378413d170 0x7f3784142b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:25.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.514+0000 7f378893c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f378413fd40 0x7f37841398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:25.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.514+0000 7f378893c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37840aef40 con 0x7f378413d170 2023-11-28T17:10:25.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.514+0000 7f378893c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3784142380 con 0x7f37840ce460 2023-11-28T17:10:25.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.514+0000 7f378893c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3784139df0 con 0x7f378413fd40 2023-11-28T17:10:25.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.514+0000 7f3781d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f378413d170 0x7f3784142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:25.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.514+0000 7f3781d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f378413d170 0x7f3784142b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57628/0 (socket says 172.21.15.116:57628) 2023-11-28T17:10:25.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.514+0000 7f3781d9b640 1 -- 172.21.15.116:0/1294209728 learned_addr learned my addr 172.21.15.116:0/1294209728 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:25.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.515+0000 7f3781d9b640 1 -- 172.21.15.116:0/1294209728 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f378413fd40 msgr2=0x7f37841398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:25.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.515+0000 7f3781d9b640 1 --2- 172.21.15.116:0/1294209728 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f378413fd40 0x7f37841398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.515+0000 7f3781d9b640 1 -- 172.21.15.116:0/1294209728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37840ce460 msgr2=0x7f3784141300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:25.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.515+0000 7f378259c640 1 --2- 172.21.15.116:0/1294209728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37840ce460 0x7f3784141300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:25.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.515+0000 7f3781d9b640 1 --2- 172.21.15.116:0/1294209728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37840ce460 0x7f3784141300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.515+0000 7f3781d9b640 1 -- 172.21.15.116:0/1294209728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377400b770 con 0x7f378413d170 2023-11-28T17:10:25.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.515+0000 7f3781d9b640 1 --2- 172.21.15.116:0/1294209728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f378413d170 0x7f3784142b80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f377800b3c0 tx=0x7f377800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:25.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.515+0000 7f3772ffd640 1 -- 172.21.15.116:0/1294209728 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778011830 con 0x7f378413d170 2023-11-28T17:10:25.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.515+0000 7f3772ffd640 1 -- 172.21.15.116:0/1294209728 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3778011990 con 0x7f378413d170 2023-11-28T17:10:25.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.515+0000 7f3772ffd640 1 -- 172.21.15.116:0/1294209728 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778011d80 con 0x7f378413d170 2023-11-28T17:10:25.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.515+0000 7f378893c640 1 -- 172.21.15.116:0/1294209728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378413e000 con 0x7f378413d170 2023-11-28T17:10:25.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.515+0000 7f378893c640 1 -- 172.21.15.116:0/1294209728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37840aebd0 con 0x7f378413d170 2023-11-28T17:10:25.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.516+0000 7f378893c640 1 -- 172.21.15.116:0/1294209728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3784067980 con 0x7f378413d170 2023-11-28T17:10:25.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.517+0000 7f3772ffd640 1 -- 172.21.15.116:0/1294209728 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f377801b460 con 0x7f378413d170 2023-11-28T17:10:25.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.518+0000 7f3772ffd640 1 --2- 172.21.15.116:0/1294209728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f376406b2a0 0x7f376406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:25.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.518+0000 7f3772ffd640 1 -- 172.21.15.116:0/1294209728 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3778089760 con 0x7f378413d170 2023-11-28T17:10:25.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.519+0000 7f378259c640 1 --2- 172.21.15.116:0/1294209728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f376406b2a0 0x7f376406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:25.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.519+0000 7f378259c640 1 --2- 172.21.15.116:0/1294209728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f376406b2a0 0x7f376406d760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3774038000 tx=0x7f3774032000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:25.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.523+0000 7f3772ffd640 1 -- 172.21.15.116:0/1294209728 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37780545d0 con 0x7f378413d170 2023-11-28T17:10:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.764+0000 7f378893c640 1 -- 172.21.15.116:0/1294209728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f37840af080 con 0x7f378413d170 2023-11-28T17:10:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.769+0000 7f3772ffd640 1 -- 172.21.15.116:0/1294209728 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f37840af080 con 0x7f378413d170 2023-11-28T17:10:25.770 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2023-11-28T17:10:25.771 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBtHmZlPRewFxAAfLJPZrTljlPIoghT5ZzoBA== 2023-11-28T17:10:25.771 INFO:teuthology.orchestra.run.smithi116.stdout: caps mds = "allow" 2023-11-28T17:10:25.771 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "allow r" 2023-11-28T17:10:25.771 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-28T17:10:25.771 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.0 2023-11-28T17:10:25.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.771+0000 7f378893c640 1 -- 172.21.15.116:0/1294209728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f376406b2a0 msgr2=0x7f376406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:25.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.771+0000 7f378893c640 1 --2- 172.21.15.116:0/1294209728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f376406b2a0 0x7f376406d760 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3774038000 tx=0x7f3774032000 comp rx=0 tx=0).stop 2023-11-28T17:10:25.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.771+0000 7f378893c640 1 -- 172.21.15.116:0/1294209728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f378413d170 msgr2=0x7f3784142b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:25.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.771+0000 7f378893c640 1 --2- 172.21.15.116:0/1294209728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f378413d170 0x7f3784142b80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f377800b3c0 tx=0x7f377800b880 comp rx=0 tx=0).stop 2023-11-28T17:10:25.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.771+0000 7f378893c640 1 -- 172.21.15.116:0/1294209728 shutdown_connections 2023-11-28T17:10:25.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.771+0000 7f378893c640 1 --2- 172.21.15.116:0/1294209728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f376406b2a0 0x7f376406d760 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.771+0000 7f378893c640 1 --2- 172.21.15.116:0/1294209728 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f378413fd40 0x7f37841398b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.771+0000 7f378893c640 1 --2- 172.21.15.116:0/1294209728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f378413d170 0x7f3784142b80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.771+0000 7f378893c640 1 --2- 172.21.15.116:0/1294209728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37840ce460 0x7f3784141300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:25.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.772+0000 7f378893c640 1 -- 172.21.15.116:0/1294209728 >> 172.21.15.116:0/1294209728 conn(0x7f37840b0cf0 msgr2=0x7f3784158c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:25.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.772+0000 7f378893c640 1 -- 172.21.15.116:0/1294209728 shutdown_connections 2023-11-28T17:10:25.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.772+0000 7f378893c640 1 -- 172.21.15.116:0/1294209728 wait complete. 2023-11-28T17:10:25.791 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:10:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.999+0000 7f6aed25c640 1 Processor -- start 2023-11-28T17:10:26.000 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:25.999+0000 7f6aed25c640 1 -- start start 2023-11-28T17:10:26.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.000+0000 7f6aed25c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6ae8144430 0x7f6ae815f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.000+0000 7f6aed25c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6ae806c250 0x7f6ae806c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.001+0000 7f6aed25c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae8070bd0 con 0x7f6ae80703f0 2023-11-28T17:10:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.001+0000 7f6aed25c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae8144810 con 0x7f6ae8144430 2023-11-28T17:10:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.001+0000 7f6aed25c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae806cc20 con 0x7f6ae806c250 2023-11-28T17:10:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.001+0000 7f6ae6d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6ae8144430 0x7f6ae815f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.001+0000 7f6ae6d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6ae8144430 0x7f6ae815f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57642/0 (socket says 172.21.15.116:57642) 2023-11-28T17:10:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.001+0000 7f6ae6d9d640 1 -- 172.21.15.116:0/2096460469 learned_addr learned my addr 172.21.15.116:0/2096460469 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.001+0000 7f6ae759e640 1 --2- 172.21.15.116:0/2096460469 >> v2:172.21.15.156:3300/0 conn(0x7f6ae806c250 0x7f6ae806c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.001+0000 7f6ae6d9d640 1 -- 172.21.15.116:0/2096460469 >> v2:172.21.15.156:3300/0 conn(0x7f6ae806c250 msgr2=0x7f6ae806c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.001+0000 7f6ae6d9d640 1 --2- 172.21.15.116:0/2096460469 >> v2:172.21.15.156:3300/0 conn(0x7f6ae806c250 0x7f6ae806c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.001+0000 7f6ae6d9d640 1 -- 172.21.15.116:0/2096460469 >> v1:172.21.15.156:6789/0 conn(0x7f6ae80703f0 legacy=0x7f6ae8071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:26.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.001+0000 7f6ae6d9d640 1 -- 172.21.15.116:0/2096460469 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae8070eb0 con 0x7f6ae8144430 2023-11-28T17:10:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.003+0000 7f6ae6d9d640 1 --2- 172.21.15.116:0/2096460469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ae8144430 0x7f6ae815f7b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6ae806ce10 tx=0x7f6ad802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84725294c9502fe server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.003+0000 7f6ae5d9b640 1 -- 172.21.15.116:0/2096460469 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad802ca80 con 0x7f6ae8144430 2023-11-28T17:10:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.003+0000 7f6ae5d9b640 1 -- 172.21.15.116:0/2096460469 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad802cbe0 con 0x7f6ae8144430 2023-11-28T17:10:26.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.003+0000 7f6ae5d9b640 1 -- 172.21.15.116:0/2096460469 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad80343f0 con 0x7f6ae8144430 2023-11-28T17:10:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.004+0000 7f6aed25c640 1 -- 172.21.15.116:0/2096460469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ae8144430 msgr2=0x7f6ae815f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.004+0000 7f6aed25c640 1 --2- 172.21.15.116:0/2096460469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ae8144430 0x7f6ae815f7b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6ae806ce10 tx=0x7f6ad802bee0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.004+0000 7f6aed25c640 1 -- 172.21.15.116:0/2096460469 shutdown_connections 2023-11-28T17:10:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.004+0000 7f6aed25c640 1 --2- 172.21.15.116:0/2096460469 >> v2:172.21.15.156:3300/0 conn(0x7f6ae806c250 0x7f6ae806c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.004+0000 7f6aed25c640 1 --2- 172.21.15.116:0/2096460469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ae8144430 0x7f6ae815f7b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.004+0000 7f6aed25c640 1 -- 172.21.15.116:0/2096460469 >> 172.21.15.116:0/2096460469 conn(0x7f6ae806a4c0 msgr2=0x7f6ae806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.004+0000 7f6aed25c640 1 -- 172.21.15.116:0/2096460469 shutdown_connections 2023-11-28T17:10:26.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.004+0000 7f6aed25c640 1 -- 172.21.15.116:0/2096460469 wait complete. 2023-11-28T17:10:26.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.005+0000 7f6aed25c640 1 Processor -- start 2023-11-28T17:10:26.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.006+0000 7f6aed25c640 1 -- start start 2023-11-28T17:10:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6aed25c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ae806c250 0x7f6ae81517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6aed25c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ae80703f0 0x7f6ae813a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6aed25c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ae8144430 0x7f6ae813a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6aed25c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae8151d20 con 0x7f6ae8144430 2023-11-28T17:10:26.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6aed25c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae815b4f0 con 0x7f6ae80703f0 2023-11-28T17:10:26.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6aed25c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae815b7d0 con 0x7f6ae806c250 2023-11-28T17:10:26.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6ae759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ae8144430 0x7f6ae813a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:26.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6ae759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ae8144430 0x7f6ae813a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57644/0 (socket says 172.21.15.116:57644) 2023-11-28T17:10:26.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6ae759e640 1 -- 172.21.15.116:0/3418455201 learned_addr learned my addr 172.21.15.116:0/3418455201 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:26.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6ae659c640 1 --2- 172.21.15.116:0/3418455201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ae80703f0 0x7f6ae813a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:26.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6ae759e640 1 -- 172.21.15.116:0/3418455201 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ae806c250 msgr2=0x7f6ae81517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6ae759e640 1 --2- 172.21.15.116:0/3418455201 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ae806c250 0x7f6ae81517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6ae759e640 1 -- 172.21.15.116:0/3418455201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ae80703f0 msgr2=0x7f6ae813a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6ae759e640 1 --2- 172.21.15.116:0/3418455201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ae80703f0 0x7f6ae813a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6ae759e640 1 -- 172.21.15.116:0/3418455201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad8009b30 con 0x7f6ae8144430 2023-11-28T17:10:26.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.007+0000 7f6ae659c640 1 --2- 172.21.15.116:0/3418455201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ae80703f0 0x7f6ae813a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:26.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.008+0000 7f6ae759e640 1 --2- 172.21.15.116:0/3418455201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ae8144430 0x7f6ae813a830 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000d5d0 tx=0x7f6ae000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:26.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.008+0000 7f6acf7fe640 1 -- 172.21.15.116:0/3418455201 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae00125f0 con 0x7f6ae8144430 2023-11-28T17:10:26.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.008+0000 7f6acf7fe640 1 -- 172.21.15.116:0/3418455201 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae0012770 con 0x7f6ae8144430 2023-11-28T17:10:26.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.008+0000 7f6acf7fe640 1 -- 172.21.15.116:0/3418455201 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0012b20 con 0x7f6ae8144430 2023-11-28T17:10:26.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.008+0000 7f6aed25c640 1 -- 172.21.15.116:0/3418455201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae815bab0 con 0x7f6ae8144430 2023-11-28T17:10:26.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.008+0000 7f6aed25c640 1 -- 172.21.15.116:0/3418455201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae80b3f40 con 0x7f6ae8144430 2023-11-28T17:10:26.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.009+0000 7f6aed25c640 1 -- 172.21.15.116:0/3418455201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae80693d0 con 0x7f6ae8144430 2023-11-28T17:10:26.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.010+0000 7f6acf7fe640 1 -- 172.21.15.116:0/3418455201 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6ae0012c80 con 0x7f6ae8144430 2023-11-28T17:10:26.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.011+0000 7f6acf7fe640 1 --2- 172.21.15.116:0/3418455201 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6ad406b2d0 0x7f6ad406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.011+0000 7f6acf7fe640 1 -- 172.21.15.116:0/3418455201 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ae008a0f0 con 0x7f6ae8144430 2023-11-28T17:10:26.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.011+0000 7f6ae6d9d640 1 --2- 172.21.15.116:0/3418455201 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6ad406b2d0 0x7f6ad406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:26.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.012+0000 7f6ae6d9d640 1 --2- 172.21.15.116:0/3418455201 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6ad406b2d0 0x7f6ad406d790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8035000 tx=0x7f6ad802e000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:26.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.016+0000 7f6acf7fe640 1 -- 172.21.15.116:0/3418455201 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ae0054ed0 con 0x7f6ae8144430 2023-11-28T17:10:26.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.244+0000 7f6aed25c640 1 -- 172.21.15.116:0/3418455201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ae80cd020 con 0x7f6ae8144430 2023-11-28T17:10:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.245+0000 7f6acf7fe640 1 -- 172.21.15.116:0/3418455201 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f6ae00584f0 con 0x7f6ae8144430 2023-11-28T17:10:26.247 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:26.247 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/491158514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":491158514},{"type":"v1","addr":"172.21.15.156:6834","nonce":491158514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3927645596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3927645596},{"type":"v1","addr":"172.21.15.116:6835","nonce":3927645596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4738,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/141796020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":141796020},{"type":"v1","addr":"172.21.15.156:6835","nonce":141796020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:10:19.725913+0000","modified":"2023-11-28T17:10:22.808343+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6837/3177126349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3177126349},{"type":"v1","addr":"172.21.15.116:6837","nonce":3177126349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T17:10:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 15 2023-11-28T17:10:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.248+0000 7f6aed25c640 1 -- 172.21.15.116:0/3418455201 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6ad406b2d0 msgr2=0x7f6ad406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.248+0000 7f6aed25c640 1 --2- 172.21.15.116:0/3418455201 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6ad406b2d0 0x7f6ad406d790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8035000 tx=0x7f6ad802e000 comp rx=0 tx=0).stop 2023-11-28T17:10:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.248+0000 7f6aed25c640 1 -- 172.21.15.116:0/3418455201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ae8144430 msgr2=0x7f6ae813a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.248+0000 7f6aed25c640 1 --2- 172.21.15.116:0/3418455201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ae8144430 0x7f6ae813a830 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000d5d0 tx=0x7f6ae000da90 comp rx=0 tx=0).stop 2023-11-28T17:10:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.248+0000 7f6aed25c640 1 -- 172.21.15.116:0/3418455201 shutdown_connections 2023-11-28T17:10:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.248+0000 7f6aed25c640 1 --2- 172.21.15.116:0/3418455201 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6ad406b2d0 0x7f6ad406d790 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.248+0000 7f6aed25c640 1 --2- 172.21.15.116:0/3418455201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ae8144430 0x7f6ae813a830 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.248+0000 7f6aed25c640 1 --2- 172.21.15.116:0/3418455201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ae80703f0 0x7f6ae813a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.248+0000 7f6aed25c640 1 --2- 172.21.15.116:0/3418455201 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ae806c250 0x7f6ae81517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.248+0000 7f6aed25c640 1 -- 172.21.15.116:0/3418455201 >> 172.21.15.116:0/3418455201 conn(0x7f6ae806a4c0 msgr2=0x7f6ae813cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.248+0000 7f6aed25c640 1 -- 172.21.15.116:0/3418455201 shutdown_connections 2023-11-28T17:10:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.248+0000 7f6aed25c640 1 -- 172.21.15.116:0/3418455201 wait complete. 2023-11-28T17:10:26.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T17:10:19.725913+0000', 'modified': '2023-11-28T17:10:22.808343+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5202}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5202': {'gid': 5202, 'name': 'c', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.116:6837/3177126349', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6836', 'nonce': 3177126349}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 3177126349}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T17:10:26.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T17:10:26.671 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T17:10:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.892+0000 7feb26790640 1 Processor -- start 2023-11-28T17:10:26.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.892+0000 7feb26790640 1 -- start start 2023-11-28T17:10:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.893+0000 7feb26790640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feb2015dab0 0x7feb200703f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.893+0000 7feb26790640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb20154220 con 0x7feb200b3310 2023-11-28T17:10:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.893+0000 7feb26790640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2015d2f0 con 0x7feb2006cae0 2023-11-28T17:10:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.893+0000 7feb26790640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2015a3c0 con 0x7feb2015dab0 2023-11-28T17:10:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.894+0000 7feb24d2d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feb2015dab0 0x7feb200703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.894+0000 7feb17fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7feb200b3310 0x7feb20159cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49088/0 (socket says 172.21.15.116:49088) 2023-11-28T17:10:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.894+0000 7feb17fff640 1 -- 172.21.15.116:0/3914191495 learned_addr learned my addr 172.21.15.116:0/3914191495 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.894+0000 7feb24d2d640 1 -- 172.21.15.116:0/3914191495 >> v1:172.21.15.156:6789/0 conn(0x7feb2006cae0 legacy=0x7feb2006cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.894+0000 7feb24d2d640 1 -- 172.21.15.116:0/3914191495 >> v1:172.21.15.116:6789/0 conn(0x7feb200b3310 legacy=0x7feb20159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.894+0000 7feb1f7fe640 1 -- 172.21.15.116:0/3914191495 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592088984 0 0) 0x7feb08001110 con 0x7feb200b3310 2023-11-28T17:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.894+0000 7feb24d2d640 1 -- 172.21.15.116:0/3914191495 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2015a5c0 con 0x7feb2015dab0 2023-11-28T17:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.896+0000 7feb24d2d640 1 --2- 172.21.15.116:0/3914191495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb2015dab0 0x7feb200703f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7feb18009d90 tx=0x7feb1802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf29286a5196ff8f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.896+0000 7feb1f7fe640 1 -- 172.21.15.116:0/3914191495 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1802cd00 con 0x7feb2015dab0 2023-11-28T17:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.896+0000 7feb1f7fe640 1 -- 172.21.15.116:0/3914191495 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1802ce60 con 0x7feb2015dab0 2023-11-28T17:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.896+0000 7feb1f7fe640 1 -- 172.21.15.116:0/3914191495 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18034660 con 0x7feb2015dab0 2023-11-28T17:10:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.897+0000 7feb26790640 1 -- 172.21.15.116:0/3914191495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb2015dab0 msgr2=0x7feb200703f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.897+0000 7feb26790640 1 --2- 172.21.15.116:0/3914191495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb2015dab0 0x7feb200703f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7feb18009d90 tx=0x7feb1802bf50 comp rx=0 tx=0).stop 2023-11-28T17:10:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.897+0000 7feb26790640 1 -- 172.21.15.116:0/3914191495 shutdown_connections 2023-11-28T17:10:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.897+0000 7feb26790640 1 --2- 172.21.15.116:0/3914191495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb2015dab0 0x7feb200703f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.897+0000 7feb26790640 1 -- 172.21.15.116:0/3914191495 >> 172.21.15.116:0/3914191495 conn(0x7feb2006a4c0 msgr2=0x7feb2006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.897+0000 7feb26790640 1 -- 172.21.15.116:0/3914191495 shutdown_connections 2023-11-28T17:10:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.897+0000 7feb26790640 1 -- 172.21.15.116:0/3914191495 wait complete. 2023-11-28T17:10:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.898+0000 7feb26790640 1 Processor -- start 2023-11-28T17:10:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.899+0000 7feb26790640 1 -- start start 2023-11-28T17:10:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.900+0000 7feb26790640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb2006cae0 0x7feb20155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.900+0000 7feb26790640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb200b3310 0x7feb201ec540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.900+0000 7feb26790640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb2015dab0 0x7feb2014ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.900+0000 7feb26790640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb201efdb0 con 0x7feb200b3310 2023-11-28T17:10:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.900+0000 7feb26790640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb201ef410 con 0x7feb2015dab0 2023-11-28T17:10:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.900+0000 7feb26790640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb201ef6f0 con 0x7feb2006cae0 2023-11-28T17:10:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.900+0000 7feb17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb200b3310 0x7feb201ec540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.900+0000 7feb17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb200b3310 0x7feb201ec540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57660/0 (socket says 172.21.15.116:57660) 2023-11-28T17:10:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.900+0000 7feb17fff640 1 -- 172.21.15.116:0/2579821823 learned_addr learned my addr 172.21.15.116:0/2579821823 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.900+0000 7feb17fff640 1 -- 172.21.15.116:0/2579821823 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb2006cae0 msgr2=0x7feb20155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.900+0000 7feb17fff640 1 --2- 172.21.15.116:0/2579821823 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb2006cae0 0x7feb20155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.900+0000 7feb17fff640 1 -- 172.21.15.116:0/2579821823 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb2015dab0 msgr2=0x7feb2014ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.900+0000 7feb17fff640 1 --2- 172.21.15.116:0/2579821823 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb2015dab0 0x7feb2014ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.900+0000 7feb17fff640 1 -- 172.21.15.116:0/2579821823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb18009a70 con 0x7feb200b3310 2023-11-28T17:10:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.901+0000 7feb17fff640 1 --2- 172.21.15.116:0/2579821823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb200b3310 0x7feb201ec540 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7feb0800c750 tx=0x7feb0800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.901+0000 7feb1d7fa640 1 -- 172.21.15.116:0/2579821823 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb080072d0 con 0x7feb200b3310 2023-11-28T17:10:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.901+0000 7feb1d7fa640 1 -- 172.21.15.116:0/2579821823 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb08007430 con 0x7feb200b3310 2023-11-28T17:10:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.901+0000 7feb1d7fa640 1 -- 172.21.15.116:0/2579821823 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08001d30 con 0x7feb200b3310 2023-11-28T17:10:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.901+0000 7feb26790640 1 -- 172.21.15.116:0/2579821823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb201ef9d0 con 0x7feb200b3310 2023-11-28T17:10:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.901+0000 7feb26790640 1 -- 172.21.15.116:0/2579821823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb201ef0c0 con 0x7feb200b3310 2023-11-28T17:10:26.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.904+0000 7feb1d7fa640 1 -- 172.21.15.116:0/2579821823 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7feb080047e0 con 0x7feb200b3310 2023-11-28T17:10:26.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.905+0000 7feb1d7fa640 1 --2- 172.21.15.116:0/2579821823 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feb0006b2a0 0x7feb0006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.905+0000 7feb1d7fa640 1 -- 172.21.15.116:0/2579821823 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb08088c90 con 0x7feb200b3310 2023-11-28T17:10:26.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.905+0000 7feb1ffff640 1 --2- 172.21.15.116:0/2579821823 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feb0006b2a0 0x7feb0006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:26.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.906+0000 7feb293ff740 1 Processor -- start 2023-11-28T17:10:26.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.906+0000 7feb293ff740 1 -- start start 2023-11-28T17:10:26.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.906+0000 7feb1ffff640 1 --2- 172.21.15.116:0/2579821823 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feb0006b2a0 0x7feb0006d760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7feb10000f60 tx=0x7feb100079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.907+0000 7feb293ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b2081f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.907+0000 7feb293ff740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b1d80c0 0x55700b205080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.907+0000 7feb293ff740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b1db990 0x55700b2055c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.907+0000 7feb293ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55700b13c880 con 0x55700b207e10 2023-11-28T17:10:26.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.907+0000 7feb293ff740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55700b14bae0 con 0x55700b1db990 2023-11-28T17:10:26.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.907+0000 7feb293ff740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55700b1491d0 con 0x55700b1d80c0 2023-11-28T17:10:26.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.907+0000 7feb1ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b2081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:26.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.907+0000 7feb1ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b2081f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57662/0 (socket says 172.21.15.116:57662) 2023-11-28T17:10:26.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.907+0000 7feb1ffff640 1 -- 172.21.15.116:0/90973172 learned_addr learned my addr 172.21.15.116:0/90973172 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:26.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.907+0000 7feb1ffff640 1 -- 172.21.15.116:0/90973172 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b1d80c0 msgr2=0x55700b205080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:26.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.907+0000 7feb1ffff640 1 --2- 172.21.15.116:0/90973172 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b1d80c0 0x55700b205080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.907+0000 7feb1ffff640 1 -- 172.21.15.116:0/90973172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b1db990 msgr2=0x55700b2055c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:26.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.907+0000 7feb1ffff640 1 --2- 172.21.15.116:0/90973172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b1db990 0x55700b2055c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.907+0000 7feb1ffff640 1 -- 172.21.15.116:0/90973172 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55700b10c860 con 0x55700b207e10 2023-11-28T17:10:26.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.908+0000 7feb1ffff640 1 --2- 172.21.15.116:0/90973172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b2081f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7feb10016700 tx=0x7feb1000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654a586a52bd7cc8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.908+0000 7feb15ffb640 1 -- 172.21.15.116:0/90973172 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1001b070 con 0x55700b207e10 2023-11-28T17:10:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.908+0000 7feb15ffb640 1 -- 172.21.15.116:0/90973172 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb100163a0 con 0x55700b207e10 2023-11-28T17:10:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.908+0000 7feb15ffb640 1 -- 172.21.15.116:0/90973172 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10005350 con 0x55700b207e10 2023-11-28T17:10:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.909+0000 7feb293ff740 1 -- 172.21.15.116:0/90973172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 msgr2=0x55700b2081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.909+0000 7feb293ff740 1 --2- 172.21.15.116:0/90973172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b2081f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7feb10016700 tx=0x7feb1000f000 comp rx=0 tx=0).stop 2023-11-28T17:10:26.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.909+0000 7feb293ff740 1 -- 172.21.15.116:0/90973172 shutdown_connections 2023-11-28T17:10:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.909+0000 7feb293ff740 1 --2- 172.21.15.116:0/90973172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b1db990 0x55700b2055c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.909+0000 7feb293ff740 1 --2- 172.21.15.116:0/90973172 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b1d80c0 0x55700b205080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.909+0000 7feb293ff740 1 --2- 172.21.15.116:0/90973172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b2081f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.909+0000 7feb293ff740 1 -- 172.21.15.116:0/90973172 >> 172.21.15.116:0/90973172 conn(0x55700b1da330 msgr2=0x55700b222c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.909+0000 7feb293ff740 1 -- 172.21.15.116:0/90973172 shutdown_connections 2023-11-28T17:10:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.909+0000 7feb293ff740 1 -- 172.21.15.116:0/90973172 wait complete. 2023-11-28T17:10:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.909+0000 7feb293ff740 1 -- start start 2023-11-28T17:10:26.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.910+0000 7feb293ff740 1 Processor -- start 2023-11-28T17:10:26.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb293ff740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b1db990 0x55700b1f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb293ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b2100f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb293ff740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b210630 0x55700b209ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb293ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55700b144ac0 con 0x55700b207e10 2023-11-28T17:10:26.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb293ff740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55700b0e12f0 con 0x55700b1db990 2023-11-28T17:10:26.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb293ff740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55700b0fec70 con 0x55700b210630 2023-11-28T17:10:26.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb293ff740 10 client.0 _ll_register_callbacks cb 0x55700ae56ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:10:26.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b2100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:26.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b2100f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57670/0 (socket says 172.21.15.116:57670) 2023-11-28T17:10:26.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb17fff640 1 -- 172.21.15.116:0/824871993 learned_addr learned my addr 172.21.15.116:0/824871993 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:26.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb17fff640 1 -- 172.21.15.116:0/824871993 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b210630 msgr2=0x55700b209ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:26.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb17fff640 1 --2- 172.21.15.116:0/824871993 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b210630 0x55700b209ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb17fff640 1 -- 172.21.15.116:0/824871993 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b1db990 msgr2=0x55700b1f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:26.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb17fff640 1 --2- 172.21.15.116:0/824871993 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b1db990 0x55700b1f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.911+0000 7feb17fff640 1 -- 172.21.15.116:0/824871993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2015d2f0 con 0x55700b207e10 2023-11-28T17:10:26.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.912+0000 7feb17fff640 1 --2- 172.21.15.116:0/824871993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b2100f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7feb08007800 tx=0x7feb08057ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d180858b5385439 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:26.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.912+0000 7feafeffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-28T17:10:26.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.912+0000 7feafeffd640 1 -- 172.21.15.116:0/824871993 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0805f820 con 0x55700b207e10 2023-11-28T17:10:26.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.912+0000 7feafeffd640 1 -- 172.21.15.116:0/824871993 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb0805fab0 con 0x55700b207e10 2023-11-28T17:10:26.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.912+0000 7feb293ff740 1 -- 172.21.15.116:0/824871993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55700b13daa0 con 0x55700b207e10 2023-11-28T17:10:26.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.912+0000 7feafeffd640 1 -- 172.21.15.116:0/824871993 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb08058ae0 con 0x55700b207e10 2023-11-28T17:10:26.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.912+0000 7feb293ff740 10 client.5385 fetch_fsmap learned FSMap version 15 2023-11-28T17:10:26.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.912+0000 7feb293ff740 1 -- 172.21.15.116:0/824871993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x55700b143140 con 0x55700b207e10 2023-11-28T17:10:26.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.912+0000 7feafeffd640 1 -- 172.21.15.116:0/824871993 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55700b13daa0 con 0x55700b207e10 2023-11-28T17:10:26.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.912+0000 7feafeffd640 1 -- 172.21.15.116:0/824871993 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7feb08058c40 con 0x55700b207e10 2023-11-28T17:10:26.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.913+0000 7feb293ff740 10 client.5385 fetch_fsmap finished waiting for FSMap version 15 2023-11-28T17:10:26.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.913+0000 7feb293ff740 10 client.5385 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T17:10:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.913+0000 7feb293ff740 20 client.5385 populate_metadata read hostname 'smithi116' 2023-11-28T17:10:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.913+0000 7feb293ff740 1 --2- 172.21.15.116:0/824871993 >> [v2:172.21.15.116:6836/3177126349,v1:172.21.15.116:6837/3177126349] conn(0x55700b1f3790 0x55700b1f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.913+0000 7feb293ff740 4 client.5385 mds_command: new command op to 5202 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T17:10:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.913+0000 7feb293ff740 1 -- 172.21.15.116:0/824871993 --> [v2:172.21.15.116:6836/3177126349,v1:172.21.15.116:6837/3177126349] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55700b13d7c0 con 0x55700b1f3790 2023-11-28T17:10:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.913+0000 7feb1ffff640 1 --2- 172.21.15.116:0/824871993 >> [v2:172.21.15.116:6836/3177126349,v1:172.21.15.116:6837/3177126349] conn(0x55700b1f3790 0x55700b1f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.913+0000 7feb1ffff640 1 --2- 172.21.15.116:0/824871993 >> [v2:172.21.15.116:6836/3177126349,v1:172.21.15.116:6837/3177126349] conn(0x55700b1f3790 0x55700b1f3b70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c18675563eec43db server_cookie=343de8a847f43318 in_seq=0 out_seq=0 2023-11-28T17:10:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.913+0000 7feafeffd640 10 client.5385 ms_handle_connect on v2:172.21.15.116:6836/3177126349 2023-11-28T17:10:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.914+0000 7feafeffd640 1 -- 172.21.15.116:0/824871993 <== mds.0 v2:172.21.15.116:6836/3177126349 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55700b13d7c0 con 0x55700b1f3790 2023-11-28T17:10:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.914+0000 7feafeffd640 10 client.5385 handle_command_reply: tid=0 2023-11-28T17:10:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.914+0000 7feb293ff740 1 client.5385 shutdown 2023-11-28T17:10:26.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.914+0000 7feb293ff740 1 -- 172.21.15.116:0/824871993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 msgr2=0x55700b2100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.914+0000 7feb293ff740 1 --2- 172.21.15.116:0/824871993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b2100f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7feb08007800 tx=0x7feb08057ff0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.914+0000 7feb293ff740 1 -- 172.21.15.116:0/824871993 shutdown_connections 2023-11-28T17:10:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.914+0000 7feb293ff740 1 --2- 172.21.15.116:0/824871993 >> [v2:172.21.15.116:6836/3177126349,v1:172.21.15.116:6837/3177126349] conn(0x55700b1f3790 0x55700b1f3b70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.914+0000 7feb293ff740 1 --2- 172.21.15.116:0/824871993 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b210630 0x55700b209ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.914+0000 7feb293ff740 1 --2- 172.21.15.116:0/824871993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b2100f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.914+0000 7feb293ff740 1 --2- 172.21.15.116:0/824871993 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b1db990 0x55700b1f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.914+0000 7feafeffd640 0 client.5385 ms_handle_reset on v2:172.21.15.116:6836/3177126349 2023-11-28T17:10:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.914+0000 7feb293ff740 1 -- 172.21.15.116:0/824871993 >> 172.21.15.116:0/824871993 conn(0x55700b1d80c0 msgr2=0x55700b20bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.915+0000 7feb293ff740 1 -- 172.21.15.116:0/824871993 shutdown_connections 2023-11-28T17:10:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.915+0000 7feb293ff740 1 -- 172.21.15.116:0/824871993 wait complete. 2023-11-28T17:10:26.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.915+0000 7feb293ff740 20 client.5385 trim_cache size 0 max 16384 2023-11-28T17:10:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.932+0000 7feb293ff740 1 Processor -- start 2023-11-28T17:10:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.932+0000 7feb293ff740 1 -- start start 2023-11-28T17:10:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.933+0000 7feb293ff740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b1db990 0x55700b209ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.933+0000 7feb293ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b225280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.933+0000 7feb293ff740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b210630 0x55700b1f1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.933+0000 7feb293ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55700aea7f20 con 0x55700b207e10 2023-11-28T17:10:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.933+0000 7feb293ff740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55700b1f2990 con 0x55700b210630 2023-11-28T17:10:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.933+0000 7feb293ff740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55700b20a0e0 con 0x55700b1db990 2023-11-28T17:10:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.933+0000 7feb17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b225280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.933+0000 7feb17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b225280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57678/0 (socket says 172.21.15.116:57678) 2023-11-28T17:10:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.933+0000 7feb17fff640 1 -- 172.21.15.116:0/1854879800 learned_addr learned my addr 172.21.15.116:0/1854879800 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:26.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.933+0000 7feb17fff640 1 -- 172.21.15.116:0/1854879800 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b1db990 msgr2=0x55700b209ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.933+0000 7feb17fff640 1 --2- 172.21.15.116:0/1854879800 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b1db990 0x55700b209ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.933+0000 7feb17fff640 1 -- 172.21.15.116:0/1854879800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b210630 msgr2=0x55700b1f1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.933+0000 7feb17fff640 1 --2- 172.21.15.116:0/1854879800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b210630 0x55700b1f1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.933+0000 7feb17fff640 1 -- 172.21.15.116:0/1854879800 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55700b0e0b80 con 0x55700b207e10 2023-11-28T17:10:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.934+0000 7feb17fff640 1 --2- 172.21.15.116:0/1854879800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b225280 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7feb080541d0 tx=0x7feb0805b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b45a8d94f8a2f242 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.934+0000 7feb167fc640 1 -- 172.21.15.116:0/1854879800 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0805f820 con 0x55700b207e10 2023-11-28T17:10:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.934+0000 7feb167fc640 1 -- 172.21.15.116:0/1854879800 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb08080e90 con 0x55700b207e10 2023-11-28T17:10:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.934+0000 7feb167fc640 1 -- 172.21.15.116:0/1854879800 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb080538e0 con 0x55700b207e10 2023-11-28T17:10:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.934+0000 7feb293ff740 1 -- 172.21.15.116:0/1854879800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 msgr2=0x55700b225280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.934+0000 7feb293ff740 1 --2- 172.21.15.116:0/1854879800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b225280 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7feb080541d0 tx=0x7feb0805b000 comp rx=0 tx=0).stop 2023-11-28T17:10:26.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.935+0000 7feb293ff740 1 -- 172.21.15.116:0/1854879800 shutdown_connections 2023-11-28T17:10:26.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.935+0000 7feb293ff740 1 --2- 172.21.15.116:0/1854879800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b210630 0x55700b1f1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.935+0000 7feb293ff740 1 --2- 172.21.15.116:0/1854879800 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b207e10 0x55700b225280 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.935+0000 7feb293ff740 1 --2- 172.21.15.116:0/1854879800 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b1db990 0x55700b209ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.935+0000 7feb293ff740 1 -- 172.21.15.116:0/1854879800 >> 172.21.15.116:0/1854879800 conn(0x55700b1d80c0 msgr2=0x55700b204b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:26.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.935+0000 7feb293ff740 1 -- 172.21.15.116:0/1854879800 shutdown_connections 2023-11-28T17:10:26.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.935+0000 7feb293ff740 1 -- 172.21.15.116:0/1854879800 wait complete. 2023-11-28T17:10:26.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.935+0000 7feb293ff740 1 -- start start 2023-11-28T17:10:26.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.935+0000 7feb293ff740 1 Processor -- start 2023-11-28T17:10:26.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.936+0000 7feb293ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b1db990 0x55700b295fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.936+0000 7feb293ff740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b207e10 0x55700b208db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.936+0000 7feb293ff740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b210630 0x55700b204290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.936+0000 7feb293ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55700b2085a0 con 0x55700b1db990 2023-11-28T17:10:26.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.936+0000 7feb293ff740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55700b1dd670 con 0x55700b210630 2023-11-28T17:10:26.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.936+0000 7feb293ff740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55700b296510 con 0x55700b207e10 2023-11-28T17:10:26.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.936+0000 7feb293ff740 10 client.0 _ll_register_callbacks cb 0x7feb08007db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:10:26.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.937+0000 7feb1ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b1db990 0x55700b295fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:26.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.937+0000 7feb1ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b1db990 0x55700b295fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57680/0 (socket says 172.21.15.116:57680) 2023-11-28T17:10:26.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.937+0000 7feb1ffff640 1 -- 172.21.15.116:0/3484982730 learned_addr learned my addr 172.21.15.116:0/3484982730 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:26.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.937+0000 7feb1ffff640 1 -- 172.21.15.116:0/3484982730 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b207e10 msgr2=0x55700b208db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:26.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.937+0000 7feb1ffff640 1 --2- 172.21.15.116:0/3484982730 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b207e10 0x55700b208db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.937+0000 7feb1ffff640 1 -- 172.21.15.116:0/3484982730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b210630 msgr2=0x55700b204290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.937+0000 7feb1ffff640 1 --2- 172.21.15.116:0/3484982730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b210630 0x55700b204290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.937+0000 7feb1ffff640 1 -- 172.21.15.116:0/3484982730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55700b143140 con 0x55700b1db990 2023-11-28T17:10:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.937+0000 7feb1ffff640 1 --2- 172.21.15.116:0/3484982730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b1db990 0x55700b295fd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7feb10004000 tx=0x7feb10001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7cf3d26ae697ab server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.937+0000 7feafeffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-28T17:10:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.937+0000 7feafeffd640 1 -- 172.21.15.116:0/3484982730 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10028d70 con 0x55700b1db990 2023-11-28T17:10:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.937+0000 7feafeffd640 1 -- 172.21.15.116:0/3484982730 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb10026040 con 0x55700b1db990 2023-11-28T17:10:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.937+0000 7feb293ff740 1 -- 172.21.15.116:0/3484982730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb100163a0 con 0x55700b1db990 2023-11-28T17:10:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.938+0000 7feafeffd640 1 -- 172.21.15.116:0/3484982730 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb10028320 con 0x55700b1db990 2023-11-28T17:10:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.938+0000 7feb293ff740 10 client.5391 fetch_fsmap learned FSMap version 15 2023-11-28T17:10:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.938+0000 7feb293ff740 1 -- 172.21.15.116:0/3484982730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x55700b106170 con 0x55700b1db990 2023-11-28T17:10:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.938+0000 7feafeffd640 1 -- 172.21.15.116:0/3484982730 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb100163a0 con 0x55700b1db990 2023-11-28T17:10:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.938+0000 7feafeffd640 1 -- 172.21.15.116:0/3484982730 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7feb10028480 con 0x55700b1db990 2023-11-28T17:10:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.938+0000 7feb293ff740 10 client.5391 fetch_fsmap finished waiting for FSMap version 15 2023-11-28T17:10:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.938+0000 7feb293ff740 10 client.5391 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T17:10:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.938+0000 7feb293ff740 20 client.5391 populate_metadata read hostname 'smithi116' 2023-11-28T17:10:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.938+0000 7feb293ff740 1 --2- 172.21.15.116:0/3484982730 >> [v2:172.21.15.116:6836/3177126349,v1:172.21.15.116:6837/3177126349] conn(0x55700b23c810 0x55700b23cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.938+0000 7feb293ff740 4 client.5391 mds_command: new command op to 5202 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T17:10:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.938+0000 7feb293ff740 1 -- 172.21.15.116:0/3484982730 --> [v2:172.21.15.116:6836/3177126349,v1:172.21.15.116:6837/3177126349] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55700ae456c0 con 0x55700b23c810 2023-11-28T17:10:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.938+0000 7feb17fff640 1 --2- 172.21.15.116:0/3484982730 >> [v2:172.21.15.116:6836/3177126349,v1:172.21.15.116:6837/3177126349] conn(0x55700b23c810 0x55700b23cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.939+0000 7feb17fff640 1 --2- 172.21.15.116:0/3484982730 >> [v2:172.21.15.116:6836/3177126349,v1:172.21.15.116:6837/3177126349] conn(0x55700b23c810 0x55700b23cbf0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=588a89e3fc9ce953 server_cookie=696fff3849fb2295 in_seq=0 out_seq=0 2023-11-28T17:10:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.939+0000 7feafeffd640 10 client.5391 ms_handle_connect on v2:172.21.15.116:6836/3177126349 2023-11-28T17:10:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.939+0000 7feafeffd640 1 -- 172.21.15.116:0/3484982730 <== mds.0 v2:172.21.15.116:6836/3177126349 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55700ae456c0 con 0x55700b23c810 2023-11-28T17:10:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.939+0000 7feafeffd640 10 client.5391 handle_command_reply: tid=0 2023-11-28T17:10:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.939+0000 7feb293ff740 1 client.5391 shutdown 2023-11-28T17:10:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.939+0000 7feb293ff740 1 -- 172.21.15.116:0/3484982730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b1db990 msgr2=0x55700b295fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.939+0000 7feb293ff740 1 --2- 172.21.15.116:0/3484982730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b1db990 0x55700b295fd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7feb10004000 tx=0x7feb10001080 comp rx=0 tx=0).stop 2023-11-28T17:10:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.939+0000 7feb293ff740 1 -- 172.21.15.116:0/3484982730 shutdown_connections 2023-11-28T17:10:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.939+0000 7feb293ff740 1 --2- 172.21.15.116:0/3484982730 >> [v2:172.21.15.116:6836/3177126349,v1:172.21.15.116:6837/3177126349] conn(0x55700b23c810 0x55700b23cbf0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.939+0000 7feb293ff740 1 --2- 172.21.15.116:0/3484982730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55700b210630 0x55700b204290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.939+0000 7feb293ff740 1 --2- 172.21.15.116:0/3484982730 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55700b207e10 0x55700b208db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.939+0000 7feafeffd640 0 client.5391 ms_handle_reset on v2:172.21.15.116:6836/3177126349 2023-11-28T17:10:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.939+0000 7feb293ff740 1 --2- 172.21.15.116:0/3484982730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55700b1db990 0x55700b295fd0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.939+0000 7feb293ff740 1 -- 172.21.15.116:0/3484982730 >> 172.21.15.116:0/3484982730 conn(0x55700b1d80c0 msgr2=0x55700b1f1930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.940+0000 7feb293ff740 1 -- 172.21.15.116:0/3484982730 shutdown_connections 2023-11-28T17:10:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.940+0000 7feb293ff740 1 -- 172.21.15.116:0/3484982730 wait complete. 2023-11-28T17:10:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.940+0000 7feb293ff740 20 client.5391 trim_cache size 0 max 16384 2023-11-28T17:10:26.951 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-28T17:10:26.951 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a5f2fbcf-fde4-4c38-94da-08178784d9fc", 2023-11-28T17:10:26.951 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2023-11-28T17:10:26.951 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 5202, 2023-11-28T17:10:26.951 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2023-11-28T17:10:26.951 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2023-11-28T17:10:26.952 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2023-11-28T17:10:26.952 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 6.6218221719999999, 2023-11-28T17:10:26.952 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 15, 2023-11-28T17:10:26.952 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 34, 2023-11-28T17:10:26.952 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 34, 2023-11-28T17:10:26.952 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 16.457699714 2023-11-28T17:10:26.952 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-28T17:10:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.940+0000 7feb26790640 1 -- 172.21.15.116:0/2579821823 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feb0006b2a0 msgr2=0x7feb0006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.940+0000 7feb26790640 1 --2- 172.21.15.116:0/2579821823 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feb0006b2a0 0x7feb0006d760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7feb10000f60 tx=0x7feb100079b0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.940+0000 7feb26790640 1 -- 172.21.15.116:0/2579821823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb200b3310 msgr2=0x7feb201ec540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.940+0000 7feb26790640 1 --2- 172.21.15.116:0/2579821823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb200b3310 0x7feb201ec540 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7feb0800c750 tx=0x7feb0800cc10 comp rx=0 tx=0).stop 2023-11-28T17:10:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.940+0000 7feb26790640 1 -- 172.21.15.116:0/2579821823 shutdown_connections 2023-11-28T17:10:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.940+0000 7feb26790640 1 --2- 172.21.15.116:0/2579821823 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feb0006b2a0 0x7feb0006d760 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.940+0000 7feb26790640 1 --2- 172.21.15.116:0/2579821823 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb2015dab0 0x7feb2014ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.940+0000 7feb26790640 1 --2- 172.21.15.116:0/2579821823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb200b3310 0x7feb201ec540 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.940+0000 7feb26790640 1 --2- 172.21.15.116:0/2579821823 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb2006cae0 0x7feb20155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:26.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.940+0000 7feb26790640 1 -- 172.21.15.116:0/2579821823 >> 172.21.15.116:0/2579821823 conn(0x7feb2006a4c0 msgr2=0x7feb2006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:26.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.941+0000 7feb26790640 1 -- 172.21.15.116:0/2579821823 shutdown_connections 2023-11-28T17:10:26.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:26.941+0000 7feb26790640 1 -- 172.21.15.116:0/2579821823 wait complete. 2023-11-28T17:10:26.958 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T17:10:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.147+0000 7f30101b0640 1 Processor -- start 2023-11-28T17:10:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.147+0000 7f30101b0640 1 -- start start 2023-11-28T17:10:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.148+0000 7f30101b0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f300813cee0 0x7f300806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.148+0000 7f30101b0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3008140200 0x7f300806c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.148+0000 7f30101b0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30080aea20 con 0x7f300815a490 2023-11-28T17:10:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.148+0000 7f30101b0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f300813d2c0 con 0x7f3008140200 2023-11-28T17:10:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.148+0000 7f30101b0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300815b3a0 con 0x7f300813cee0 2023-11-28T17:10:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.149+0000 7f300df4c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f300813cee0 0x7f300806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.149+0000 7f300df4c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f300813cee0 0x7f300806c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:47496/0 (socket says 172.21.15.116:47496) 2023-11-28T17:10:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.149+0000 7f300df4c640 1 -- 172.21.15.116:0/2267775615 learned_addr learned my addr 172.21.15.116:0/2267775615 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.149+0000 7f300d74b640 1 --2- 172.21.15.116:0/2267775615 >> v2:172.21.15.156:3300/0 conn(0x7f3008140200 0x7f300806c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.149+0000 7f300df4c640 1 -- 172.21.15.116:0/2267775615 >> v2:172.21.15.156:3300/0 conn(0x7f3008140200 msgr2=0x7f300806c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.149+0000 7f300df4c640 1 --2- 172.21.15.116:0/2267775615 >> v2:172.21.15.156:3300/0 conn(0x7f3008140200 0x7f300806c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.149+0000 7f300df4c640 1 -- 172.21.15.116:0/2267775615 >> v1:172.21.15.156:6789/0 conn(0x7f300815a490 legacy=0x7f3008071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:27.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.149+0000 7f300df4c640 1 -- 172.21.15.116:0/2267775615 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300815b0c0 con 0x7f300813cee0 2023-11-28T17:10:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.151+0000 7f300df4c640 1 --2- 172.21.15.116:0/2267775615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f300813cee0 0x7f300806c250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3004009870 tx=0x7f300402db00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f785d2cc879164a5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.151+0000 7f300cf4a640 1 -- 172.21.15.116:0/2267775615 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300402f8c0 con 0x7f300813cee0 2023-11-28T17:10:27.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.151+0000 7f300cf4a640 1 -- 172.21.15.116:0/2267775615 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300402fa20 con 0x7f300813cee0 2023-11-28T17:10:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.151+0000 7f300cf4a640 1 -- 172.21.15.116:0/2267775615 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300402fdb0 con 0x7f300813cee0 2023-11-28T17:10:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.152+0000 7f30101b0640 1 -- 172.21.15.116:0/2267775615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f300813cee0 msgr2=0x7f300806c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.152+0000 7f30101b0640 1 --2- 172.21.15.116:0/2267775615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f300813cee0 0x7f300806c250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3004009870 tx=0x7f300402db00 comp rx=0 tx=0).stop 2023-11-28T17:10:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.152+0000 7f30101b0640 1 -- 172.21.15.116:0/2267775615 shutdown_connections 2023-11-28T17:10:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.152+0000 7f30101b0640 1 --2- 172.21.15.116:0/2267775615 >> v2:172.21.15.156:3300/0 conn(0x7f3008140200 0x7f300806c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:27.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.152+0000 7f30101b0640 1 --2- 172.21.15.116:0/2267775615 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f300813cee0 0x7f300806c250 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.152+0000 7f30101b0640 1 -- 172.21.15.116:0/2267775615 >> 172.21.15.116:0/2267775615 conn(0x7f300806a4c0 msgr2=0x7f300806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.152+0000 7f30101b0640 1 -- 172.21.15.116:0/2267775615 shutdown_connections 2023-11-28T17:10:27.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.152+0000 7f30101b0640 1 -- 172.21.15.116:0/2267775615 wait complete. 2023-11-28T17:10:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.154+0000 7f30101b0640 1 Processor -- start 2023-11-28T17:10:27.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.154+0000 7f30101b0640 1 -- start start 2023-11-28T17:10:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.155+0000 7f30101b0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f300813cee0 0x7f300813dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.155+0000 7f30101b0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3008140200 0x7f3008159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.155+0000 7f30101b0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f300815a490 0x7f300814fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:27.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.155+0000 7f30101b0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3008071a80 con 0x7f3008140200 2023-11-28T17:10:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.155+0000 7f30101b0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3008070e00 con 0x7f300813cee0 2023-11-28T17:10:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.155+0000 7f30101b0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30080710e0 con 0x7f300815a490 2023-11-28T17:10:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.155+0000 7f300d74b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3008140200 0x7f3008159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.155+0000 7f300d74b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3008140200 0x7f3008159410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57684/0 (socket says 172.21.15.116:57684) 2023-11-28T17:10:27.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.155+0000 7f300d74b640 1 -- 172.21.15.116:0/4233331648 learned_addr learned my addr 172.21.15.116:0/4233331648 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:27.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.156+0000 7f300e74d640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f300815a490 0x7f300814fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:27.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.156+0000 7f300d74b640 1 -- 172.21.15.116:0/4233331648 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f300815a490 msgr2=0x7f300814fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:27.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.156+0000 7f300d74b640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f300815a490 0x7f300814fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:27.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.156+0000 7f300d74b640 1 -- 172.21.15.116:0/4233331648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f300813cee0 msgr2=0x7f300813dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:27.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.156+0000 7f300df4c640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f300813cee0 0x7f300813dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:27.158 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.156+0000 7f300d74b640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f300813cee0 0x7f300813dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.156+0000 7f300d74b640 1 -- 172.21.15.116:0/4233331648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30040094d0 con 0x7f3008140200 2023-11-28T17:10:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.156+0000 7f300e74d640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f300815a490 0x7f300814fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.156+0000 7f300d74b640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3008140200 0x7f3008159410 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800e9b0 tx=0x7f2ff800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.156+0000 7f2ffe7fc640 1 -- 172.21.15.116:0/4233331648 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8009670 con 0x7f3008140200 2023-11-28T17:10:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.156+0000 7f2ffe7fc640 1 -- 172.21.15.116:0/4233331648 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff800c9a0 con 0x7f3008140200 2023-11-28T17:10:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.156+0000 7f30101b0640 1 -- 172.21.15.116:0/4233331648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30080713c0 con 0x7f3008140200 2023-11-28T17:10:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.156+0000 7f2ffe7fc640 1 -- 172.21.15.116:0/4233331648 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff800cd50 con 0x7f3008140200 2023-11-28T17:10:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.156+0000 7f30101b0640 1 -- 172.21.15.116:0/4233331648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3008071ea0 con 0x7f3008140200 2023-11-28T17:10:27.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.158+0000 7f30101b0640 1 -- 172.21.15.116:0/4233331648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30080693d0 con 0x7f3008140200 2023-11-28T17:10:27.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.159+0000 7f2ffe7fc640 1 -- 172.21.15.116:0/4233331648 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2ff8013470 con 0x7f3008140200 2023-11-28T17:10:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.160+0000 7f2ffe7fc640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2fec06b2a0 0x7f2fec06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.160+0000 7f2ffe7fc640 1 -- 172.21.15.116:0/4233331648 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ff8089d50 con 0x7f3008140200 2023-11-28T17:10:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.160+0000 7f300df4c640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2fec06b2a0 0x7f2fec06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.161+0000 7f300df4c640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2fec06b2a0 0x7f2fec06d760 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3004038000 tx=0x7f3004008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.164+0000 7f2ffe7fc640 1 -- 172.21.15.116:0/4233331648 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ff808d030 con 0x7f3008140200 2023-11-28T17:10:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.392+0000 7f30101b0640 1 -- 172.21.15.116:0/4233331648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f300806d170 con 0x7f3008140200 2023-11-28T17:10:27.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.392+0000 7f2ffe7fc640 1 -- 172.21.15.116:0/4233331648 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2ff8054c50 con 0x7f3008140200 2023-11-28T17:10:27.394 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T17:10:27.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.395+0000 7f30101b0640 1 -- 172.21.15.116:0/4233331648 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2fec06b2a0 msgr2=0x7f2fec06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:27.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.395+0000 7f30101b0640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2fec06b2a0 0x7f2fec06d760 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3004038000 tx=0x7f3004008030 comp rx=0 tx=0).stop 2023-11-28T17:10:27.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.395+0000 7f30101b0640 1 -- 172.21.15.116:0/4233331648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3008140200 msgr2=0x7f3008159410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:27.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.395+0000 7f30101b0640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3008140200 0x7f3008159410 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800e9b0 tx=0x7f2ff800ee70 comp rx=0 tx=0).stop 2023-11-28T17:10:27.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.395+0000 7f30101b0640 1 -- 172.21.15.116:0/4233331648 shutdown_connections 2023-11-28T17:10:27.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.395+0000 7f30101b0640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2fec06b2a0 0x7f2fec06d760 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:27.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.395+0000 7f30101b0640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f300815a490 0x7f300814fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:27.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.395+0000 7f30101b0640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3008140200 0x7f3008159410 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:27.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.395+0000 7f30101b0640 1 --2- 172.21.15.116:0/4233331648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f300813cee0 0x7f300813dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:27.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.395+0000 7f30101b0640 1 -- 172.21.15.116:0/4233331648 >> 172.21.15.116:0/4233331648 conn(0x7f300806a4c0 msgr2=0x7f30080af950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:27.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.395+0000 7f30101b0640 1 -- 172.21.15.116:0/4233331648 shutdown_connections 2023-11-28T17:10:27.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:27.395+0000 7f30101b0640 1 -- 172.21.15.116:0/4233331648 wait complete. 2023-11-28T17:10:27.414 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T17:10:27.414 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T17:10:27.414 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2023-11-28T17:10:27.414 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T17:10:27.414 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:27.414 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T17:10:27.414 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T17:10:27.414 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T17:10:27.414 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T17:10:27.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:27.415 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2023-11-28T17:10:27.464 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T17:10:27.464 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T17:10:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T17:10:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2023-11-28T17:10:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:27.465 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:10:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2023-11-28T17:10:27.465 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:10:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:27.465 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2023-11-28T17:10:27.465 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:10:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:27.466 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T17:10:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T17:10:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42311sec preferred_lft 42311sec 2023-11-28T17:10:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2023-11-28T17:10:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:27.466 INFO:teuthology.orchestra.run.smithi116.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T17:10:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 66:db:2d:c1:d7:e7 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T17:10:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:27.466 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::c06c:35ff:fe15:dc8e/64 scope link 2023-11-28T17:10:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:27.467 INFO:teuthology.orchestra.run.smithi116.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T17:10:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 66:db:2d:c1:d7:e7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:10:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::64db:2dff:fec1:d7e7/64 scope link 2023-11-28T17:10:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:27.467 INFO:teuthology.orchestra.run.smithi116.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T17:10:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether c2:79:c2:c8:b7:18 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:10:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::c079:c2ff:fec8:b718/64 scope link 2023-11-28T17:10:27.467 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:27.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:27.468 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T17:10:27.546 INFO:teuthology.orchestra.run.smithi116.stdout:1 2023-11-28T17:10:27.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:27.550 DEBUG:teuthology.orchestra.run.smithi116:> route 2023-11-28T17:10:27.600 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2023-11-28T17:10:27.601 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T17:10:27.601 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T17:10:27.601 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T17:10:27.601 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T17:10:27.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:27.601 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-28T17:10:27.601 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-28T17:10:27.602 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T17:10:27.602 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T17:10:27.602 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T17:10:27.602 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-28T17:10:27.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:27.784 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2023-11-28T17:10:27.836 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T17:10:27.836 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T17:10:27.837 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T17:10:27.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:27.837 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2023-11-28T17:10:27.934 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-11-28T17:10:27.934 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:10:27.935 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:27.988 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T17:10:27.988 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:10:27.988 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:28.040 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-28T17:10:28.136 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:10:28.191 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2023-11-28T17:10:28.192 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2023-11-28T17:10:28.192 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2023-11-28T17:10:28.192 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2023-11-28T17:10:28.192 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2023-11-28T17:10:28.192 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2023-11-28T17:10:28.192 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2023-11-28T17:10:28.192 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2023-11-28T17:10:28.192 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2023-11-28T17:10:28.192 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2023-11-28T17:10:28.192 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2023-11-28T17:10:28.192 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:10:28.193 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2023-11-28T17:10:28.193 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2023-11-28T17:10:28.193 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2023-11-28T17:10:28.193 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2023-11-28T17:10:28.193 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2023-11-28T17:10:28.193 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2023-11-28T17:10:28.193 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:10:28.193 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:10:28.193 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:10:28.193 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:10:28.193 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2023-11-28T17:10:28.193 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2023-11-28T17:10:28.193 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T17:10:28.194 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T17:10:28.194 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T17:10:28.194 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T17:10:28.194 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2023-11-28T17:10:28.194 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:10:28.194 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:10:28.194 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:10:28.194 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:10:28.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:28.195 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-28T17:10:28.245 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-28T17:10:28.245 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-11-28T17:10:28.251 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-28T17:10:28.330 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:10:28.377 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2023-11-28T17:10:28.377 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2023-11-28T17:10:28.377 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2023-11-28T17:10:28.377 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2023-11-28T17:10:28.377 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2023-11-28T17:10:28.377 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2023-11-28T17:10:28.378 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2023-11-28T17:10:28.378 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2023-11-28T17:10:28.378 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2023-11-28T17:10:28.378 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2023-11-28T17:10:28.378 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2023-11-28T17:10:28.378 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:10:28.378 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2023-11-28T17:10:28.379 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2023-11-28T17:10:28.379 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2023-11-28T17:10:28.379 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2023-11-28T17:10:28.379 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2023-11-28T17:10:28.379 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2023-11-28T17:10:28.379 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:10:28.379 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:10:28.379 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:10:28.379 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:10:28.379 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2023-11-28T17:10:28.379 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2023-11-28T17:10:28.379 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T17:10:28.380 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T17:10:28.380 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T17:10:28.380 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T17:10:28.380 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2023-11-28T17:10:28.380 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:10:28.380 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:10:28.380 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:10:28.380 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:10:28.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:28.381 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-28T17:10:28.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:2023-11-28T17:10:28.436+0000 7fddabe35480 -1 init, newargv = 0x5592744d66f0 newargc=15 2023-11-28T17:10:28.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[102559]: starting ceph client 2023-11-28T17:10:28.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[102559]: starting fuse 2023-11-28T17:10:29.522 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-28T17:10:29.624 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:10:29.679 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2023-11-28T17:10:29.679 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2023-11-28T17:10:29.679 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2023-11-28T17:10:29.680 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2023-11-28T17:10:29.680 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2023-11-28T17:10:29.680 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2023-11-28T17:10:29.680 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2023-11-28T17:10:29.680 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2023-11-28T17:10:29.680 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2023-11-28T17:10:29.680 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2023-11-28T17:10:29.680 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2023-11-28T17:10:29.680 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:10:29.680 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2023-11-28T17:10:29.680 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2023-11-28T17:10:29.680 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2023-11-28T17:10:29.680 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2023-11-28T17:10:29.681 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2023-11-28T17:10:29.681 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2023-11-28T17:10:29.681 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:10:29.681 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:10:29.681 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:10:29.681 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:10:29.681 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2023-11-28T17:10:29.681 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2023-11-28T17:10:29.681 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T17:10:29.681 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T17:10:29.681 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T17:10:29.681 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T17:10:29.682 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2023-11-28T17:10:29.682 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:10:29.682 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:10:29.682 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:10:29.682 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:10:29.682 INFO:teuthology.orchestra.run.smithi116.stdout:/home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:29.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:29.683 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-28T17:10:29.738 INFO:teuthology.orchestra.run.smithi116.stdout:39 2023-11-28T17:10:29.738 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2023-11-28T17:10:29.739 DEBUG:teuthology.orchestra.run.smithi116:> sudo stdin-killer -- python3 -c ' 2023-11-28T17:10:29.739 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2023-11-28T17:10:29.739 DEBUG:teuthology.orchestra.run.smithi116:> import re 2023-11-28T17:10:29.739 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-28T17:10:29.739 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2023-11-28T17:10:29.739 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:10:29.739 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2023-11-28T17:10:29.739 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-11-28T17:10:29.739 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2023-11-28T17:10:29.739 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-11-28T17:10:29.739 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:10:29.740 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2023-11-28T17:10:29.740 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2023-11-28T17:10:29.740 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2023-11-28T17:10:29.740 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2023-11-28T17:10:29.740 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:10:29.740 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2023-11-28T17:10:29.740 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-28T17:10:29.740 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-28T17:10:29.740 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-28T17:10:29.740 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2023-11-28T17:10:29.740 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2023-11-28T17:10:29.740 DEBUG:teuthology.orchestra.run.smithi116:> return f 2023-11-28T17:10:29.740 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-28T17:10:29.740 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:10:29.741 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.0")) 2023-11-28T17:10:29.741 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-28T17:10:29.914 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.0.102559.asok 2023-11-28T17:10:29.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-11-28T17:10:29.934 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.102559.asok 2023-11-28T17:10:29.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:29.934 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.102559.asok status 2023-11-28T17:10:30.176 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-28T17:10:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2023-11-28T17:10:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "47c1c547372ed5bff7984647dc200bac0e019caf", 2023-11-28T17:10:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev)", 2023-11-28T17:10:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "0", 2023-11-28T17:10:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2023-11-28T17:10:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-11-28T17:10:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "102559", 2023-11-28T17:10:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2023-11-28T17:10:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2023-11-28T17:10:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2023-11-28T17:10:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 5403, 2023-11-28T17:10:30.177 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2023-11-28T17:10:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2023-11-28T17:10:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2023-11-28T17:10:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5403 2023-11-28T17:10:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-28T17:10:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-28T17:10:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-28T17:10:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 780885702 2023-11-28T17:10:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:30.178 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-28T17:10:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-28T17:10:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-28T17:10:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 780885702 2023-11-28T17:10:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.5403 192.168.0.1:0/780885702", 2023-11-28T17:10:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.1:0/780885702", 2023-11-28T17:10:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2023-11-28T17:10:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 15, 2023-11-28T17:10:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 34, 2023-11-28T17:10:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2023-11-28T17:10:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2023-11-28T17:10:30.179 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2023-11-28T17:10:30.180 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-28T17:10:30.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:30.191 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2023-11-28T17:10:30.191 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:30.241 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2023-11-28T17:10:30.241 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:30.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:30.242 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:30.368 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T17:10:30.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.575+0000 7f5720997640 1 Processor -- start 2023-11-28T17:10:30.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.575+0000 7f5720997640 1 -- start start 2023-11-28T17:10:30.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.577+0000 7f5720997640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f571c144430 0x7f571c15f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:30.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.577+0000 7f5720997640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f571c0703f0 0x7f571c071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:30.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.577+0000 7f5720997640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c070bd0 con 0x7f571c0721b0 2023-11-28T17:10:30.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.577+0000 7f5720997640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571c144810 con 0x7f571c0703f0 2023-11-28T17:10:30.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.577+0000 7f5720997640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571c070eb0 con 0x7f571c144430 2023-11-28T17:10:30.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.577+0000 7f571a59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f571c144430 0x7f571c15f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:30.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.577+0000 7f571a59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f571c144430 0x7f571c15f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:56074/0 (socket says 172.21.15.116:56074) 2023-11-28T17:10:30.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.577+0000 7f571a59c640 1 -- 172.21.15.116:0/395057029 learned_addr learned my addr 172.21.15.116:0/395057029 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.578+0000 7f5713fff640 1 --2- 172.21.15.116:0/395057029 >> v2:172.21.15.156:3300/0 conn(0x7f571c0703f0 0x7f571c071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.578+0000 7f571a59c640 1 -- 172.21.15.116:0/395057029 >> v2:172.21.15.156:3300/0 conn(0x7f571c0703f0 msgr2=0x7f571c071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.578+0000 7f571a59c640 1 --2- 172.21.15.116:0/395057029 >> v2:172.21.15.156:3300/0 conn(0x7f571c0703f0 0x7f571c071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.578+0000 7f571a59c640 1 -- 172.21.15.116:0/395057029 >> v1:172.21.15.156:6789/0 conn(0x7f571c0721b0 legacy=0x7f571c06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.578+0000 7f571a59c640 1 -- 172.21.15.116:0/395057029 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c071190 con 0x7f571c144430 2023-11-28T17:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.579+0000 7f571a59c640 1 --2- 172.21.15.116:0/395057029 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f571c144430 0x7f571c15f7b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f570c009870 tx=0x7f570c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85d7f822b469b64e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.580+0000 7f5719d9b640 1 -- 172.21.15.116:0/395057029 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c030840 con 0x7f571c144430 2023-11-28T17:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.580+0000 7f5719d9b640 1 -- 172.21.15.116:0/395057029 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c0309a0 con 0x7f571c144430 2023-11-28T17:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.580+0000 7f5719d9b640 1 -- 172.21.15.116:0/395057029 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c030d30 con 0x7f571c144430 2023-11-28T17:10:30.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.581+0000 7f5720997640 1 -- 172.21.15.116:0/395057029 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f571c144430 msgr2=0x7f571c15f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.581+0000 7f5720997640 1 --2- 172.21.15.116:0/395057029 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f571c144430 0x7f571c15f7b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f570c009870 tx=0x7f570c02b980 comp rx=0 tx=0).stop 2023-11-28T17:10:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.581+0000 7f5720997640 1 -- 172.21.15.116:0/395057029 shutdown_connections 2023-11-28T17:10:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.581+0000 7f5720997640 1 --2- 172.21.15.116:0/395057029 >> v2:172.21.15.156:3300/0 conn(0x7f571c0703f0 0x7f571c071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.581+0000 7f5720997640 1 --2- 172.21.15.116:0/395057029 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f571c144430 0x7f571c15f7b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.581+0000 7f5720997640 1 -- 172.21.15.116:0/395057029 >> 172.21.15.116:0/395057029 conn(0x7f571c06a4c0 msgr2=0x7f571c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.581+0000 7f5720997640 1 -- 172.21.15.116:0/395057029 shutdown_connections 2023-11-28T17:10:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.581+0000 7f5720997640 1 -- 172.21.15.116:0/395057029 wait complete. 2023-11-28T17:10:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.583+0000 7f5720997640 1 Processor -- start 2023-11-28T17:10:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.583+0000 7f5720997640 1 -- start start 2023-11-28T17:10:30.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.584+0000 7f5720997640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c0703f0 0x7f571c0c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:30.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.584+0000 7f5720997640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f571c0721b0 0x7f571c1f4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:30.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.584+0000 7f5720997640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f571c144430 0x7f571c0c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:30.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.584+0000 7f5720997640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0c7630 con 0x7f571c0703f0 2023-11-28T17:10:30.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.584+0000 7f5720997640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0b0390 con 0x7f571c0721b0 2023-11-28T17:10:30.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.584+0000 7f5720997640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571c1f5460 con 0x7f571c144430 2023-11-28T17:10:30.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.585+0000 7f571a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c0703f0 0x7f571c0c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:30.587 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.585+0000 7f571a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c0703f0 0x7f571c0c5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56922/0 (socket says 172.21.15.116:56922) 2023-11-28T17:10:30.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.585+0000 7f571a59c640 1 -- 172.21.15.116:0/224375708 learned_addr learned my addr 172.21.15.116:0/224375708 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:30.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.585+0000 7f5713fff640 1 --2- 172.21.15.116:0/224375708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f571c0721b0 0x7f571c1f4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:30.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.585+0000 7f571ad9d640 1 --2- 172.21.15.116:0/224375708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f571c144430 0x7f571c0c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:30.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.585+0000 7f571a59c640 1 -- 172.21.15.116:0/224375708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f571c144430 msgr2=0x7f571c0c9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:30.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.585+0000 7f571a59c640 1 --2- 172.21.15.116:0/224375708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f571c144430 0x7f571c0c9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:30.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.585+0000 7f571a59c640 1 -- 172.21.15.116:0/224375708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f571c0721b0 msgr2=0x7f571c1f4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:30.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.585+0000 7f571a59c640 1 --2- 172.21.15.116:0/224375708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f571c0721b0 0x7f571c1f4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:30.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.585+0000 7f571a59c640 1 -- 172.21.15.116:0/224375708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c0094d0 con 0x7f571c0703f0 2023-11-28T17:10:30.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.586+0000 7f571a59c640 1 --2- 172.21.15.116:0/224375708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c0703f0 0x7f571c0c5b70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f570c02b9c0 tx=0x7f570c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:30.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.586+0000 7f5712ffd640 1 -- 172.21.15.116:0/224375708 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c03e040 con 0x7f571c0703f0 2023-11-28T17:10:30.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.586+0000 7f5712ffd640 1 -- 172.21.15.116:0/224375708 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c039cb0 con 0x7f571c0703f0 2023-11-28T17:10:30.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.586+0000 7f5712ffd640 1 -- 172.21.15.116:0/224375708 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c044630 con 0x7f571c0703f0 2023-11-28T17:10:30.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.586+0000 7f5720997640 1 -- 172.21.15.116:0/224375708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571c0c6270 con 0x7f571c0703f0 2023-11-28T17:10:30.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.586+0000 7f5720997640 1 -- 172.21.15.116:0/224375708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571c0c6830 con 0x7f571c0703f0 2023-11-28T17:10:30.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.588+0000 7f5720997640 1 -- 172.21.15.116:0/224375708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571c0693d0 con 0x7f571c0703f0 2023-11-28T17:10:30.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.588+0000 7f5712ffd640 1 -- 172.21.15.116:0/224375708 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f570c043390 con 0x7f571c0703f0 2023-11-28T17:10:30.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.589+0000 7f5712ffd640 1 --2- 172.21.15.116:0/224375708 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f56fc06b2a0 0x7f56fc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.589+0000 7f5712ffd640 1 -- 172.21.15.116:0/224375708 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f570c0abee0 con 0x7f571c0703f0 2023-11-28T17:10:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.589+0000 7f5713fff640 1 --2- 172.21.15.116:0/224375708 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f56fc06b2a0 0x7f56fc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.590+0000 7f5713fff640 1 --2- 172.21.15.116:0/224375708 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f56fc06b2a0 0x7f56fc06d760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f571c1e49d0 tx=0x7f5704009190 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:30.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.595+0000 7f5712ffd640 1 -- 172.21.15.116:0/224375708 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f570c076f20 con 0x7f571c0703f0 2023-11-28T17:10:30.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.825+0000 7f5720997640 1 -- 172.21.15.116:0/224375708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f571c06d100 con 0x7f571c0703f0 2023-11-28T17:10:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.825+0000 7f5712ffd640 1 -- 172.21.15.116:0/224375708 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7f570c034070 con 0x7f571c0703f0 2023-11-28T17:10:30.828 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T17:10:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.828+0000 7f5720997640 1 -- 172.21.15.116:0/224375708 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f56fc06b2a0 msgr2=0x7f56fc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.828+0000 7f5720997640 1 --2- 172.21.15.116:0/224375708 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f56fc06b2a0 0x7f56fc06d760 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f571c1e49d0 tx=0x7f5704009190 comp rx=0 tx=0).stop 2023-11-28T17:10:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.828+0000 7f5720997640 1 -- 172.21.15.116:0/224375708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c0703f0 msgr2=0x7f571c0c5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.828+0000 7f5720997640 1 --2- 172.21.15.116:0/224375708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c0703f0 0x7f571c0c5b70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f570c02b9c0 tx=0x7f570c004730 comp rx=0 tx=0).stop 2023-11-28T17:10:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.828+0000 7f5720997640 1 -- 172.21.15.116:0/224375708 shutdown_connections 2023-11-28T17:10:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.828+0000 7f5720997640 1 --2- 172.21.15.116:0/224375708 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f56fc06b2a0 0x7f56fc06d760 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.828+0000 7f5720997640 1 --2- 172.21.15.116:0/224375708 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f571c144430 0x7f571c0c9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.828+0000 7f5720997640 1 --2- 172.21.15.116:0/224375708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f571c0721b0 0x7f571c1f4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.828+0000 7f5720997640 1 --2- 172.21.15.116:0/224375708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f571c0703f0 0x7f571c0c5b70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.828+0000 7f5720997640 1 -- 172.21.15.116:0/224375708 >> 172.21.15.116:0/224375708 conn(0x7f571c06a4c0 msgr2=0x7f571c15e060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.828+0000 7f5720997640 1 -- 172.21.15.116:0/224375708 shutdown_connections 2023-11-28T17:10:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:30.828+0000 7f5720997640 1 -- 172.21.15.116:0/224375708 wait complete. 2023-11-28T17:10:30.847 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T17:10:30.847 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T17:10:30.847 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi156.front.sepia.ceph.com 2023-11-28T17:10:30.848 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-28T17:10:30.848 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-28T17:10:30.848 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T17:10:30.848 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T17:10:30.848 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T17:10:30.848 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-28T17:10:30.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:30.848 DEBUG:teuthology.orchestra.run.smithi156:> ip addr 2023-11-28T17:10:30.907 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T17:10:30.908 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T17:10:30.908 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T17:10:30.908 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:30.908 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2023-11-28T17:10:30.908 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:30.909 INFO:teuthology.orchestra.run.smithi156.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:10:30.909 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c0 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:30.909 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f0 2023-11-28T17:10:30.909 INFO:teuthology.orchestra.run.smithi156.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:10:30.909 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c1 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:30.909 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f1 2023-11-28T17:10:30.909 INFO:teuthology.orchestra.run.smithi156.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:10:30.909 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fc brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:30.909 INFO:teuthology.orchestra.run.smithi156.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T17:10:30.910 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fd brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:30.910 INFO:teuthology.orchestra.run.smithi156.stdout: inet 172.21.15.156/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T17:10:30.910 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft 42305sec preferred_lft 42305sec 2023-11-28T17:10:30.910 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::ec4:7aff:fe88:6dfd/64 scope link 2023-11-28T17:10:30.910 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:30.910 INFO:teuthology.orchestra.run.smithi156.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T17:10:30.910 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 4a:53:60:71:42:cd brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:30.910 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T17:10:30.910 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:30.910 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::54f0:12ff:fe79:1e2c/64 scope link 2023-11-28T17:10:30.910 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:30.910 INFO:teuthology.orchestra.run.smithi156.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T17:10:30.911 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether e6:68:69:2b:d0:94 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:10:30.911 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::e468:69ff:fe2b:d094/64 scope link 2023-11-28T17:10:30.911 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:30.911 INFO:teuthology.orchestra.run.smithi156.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T17:10:30.911 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 4a:53:60:71:42:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:10:30.911 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::4853:60ff:fe71:42cd/64 scope link 2023-11-28T17:10:30.911 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:30.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:30.912 DEBUG:teuthology.orchestra.run.smithi156:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T17:10:31.001 INFO:teuthology.orchestra.run.smithi156.stdout:1 2023-11-28T17:10:31.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:31.005 DEBUG:teuthology.orchestra.run.smithi156:> route 2023-11-28T17:10:31.058 INFO:teuthology.orchestra.run.smithi156.stdout:Kernel IP routing table 2023-11-28T17:10:31.058 INFO:teuthology.orchestra.run.smithi156.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T17:10:31.058 INFO:teuthology.orchestra.run.smithi156.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T17:10:31.058 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T17:10:31.058 INFO:teuthology.orchestra.run.smithi156.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T17:10:31.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:31.059 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2023-11-28T17:10:31.059 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2023-11-28T17:10:31.059 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T17:10:31.059 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T17:10:31.059 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T17:10:31.059 DEBUG:teuthology.orchestra.run.smithi156:> ') 2023-11-28T17:10:31.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:31.243 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2023-11-28T17:10:31.295 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-28T17:10:31.295 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-28T17:10:31.295 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-28T17:10:31.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:31.295 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set brx.0 up 2023-11-28T17:10:31.393 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-11-28T17:10:31.393 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:10:31.393 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-28T17:10:31.447 INFO:teuthology.orchestra.run.smithi156.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-28T17:10:31.447 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:10:31.447 DEBUG:teuthology.orchestra.run.smithi156:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-28T17:10:31.496 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2023-11-28T17:10:31.589 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:10:31.647 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2023-11-28T17:10:31.647 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2023-11-28T17:10:31.647 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2023-11-28T17:10:31.647 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2023-11-28T17:10:31.647 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2023-11-28T17:10:31.647 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2023-11-28T17:10:31.648 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2023-11-28T17:10:31.648 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2023-11-28T17:10:31.648 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2023-11-28T17:10:31.648 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2023-11-28T17:10:31.648 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2023-11-28T17:10:31.648 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:10:31.648 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2023-11-28T17:10:31.649 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2023-11-28T17:10:31.649 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2023-11-28T17:10:31.649 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2023-11-28T17:10:31.649 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2023-11-28T17:10:31.649 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2023-11-28T17:10:31.649 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:10:31.650 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:10:31.650 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:10:31.650 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:10:31.650 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2023-11-28T17:10:31.650 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2023-11-28T17:10:31.650 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2023-11-28T17:10:31.650 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2023-11-28T17:10:31.651 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2023-11-28T17:10:31.651 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2023-11-28T17:10:31.651 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2023-11-28T17:10:31.651 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:10:31.651 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:10:31.651 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:10:31.652 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:10:31.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:31.653 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2023-11-28T17:10:31.706 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-28T17:10:31.706 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-11-28T17:10:31.712 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2023-11-28T17:10:31.788 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:10:31.829 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2023-11-28T17:10:31.829 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2023-11-28T17:10:31.829 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2023-11-28T17:10:31.829 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2023-11-28T17:10:31.829 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2023-11-28T17:10:31.829 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2023-11-28T17:10:31.829 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2023-11-28T17:10:31.829 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2023-11-28T17:10:31.830 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2023-11-28T17:10:31.830 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2023-11-28T17:10:31.830 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2023-11-28T17:10:31.830 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:10:31.830 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2023-11-28T17:10:31.830 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2023-11-28T17:10:31.830 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2023-11-28T17:10:31.830 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2023-11-28T17:10:31.830 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2023-11-28T17:10:31.830 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2023-11-28T17:10:31.830 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:10:31.830 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:10:31.830 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:10:31.830 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:10:31.831 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2023-11-28T17:10:31.831 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2023-11-28T17:10:31.831 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2023-11-28T17:10:31.831 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2023-11-28T17:10:31.831 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2023-11-28T17:10:31.831 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2023-11-28T17:10:31.831 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2023-11-28T17:10:31.831 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:10:31.831 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:10:31.831 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:10:31.831 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:10:31.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:31.832 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2023-11-28T17:10:31.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2023-11-28T17:10:31.894+0000 7f7df5a8f480 -1 init, newargv = 0x55f30813a830 newargc=15 2023-11-28T17:10:31.896 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[94394]: starting ceph client 2023-11-28T17:10:31.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[94394]: starting fuse 2023-11-28T17:10:31.921 INFO:teuthology.orchestra.run.smithi156.stdout:39 2023-11-28T17:10:31.921 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2023-11-28T17:10:31.921 DEBUG:teuthology.orchestra.run.smithi156:> sudo stdin-killer -- python3 -c ' 2023-11-28T17:10:31.921 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2023-11-28T17:10:31.921 DEBUG:teuthology.orchestra.run.smithi156:> import re 2023-11-28T17:10:31.922 DEBUG:teuthology.orchestra.run.smithi156:> import os 2023-11-28T17:10:31.922 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2023-11-28T17:10:31.922 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:10:31.922 DEBUG:teuthology.orchestra.run.smithi156:> def _find_admin_socket(client_name): 2023-11-28T17:10:31.922 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-11-28T17:10:31.922 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2023-11-28T17:10:31.922 DEBUG:teuthology.orchestra.run.smithi156:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-11-28T17:10:31.922 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:10:31.922 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2023-11-28T17:10:31.922 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2023-11-28T17:10:31.922 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2023-11-28T17:10:31.922 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2023-11-28T17:10:31.923 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:10:31.923 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2023-11-28T17:10:31.923 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-28T17:10:31.923 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-28T17:10:31.923 DEBUG:teuthology.orchestra.run.smithi156:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-28T17:10:31.923 DEBUG:teuthology.orchestra.run.smithi156:> contents = proc_f.read() 2023-11-28T17:10:31.923 DEBUG:teuthology.orchestra.run.smithi156:> if mountpoint in contents: 2023-11-28T17:10:31.923 DEBUG:teuthology.orchestra.run.smithi156:> return f 2023-11-28T17:10:31.923 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-28T17:10:31.923 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:10:31.923 DEBUG:teuthology.orchestra.run.smithi156:> print(_find_admin_socket("client.1")) 2023-11-28T17:10:31.923 DEBUG:teuthology.orchestra.run.smithi156:> ' 2023-11-28T17:10:32.121 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.94394.asok 2023-11-28T17:10:32.125 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:10:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-11-28T17:10:32.138 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.94394.asok 2023-11-28T17:10:32.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:32.138 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.94394.asok status 2023-11-28T17:10:32.378 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-11-28T17:10:32.378 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2023-11-28T17:10:32.378 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "47c1c547372ed5bff7984647dc200bac0e019caf", 2023-11-28T17:10:32.378 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev)", 2023-11-28T17:10:32.378 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2023-11-28T17:10:32.378 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2023-11-28T17:10:32.378 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-11-28T17:10:32.378 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "94394", 2023-11-28T17:10:32.378 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2023-11-28T17:10:32.378 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:10:32.379 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2023-11-28T17:10:32.379 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2023-11-28T17:10:32.379 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 4679, 2023-11-28T17:10:32.379 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2023-11-28T17:10:32.379 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2023-11-28T17:10:32.379 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2023-11-28T17:10:32.379 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 4679 2023-11-28T17:10:32.379 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:10:32.379 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2023-11-28T17:10:32.379 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2023-11-28T17:10:32.379 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2023-11-28T17:10:32.379 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2523792801 2023-11-28T17:10:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-28T17:10:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:10:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2023-11-28T17:10:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2023-11-28T17:10:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2023-11-28T17:10:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 2523792801 2023-11-28T17:10:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:10:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.4679 192.168.0.1:0/2523792801", 2023-11-28T17:10:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "192.168.0.1:0/2523792801", 2023-11-28T17:10:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2023-11-28T17:10:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 15, 2023-11-28T17:10:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 34, 2023-11-28T17:10:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2023-11-28T17:10:32.380 INFO:teuthology.orchestra.run.smithi156.stdout: "blocklisted": false, 2023-11-28T17:10:32.381 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2023-11-28T17:10:32.381 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-11-28T17:10:32.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:32.392 DEBUG:teuthology.orchestra.run.smithi156:> stat --file-system '--printf=%T 2023-11-28T17:10:32.393 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1 2023-11-28T17:10:32.450 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2023-11-28T17:10:32.450 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-11-28T17:10:32.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:32.451 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-28T17:10:32.545 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds ms_async_reap_threshold 1 2023-11-28T17:10:32.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.769+0000 7fd31b215640 1 Processor -- start 2023-11-28T17:10:32.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.769+0000 7fd31b215640 1 -- start start 2023-11-28T17:10:32.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.771+0000 7fd31b215640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd3140b4130 0x7fd31406d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:32.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.771+0000 7fd31b215640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3140b46a0 con 0x7fd314160920 2023-11-28T17:10:32.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.771+0000 7fd31b215640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31406cb60 con 0x7fd31406bbe0 2023-11-28T17:10:32.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.771+0000 7fd31b215640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3140b3850 con 0x7fd3140b4130 2023-11-28T17:10:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.771+0000 7fd318fb1640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd314160920 0x7fd31406b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57232/0 (socket says 172.21.15.116:57232) 2023-11-28T17:10:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.771+0000 7fd318fb1640 1 -- 172.21.15.116:0/4142471956 learned_addr learned my addr 172.21.15.116:0/4142471956 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:32.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3197b2640 1 --2- 172.21.15.116:0/4142471956 >> v2:172.21.15.156:3300/0 conn(0x7fd3140b4130 0x7fd31406d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 -- 172.21.15.116:0/4142471956 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500337070 0 0) 0x7fd3140b46a0 con 0x7fd314160920 2023-11-28T17:10:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 -- 172.21.15.116:0/4142471956 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd304003520 con 0x7fd314160920 2023-11-28T17:10:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 -- 172.21.15.116:0/4142471956 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1105327249 0 0) 0x7fd304003520 con 0x7fd314160920 2023-11-28T17:10:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 -- 172.21.15.116:0/4142471956 >> v2:172.21.15.156:3300/0 conn(0x7fd3140b4130 msgr2=0x7fd31406d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 --2- 172.21.15.116:0/4142471956 >> v2:172.21.15.156:3300/0 conn(0x7fd3140b4130 0x7fd31406d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 -- 172.21.15.116:0/4142471956 >> v1:172.21.15.156:6789/0 conn(0x7fd31406bbe0 legacy=0x7fd3140b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 -- 172.21.15.116:0/4142471956 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31406ce40 con 0x7fd314160920 2023-11-28T17:10:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 -- 172.21.15.116:0/4142471956 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fd308001670 con 0x7fd314160920 2023-11-28T17:10:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 -- 172.21.15.116:0/4142471956 >> v1:172.21.15.116:6789/0 conn(0x7fd314160920 legacy=0x7fd31406b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 --2- 172.21.15.116:0/4142471956 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd3040039c0 0x7fd304003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 --2- 172.21.15.116:0/4142471956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3040047c0 0x7fd304004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:32.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 --2- 172.21.15.116:0/4142471956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd304005180 0x7fd30400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:32.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 -- 172.21.15.116:0/4142471956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30400c050 con 0x7fd3040047c0 2023-11-28T17:10:32.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 -- 172.21.15.116:0/4142471956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30400c1c0 con 0x7fd304005180 2023-11-28T17:10:32.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.772+0000 7fd3137fe640 1 -- 172.21.15.116:0/4142471956 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30400c380 con 0x7fd3040039c0 2023-11-28T17:10:32.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.773+0000 7fd313fff640 1 --2- 172.21.15.116:0/4142471956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3040047c0 0x7fd304004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:32.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.773+0000 7fd3197b2640 1 --2- 172.21.15.116:0/4142471956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd304005180 0x7fd30400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:32.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.773+0000 7fd313fff640 1 -- 172.21.15.116:0/4142471956 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd3040039c0 msgr2=0x7fd304003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:32.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.773+0000 7fd313fff640 1 --2- 172.21.15.116:0/4142471956 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd3040039c0 0x7fd304003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:32.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.773+0000 7fd313fff640 1 -- 172.21.15.116:0/4142471956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd304005180 msgr2=0x7fd30400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:32.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.773+0000 7fd313fff640 1 --2- 172.21.15.116:0/4142471956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd304005180 0x7fd30400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:32.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.773+0000 7fd313fff640 1 -- 172.21.15.116:0/4142471956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd304003520 con 0x7fd3040047c0 2023-11-28T17:10:32.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.773+0000 7fd3197b2640 1 -- 172.21.15.116:0/4142471956 reap_dead start 2023-11-28T17:10:32.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.774+0000 7fd313fff640 1 --2- 172.21.15.116:0/4142471956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3040047c0 0x7fd304004c40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd300001750 tx=0x7fd3040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41fb7a06c223e3a8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:32.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.774+0000 7fd3137fe640 1 -- 172.21.15.116:0/4142471956 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3000052d0 con 0x7fd3040047c0 2023-11-28T17:10:32.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.775+0000 7fd3137fe640 1 -- 172.21.15.116:0/4142471956 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30002e050 con 0x7fd3040047c0 2023-11-28T17:10:32.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.775+0000 7fd3137fe640 1 -- 172.21.15.116:0/4142471956 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd300003830 con 0x7fd3040047c0 2023-11-28T17:10:32.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.775+0000 7fd31b215640 1 -- 172.21.15.116:0/4142471956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3040047c0 msgr2=0x7fd304004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:32.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.775+0000 7fd31b215640 1 --2- 172.21.15.116:0/4142471956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3040047c0 0x7fd304004c40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd300001750 tx=0x7fd3040057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:32.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.775+0000 7fd31b215640 1 -- 172.21.15.116:0/4142471956 shutdown_connections 2023-11-28T17:10:32.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.775+0000 7fd31b215640 1 --2- 172.21.15.116:0/4142471956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3040047c0 0x7fd304004c40 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:32.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.775+0000 7fd31b215640 1 -- 172.21.15.116:0/4142471956 >> 172.21.15.116:0/4142471956 conn(0x7fd3140b1b90 msgr2=0x7fd314070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:32.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.775+0000 7fd31b215640 1 -- 172.21.15.116:0/4142471956 shutdown_connections 2023-11-28T17:10:32.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.775+0000 7fd31b215640 1 -- 172.21.15.116:0/4142471956 wait complete. 2023-11-28T17:10:32.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.777+0000 7fd31b215640 1 Processor -- start 2023-11-28T17:10:32.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.777+0000 7fd31b215640 1 -- start start 2023-11-28T17:10:32.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.778+0000 7fd31b215640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd3040047c0 0x7fd314149180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.778+0000 7fd31b215640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3140ce460 0x7fd31414a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.778+0000 7fd31b215640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3140ce8e0 0x7fd31406acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.778+0000 7fd31b215640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd314153e90 con 0x7fd3140ce8e0 2023-11-28T17:10:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.778+0000 7fd31b215640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3141496c0 con 0x7fd3140ce460 2023-11-28T17:10:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.778+0000 7fd31b215640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3141540e0 con 0x7fd3040047c0 2023-11-28T17:10:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.779+0000 7fd3197b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3140ce8e0 0x7fd31406acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.779+0000 7fd3197b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3140ce8e0 0x7fd31406acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56934/0 (socket says 172.21.15.116:56934) 2023-11-28T17:10:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.779+0000 7fd3197b2640 1 -- 172.21.15.116:0/1997171904 learned_addr learned my addr 172.21.15.116:0/1997171904 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.779+0000 7fd318fb1640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd3040047c0 0x7fd314149180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.779+0000 7fd313fff640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3140ce460 0x7fd31414a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.779+0000 7fd3197b2640 1 -- 172.21.15.116:0/1997171904 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd3040047c0 msgr2=0x7fd314149180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.779+0000 7fd3197b2640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd3040047c0 0x7fd314149180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.779+0000 7fd3197b2640 1 -- 172.21.15.116:0/1997171904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3140ce460 msgr2=0x7fd31414a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.779+0000 7fd3197b2640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3140ce460 0x7fd31414a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.779+0000 7fd3197b2640 1 -- 172.21.15.116:0/1997171904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3141543c0 con 0x7fd3140ce8e0 2023-11-28T17:10:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.779+0000 7fd313fff640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3140ce460 0x7fd31414a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.779+0000 7fd3197b2640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3140ce8e0 0x7fd31406acc0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd314070a00 tx=0x7fd30c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.780+0000 7fd3117fa640 1 -- 172.21.15.116:0/1997171904 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c016420 con 0x7fd3140ce8e0 2023-11-28T17:10:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.780+0000 7fd3117fa640 1 -- 172.21.15.116:0/1997171904 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c016580 con 0x7fd3140ce8e0 2023-11-28T17:10:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.780+0000 7fd31b215640 1 -- 172.21.15.116:0/1997171904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3141546a0 con 0x7fd3140ce8e0 2023-11-28T17:10:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.780+0000 7fd3117fa640 1 -- 172.21.15.116:0/1997171904 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c016930 con 0x7fd3140ce8e0 2023-11-28T17:10:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.780+0000 7fd31b215640 1 -- 172.21.15.116:0/1997171904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd314154c60 con 0x7fd3140ce8e0 2023-11-28T17:10:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.781+0000 7fd31b215640 1 -- 172.21.15.116:0/1997171904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd314069360 con 0x7fd3140ce8e0 2023-11-28T17:10:32.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.782+0000 7fd3117fa640 1 -- 172.21.15.116:0/1997171904 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fd30c016a90 con 0x7fd3140ce8e0 2023-11-28T17:10:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.783+0000 7fd3117fa640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd2f806b2a0 0x7fd2f806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.783+0000 7fd3117fa640 1 -- 172.21.15.116:0/1997171904 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd30c08ab30 con 0x7fd3140ce8e0 2023-11-28T17:10:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.783+0000 7fd318fb1640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd2f806b2a0 0x7fd2f806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.784+0000 7fd318fb1640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd2f806b2a0 0x7fd2f806d760 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd308001860 tx=0x7fd308001690 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:32.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.788+0000 7fd3117fa640 1 -- 172.21.15.116:0/1997171904 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd30c0559a0 con 0x7fd3140ce8e0 2023-11-28T17:10:32.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.977+0000 7fd31b215640 1 -- 172.21.15.116:0/1997171904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=ms_async_reap_threshold}] v 0) v1 -- 0x7fd314071ef0 con 0x7fd3140ce8e0 2023-11-28T17:10:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.984+0000 7fd3117fa640 1 -- 172.21.15.116:0/1997171904 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=ms_async_reap_threshold}]=0 v11)=0 v11) v1 ==== 121+0+0 (secure 0 0 0) 0x7fd30c058fc0 con 0x7fd3140ce8e0 2023-11-28T17:10:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.987+0000 7fd31b215640 1 -- 172.21.15.116:0/1997171904 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd2f806b2a0 msgr2=0x7fd2f806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.987+0000 7fd31b215640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd2f806b2a0 0x7fd2f806d760 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd308001860 tx=0x7fd308001690 comp rx=0 tx=0).stop 2023-11-28T17:10:32.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.987+0000 7fd31b215640 1 -- 172.21.15.116:0/1997171904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3140ce8e0 msgr2=0x7fd31406acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.987+0000 7fd31b215640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3140ce8e0 0x7fd31406acc0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd314070a00 tx=0x7fd30c00f520 comp rx=0 tx=0).stop 2023-11-28T17:10:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.987+0000 7fd31b215640 1 -- 172.21.15.116:0/1997171904 shutdown_connections 2023-11-28T17:10:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.987+0000 7fd31b215640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd2f806b2a0 0x7fd2f806d760 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.987+0000 7fd31b215640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3140ce8e0 0x7fd31406acc0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.987+0000 7fd31b215640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3140ce460 0x7fd31414a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.987+0000 7fd31b215640 1 --2- 172.21.15.116:0/1997171904 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd3040047c0 0x7fd314149180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.987+0000 7fd31b215640 1 -- 172.21.15.116:0/1997171904 >> 172.21.15.116:0/1997171904 conn(0x7fd3140b1b90 msgr2=0x7fd31415cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.987+0000 7fd31b215640 1 -- 172.21.15.116:0/1997171904 shutdown_connections 2023-11-28T17:10:32.989 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:32.987+0000 7fd31b215640 1 -- 172.21.15.116:0/1997171904 wait complete. 2023-11-28T17:10:33.007 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:10:33.008 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:10:33.057 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:10:33.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.316+0000 7f2327264640 1 Processor -- start 2023-11-28T17:10:33.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.316+0000 7f2327264640 1 -- start start 2023-11-28T17:10:33.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.317+0000 7f2327264640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2320144430 0x7f232015f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:33.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.317+0000 7f2327264640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f232006c250 0x7f232006c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:33.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.317+0000 7f2327264640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2320070bd0 con 0x7f23200703f0 2023-11-28T17:10:33.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.317+0000 7f2327264640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2320144810 con 0x7f232006c250 2023-11-28T17:10:33.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.317+0000 7f2327264640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f232006cc20 con 0x7f2320144430 2023-11-28T17:10:33.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.318+0000 7f2325801640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f232006c250 0x7f232006c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:33.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.318+0000 7f231ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f23200703f0 0x7f2320071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57238/0 (socket says 172.21.15.116:57238) 2023-11-28T17:10:33.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.318+0000 7f231ffff640 1 -- 172.21.15.116:0/408378100 learned_addr learned my addr 172.21.15.116:0/408378100 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:33.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.318+0000 7f2325000640 1 --2- 172.21.15.116:0/408378100 >> v2:172.21.15.156:3300/0 conn(0x7f2320144430 0x7f232015f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:33.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.318+0000 7f2325801640 1 -- 172.21.15.116:0/408378100 >> v2:172.21.15.156:3300/0 conn(0x7f2320144430 msgr2=0x7f232015f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:33.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.318+0000 7f2325801640 1 --2- 172.21.15.116:0/408378100 >> v2:172.21.15.156:3300/0 conn(0x7f2320144430 0x7f232015f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.318+0000 7f2325801640 1 -- 172.21.15.116:0/408378100 >> v1:172.21.15.116:6789/0 conn(0x7f23200703f0 legacy=0x7f2320071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:33.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.318+0000 7f2325801640 1 -- 172.21.15.116:0/408378100 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2320070eb0 con 0x7f232006c250 2023-11-28T17:10:33.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.318+0000 7f231f7fe640 1 -- 172.21.15.116:0/408378100 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236070233 0 0) 0x7f2320070bd0 con 0x7f23200703f0 2023-11-28T17:10:33.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.320+0000 7f2325801640 1 --2- 172.21.15.116:0/408378100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232006c250 0x7f232006c6e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f232006e850 tx=0x7f231802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c0f09726e63951 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:33.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.320+0000 7f231f7fe640 1 -- 172.21.15.116:0/408378100 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231802cc80 con 0x7f232006c250 2023-11-28T17:10:33.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.320+0000 7f231f7fe640 1 -- 172.21.15.116:0/408378100 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231802cde0 con 0x7f232006c250 2023-11-28T17:10:33.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.320+0000 7f231f7fe640 1 -- 172.21.15.116:0/408378100 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2318034660 con 0x7f232006c250 2023-11-28T17:10:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.320+0000 7f2327264640 1 -- 172.21.15.116:0/408378100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232006c250 msgr2=0x7f232006c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.320+0000 7f2327264640 1 --2- 172.21.15.116:0/408378100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232006c250 0x7f232006c6e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f232006e850 tx=0x7f231802bed0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.320+0000 7f2327264640 1 -- 172.21.15.116:0/408378100 shutdown_connections 2023-11-28T17:10:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.320+0000 7f2327264640 1 --2- 172.21.15.116:0/408378100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232006c250 0x7f232006c6e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.320+0000 7f2327264640 1 --2- 172.21.15.116:0/408378100 >> v2:172.21.15.156:3300/0 conn(0x7f2320144430 0x7f232015f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.320+0000 7f2327264640 1 -- 172.21.15.116:0/408378100 >> 172.21.15.116:0/408378100 conn(0x7f232006a4c0 msgr2=0x7f232006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.321+0000 7f2327264640 1 -- 172.21.15.116:0/408378100 shutdown_connections 2023-11-28T17:10:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.321+0000 7f2327264640 1 -- 172.21.15.116:0/408378100 wait complete. 2023-11-28T17:10:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.322+0000 7f2327264640 1 Processor -- start 2023-11-28T17:10:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.322+0000 7f2327264640 1 -- start start 2023-11-28T17:10:33.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.323+0000 7f2327264640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232006c250 0x7f23200c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:33.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.323+0000 7f2327264640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23200703f0 0x7f23200c3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:33.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.323+0000 7f2327264640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2320144430 0x7f232014b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:33.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.323+0000 7f2327264640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23201e55b0 con 0x7f232006c250 2023-11-28T17:10:33.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.323+0000 7f2327264640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23201e5890 con 0x7f23200703f0 2023-11-28T17:10:33.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.323+0000 7f2327264640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23201e5b70 con 0x7f2320144430 2023-11-28T17:10:33.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.324+0000 7f2325000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232006c250 0x7f23200c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:33.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.324+0000 7f2325000640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232006c250 0x7f23200c91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56954/0 (socket says 172.21.15.116:56954) 2023-11-28T17:10:33.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.324+0000 7f2325000640 1 -- 172.21.15.116:0/107092627 learned_addr learned my addr 172.21.15.116:0/107092627 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:33.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.324+0000 7f231ffff640 1 --2- 172.21.15.116:0/107092627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23200703f0 0x7f23200c3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:33.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.324+0000 7f2325000640 1 -- 172.21.15.116:0/107092627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2320144430 msgr2=0x7f232014b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:33.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.324+0000 7f2325000640 1 --2- 172.21.15.116:0/107092627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2320144430 0x7f232014b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.324+0000 7f2325000640 1 -- 172.21.15.116:0/107092627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23200703f0 msgr2=0x7f23200c3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:33.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.324+0000 7f2325000640 1 --2- 172.21.15.116:0/107092627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23200703f0 0x7f23200c3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.324+0000 7f2325000640 1 -- 172.21.15.116:0/107092627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2318009a70 con 0x7f232006c250 2023-11-28T17:10:33.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.325+0000 7f2325000640 1 --2- 172.21.15.116:0/107092627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232006c250 0x7f23200c91c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f231400d800 tx=0x7f231400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:33.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.325+0000 7f231d7fa640 1 -- 172.21.15.116:0/107092627 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231400b6c0 con 0x7f232006c250 2023-11-28T17:10:33.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.325+0000 7f231d7fa640 1 -- 172.21.15.116:0/107092627 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231400b820 con 0x7f232006c250 2023-11-28T17:10:33.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.325+0000 7f231d7fa640 1 -- 172.21.15.116:0/107092627 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231400bbb0 con 0x7f232006c250 2023-11-28T17:10:33.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.325+0000 7f2327264640 1 -- 172.21.15.116:0/107092627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23201e5e50 con 0x7f232006c250 2023-11-28T17:10:33.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.325+0000 7f2327264640 1 -- 172.21.15.116:0/107092627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23201e5160 con 0x7f232006c250 2023-11-28T17:10:33.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.326+0000 7f2327264640 1 -- 172.21.15.116:0/107092627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23200693d0 con 0x7f232006c250 2023-11-28T17:10:33.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.327+0000 7f231d7fa640 1 -- 172.21.15.116:0/107092627 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f231400bd10 con 0x7f232006c250 2023-11-28T17:10:33.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.328+0000 7f231d7fa640 1 --2- 172.21.15.116:0/107092627 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f230406b2a0 0x7f230406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.328+0000 7f231d7fa640 1 -- 172.21.15.116:0/107092627 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f231408a1d0 con 0x7f232006c250 2023-11-28T17:10:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.329+0000 7f231ffff640 1 --2- 172.21.15.116:0/107092627 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f230406b2a0 0x7f230406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.329+0000 7f231ffff640 1 --2- 172.21.15.116:0/107092627 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f230406b2a0 0x7f230406d760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f23200c9d50 tx=0x7f230c007de0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.332+0000 7f231d7fa640 1 -- 172.21.15.116:0/107092627 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2314054f10 con 0x7f232006c250 2023-11-28T17:10:33.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.527+0000 7f2327264640 1 -- 172.21.15.116:0/107092627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23201e52f0 con 0x7f232006c250 2023-11-28T17:10:33.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.528+0000 7f231d7fa640 1 -- 172.21.15.116:0/107092627 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16521 (secure 0 0 0) 0x7f2314058530 con 0x7f232006c250 2023-11-28T17:10:33.530 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:33.530 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":34,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:10:21.316243+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:10:17.338444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-28T17:10:18.766338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:10:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.531+0000 7f2327264640 1 -- 172.21.15.116:0/107092627 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f230406b2a0 msgr2=0x7f230406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.531+0000 7f2327264640 1 --2- 172.21.15.116:0/107092627 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f230406b2a0 0x7f230406d760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f23200c9d50 tx=0x7f230c007de0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.531+0000 7f2327264640 1 -- 172.21.15.116:0/107092627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232006c250 msgr2=0x7f23200c91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.531+0000 7f2327264640 1 --2- 172.21.15.116:0/107092627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232006c250 0x7f23200c91c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f231400d800 tx=0x7f231400dcc0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.532+0000 7f2327264640 1 -- 172.21.15.116:0/107092627 shutdown_connections 2023-11-28T17:10:33.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.532+0000 7f2327264640 1 --2- 172.21.15.116:0/107092627 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f230406b2a0 0x7f230406d760 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.532+0000 7f2327264640 1 --2- 172.21.15.116:0/107092627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2320144430 0x7f232014b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.532+0000 7f2327264640 1 --2- 172.21.15.116:0/107092627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23200703f0 0x7f23200c3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.532+0000 7f2327264640 1 --2- 172.21.15.116:0/107092627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f232006c250 0x7f23200c91c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.532+0000 7f2327264640 1 -- 172.21.15.116:0/107092627 >> 172.21.15.116:0/107092627 conn(0x7f232006a4c0 msgr2=0x7f2320143210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.532+0000 7f2327264640 1 -- 172.21.15.116:0/107092627 shutdown_connections 2023-11-28T17:10:33.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.532+0000 7f2327264640 1 -- 172.21.15.116:0/107092627 wait complete. 2023-11-28T17:10:33.553 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:10:17.338444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:10:18.766338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:10:33.554 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:10:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.766+0000 7feb41a3d640 1 Processor -- start 2023-11-28T17:10:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.766+0000 7feb41a3d640 1 -- start start 2023-11-28T17:10:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.768+0000 7feb41a3d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb3c072250 0x7feb3c15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.768+0000 7feb41a3d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3c0cd320 con 0x7feb3c15b0c0 2023-11-28T17:10:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.768+0000 7feb41a3d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3c0cc710 con 0x7feb3c0ce940 2023-11-28T17:10:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.768+0000 7feb41a3d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c0cedf0 con 0x7feb3c072250 2023-11-28T17:10:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.768+0000 7feb3affd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7feb3c15b0c0 0x7feb3c071b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57242/0 (socket says 172.21.15.116:57242) 2023-11-28T17:10:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.768+0000 7feb3affd640 1 -- 172.21.15.116:0/1077753440 learned_addr learned my addr 172.21.15.116:0/1077753440 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb3a7fc640 1 --2- 172.21.15.116:0/1077753440 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb3c072250 0x7feb3c15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 -- 172.21.15.116:0/1077753440 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618860955 0 0) 0x7feb3c0cd320 con 0x7feb3c15b0c0 2023-11-28T17:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 -- 172.21.15.116:0/1077753440 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb24003520 con 0x7feb3c15b0c0 2023-11-28T17:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 -- 172.21.15.116:0/1077753440 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 485165464 0 0) 0x7feb24003520 con 0x7feb3c15b0c0 2023-11-28T17:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 -- 172.21.15.116:0/1077753440 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb3c072250 msgr2=0x7feb3c15ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 --2- 172.21.15.116:0/1077753440 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb3c072250 0x7feb3c15ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 -- 172.21.15.116:0/1077753440 >> v1:172.21.15.156:6789/0 conn(0x7feb3c0ce940 legacy=0x7feb3c15b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 -- 172.21.15.116:0/1077753440 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3c0cc9f0 con 0x7feb3c15b0c0 2023-11-28T17:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 -- 172.21.15.116:0/1077753440 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7feb2c001670 con 0x7feb3c15b0c0 2023-11-28T17:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 -- 172.21.15.116:0/1077753440 >> v1:172.21.15.116:6789/0 conn(0x7feb3c15b0c0 legacy=0x7feb3c071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 --2- 172.21.15.116:0/1077753440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb240039a0 0x7feb24003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 --2- 172.21.15.116:0/1077753440 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb3c072250 0x7feb240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 --2- 172.21.15.116:0/1077753440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb24004d00 0x7feb24005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 -- 172.21.15.116:0/1077753440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2400ba80 con 0x7feb240039a0 2023-11-28T17:10:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 -- 172.21.15.116:0/1077753440 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2400bbf0 con 0x7feb24004d00 2023-11-28T17:10:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.769+0000 7feb39ffb640 1 -- 172.21.15.116:0/1077753440 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2400bdb0 con 0x7feb3c072250 2023-11-28T17:10:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.770+0000 7feb3affd640 1 --2- 172.21.15.116:0/1077753440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb240039a0 0x7feb24003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.770+0000 7feb3a7fc640 1 --2- 172.21.15.116:0/1077753440 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb3c072250 0x7feb240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.770+0000 7feb3b7fe640 1 --2- 172.21.15.116:0/1077753440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb24004d00 0x7feb24005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.770+0000 7feb3affd640 1 -- 172.21.15.116:0/1077753440 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb3c072250 msgr2=0x7feb240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.770+0000 7feb3affd640 1 --2- 172.21.15.116:0/1077753440 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb3c072250 0x7feb240047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.770+0000 7feb3affd640 1 -- 172.21.15.116:0/1077753440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb24004d00 msgr2=0x7feb24005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.770+0000 7feb3affd640 1 --2- 172.21.15.116:0/1077753440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb24004d00 0x7feb24005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.770+0000 7feb3affd640 1 -- 172.21.15.116:0/1077753440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb24003520 con 0x7feb240039a0 2023-11-28T17:10:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.770+0000 7feb3b7fe640 1 --2- 172.21.15.116:0/1077753440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb24004d00 0x7feb24005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.771+0000 7feb3affd640 1 --2- 172.21.15.116:0/1077753440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb240039a0 0x7feb24003dc0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00c240 tx=0x7feb240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37efa6a7cdc3b4e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.771+0000 7feb39ffb640 1 -- 172.21.15.116:0/1077753440 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c005ad0 con 0x7feb240039a0 2023-11-28T17:10:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.772+0000 7feb39ffb640 1 -- 172.21.15.116:0/1077753440 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c005c30 con 0x7feb240039a0 2023-11-28T17:10:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.772+0000 7feb39ffb640 1 -- 172.21.15.116:0/1077753440 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c02c810 con 0x7feb240039a0 2023-11-28T17:10:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.772+0000 7feb41a3d640 1 -- 172.21.15.116:0/1077753440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb240039a0 msgr2=0x7feb24003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.772+0000 7feb41a3d640 1 --2- 172.21.15.116:0/1077753440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb240039a0 0x7feb24003dc0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00c240 tx=0x7feb240057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.772+0000 7feb3b7fe640 1 -- 172.21.15.116:0/1077753440 reap_dead start 2023-11-28T17:10:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.772+0000 7feb41a3d640 1 -- 172.21.15.116:0/1077753440 shutdown_connections 2023-11-28T17:10:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.772+0000 7feb41a3d640 1 -- 172.21.15.116:0/1077753440 >> 172.21.15.116:0/1077753440 conn(0x7feb3c0b0cf0 msgr2=0x7feb3c0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.772+0000 7feb41a3d640 1 -- 172.21.15.116:0/1077753440 shutdown_connections 2023-11-28T17:10:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.772+0000 7feb41a3d640 1 -- 172.21.15.116:0/1077753440 wait complete. 2023-11-28T17:10:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.774+0000 7feb41a3d640 1 Processor -- start 2023-11-28T17:10:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.774+0000 7feb41a3d640 1 -- start start 2023-11-28T17:10:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.775+0000 7feb41a3d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb240039a0 0x7feb3c15a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.775+0000 7feb41a3d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb3c1596c0 0x7feb3c157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.775+0000 7feb41a3d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb3c1468f0 0x7feb3c15caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.775+0000 7feb41a3d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c1592a0 con 0x7feb240039a0 2023-11-28T17:10:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.775+0000 7feb41a3d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c15a5c0 con 0x7feb3c1596c0 2023-11-28T17:10:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.775+0000 7feb41a3d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c158410 con 0x7feb3c1468f0 2023-11-28T17:10:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.775+0000 7feb3affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb240039a0 0x7feb3c15a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.775+0000 7feb3affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb240039a0 0x7feb3c15a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56976/0 (socket says 172.21.15.116:56976) 2023-11-28T17:10:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.775+0000 7feb3affd640 1 -- 172.21.15.116:0/2667089572 learned_addr learned my addr 172.21.15.116:0/2667089572 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.776+0000 7feb3b7fe640 1 --2- 172.21.15.116:0/2667089572 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb3c1468f0 0x7feb3c15caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.776+0000 7feb3affd640 1 -- 172.21.15.116:0/2667089572 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb3c1468f0 msgr2=0x7feb3c15caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.776+0000 7feb3affd640 1 --2- 172.21.15.116:0/2667089572 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb3c1468f0 0x7feb3c15caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.776+0000 7feb3affd640 1 -- 172.21.15.116:0/2667089572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb3c1596c0 msgr2=0x7feb3c157ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.776+0000 7feb3affd640 1 --2- 172.21.15.116:0/2667089572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb3c1596c0 0x7feb3c157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.776+0000 7feb3affd640 1 -- 172.21.15.116:0/2667089572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3c15b620 con 0x7feb240039a0 2023-11-28T17:10:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.776+0000 7feb3affd640 1 --2- 172.21.15.116:0/2667089572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb240039a0 0x7feb3c15a080 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7feb2c001930 tx=0x7feb2c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.776+0000 7feb23fff640 1 -- 172.21.15.116:0/2667089572 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c03c040 con 0x7feb240039a0 2023-11-28T17:10:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.777+0000 7feb23fff640 1 -- 172.21.15.116:0/2667089572 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c02cc20 con 0x7feb240039a0 2023-11-28T17:10:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.777+0000 7feb23fff640 1 -- 172.21.15.116:0/2667089572 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c007070 con 0x7feb240039a0 2023-11-28T17:10:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.777+0000 7feb41a3d640 1 -- 172.21.15.116:0/2667089572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb3c146da0 con 0x7feb240039a0 2023-11-28T17:10:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.777+0000 7feb41a3d640 1 -- 172.21.15.116:0/2667089572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3c158900 con 0x7feb240039a0 2023-11-28T17:10:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.778+0000 7feb41a3d640 1 -- 172.21.15.116:0/2667089572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb3c069360 con 0x7feb240039a0 2023-11-28T17:10:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.779+0000 7feb23fff640 1 -- 172.21.15.116:0/2667089572 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7feb2c032070 con 0x7feb240039a0 2023-11-28T17:10:33.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.780+0000 7feb23fff640 1 --2- 172.21.15.116:0/2667089572 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feb2806b2a0 0x7feb2806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:33.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.780+0000 7feb23fff640 1 -- 172.21.15.116:0/2667089572 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb2c0aa870 con 0x7feb240039a0 2023-11-28T17:10:33.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.780+0000 7feb3a7fc640 1 --2- 172.21.15.116:0/2667089572 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feb2806b2a0 0x7feb2806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:33.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.781+0000 7feb3a7fc640 1 --2- 172.21.15.116:0/2667089572 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feb2806b2a0 0x7feb2806d760 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feb3c159d20 tx=0x7feb30008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:33.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.784+0000 7feb23fff640 1 -- 172.21.15.116:0/2667089572 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb2c0756f0 con 0x7feb240039a0 2023-11-28T17:10:33.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.974+0000 7feb41a3d640 1 -- 172.21.15.116:0/2667089572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feb3c0b1990 con 0x7feb240039a0 2023-11-28T17:10:33.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.975+0000 7feb23fff640 1 -- 172.21.15.116:0/2667089572 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7feb2c078d10 con 0x7feb240039a0 2023-11-28T17:10:33.977 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:10:33.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.978+0000 7feb41a3d640 1 -- 172.21.15.116:0/2667089572 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feb2806b2a0 msgr2=0x7feb2806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.978+0000 7feb41a3d640 1 --2- 172.21.15.116:0/2667089572 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feb2806b2a0 0x7feb2806d760 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feb3c159d20 tx=0x7feb30008040 comp rx=0 tx=0).stop 2023-11-28T17:10:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.978+0000 7feb41a3d640 1 -- 172.21.15.116:0/2667089572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb240039a0 msgr2=0x7feb3c15a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.978+0000 7feb41a3d640 1 --2- 172.21.15.116:0/2667089572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb240039a0 0x7feb3c15a080 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7feb2c001930 tx=0x7feb2c001a00 comp rx=0 tx=0).stop 2023-11-28T17:10:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.978+0000 7feb41a3d640 1 -- 172.21.15.116:0/2667089572 shutdown_connections 2023-11-28T17:10:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.978+0000 7feb41a3d640 1 --2- 172.21.15.116:0/2667089572 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7feb2806b2a0 0x7feb2806d760 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.978+0000 7feb41a3d640 1 --2- 172.21.15.116:0/2667089572 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb3c1468f0 0x7feb3c15caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.978+0000 7feb41a3d640 1 --2- 172.21.15.116:0/2667089572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb3c1596c0 0x7feb3c157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.978+0000 7feb41a3d640 1 --2- 172.21.15.116:0/2667089572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb240039a0 0x7feb3c15a080 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.978+0000 7feb41a3d640 1 -- 172.21.15.116:0/2667089572 >> 172.21.15.116:0/2667089572 conn(0x7feb3c0b0cf0 msgr2=0x7feb3c071860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.978+0000 7feb41a3d640 1 -- 172.21.15.116:0/2667089572 shutdown_connections 2023-11-28T17:10:33.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:33.978+0000 7feb41a3d640 1 -- 172.21.15.116:0/2667089572 wait complete. 2023-11-28T17:10:33.998 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:10:34.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.205+0000 7f8c133ef640 1 Processor -- start 2023-11-28T17:10:34.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.205+0000 7f8c133ef640 1 -- start start 2023-11-28T17:10:34.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.207+0000 7f8c133ef640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8c0c15e2d0 0x7f8c0c06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:34.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.207+0000 7f8c133ef640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8c0c0b2360 0x7f8c0c15d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:34.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.207+0000 7f8c133ef640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c0c15e6b0 con 0x7f8c0c06bf70 2023-11-28T17:10:34.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.207+0000 7f8c133ef640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c0c06c530 con 0x7f8c0c15e2d0 2023-11-28T17:10:34.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.207+0000 7f8c133ef640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c0c06c810 con 0x7f8c0c0b2360 2023-11-28T17:10:34.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.207+0000 7f8c123ed640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8c0c15e2d0 0x7f8c0c06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:34.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.207+0000 7f8c11bec640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8c0c06bf70 0x7f8c0c0b1c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57250/0 (socket says 172.21.15.116:57250) 2023-11-28T17:10:34.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.207+0000 7f8c11bec640 1 -- 172.21.15.116:0/1024921548 learned_addr learned my addr 172.21.15.116:0/1024921548 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:34.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.208+0000 7f8c12bee640 1 --2- 172.21.15.116:0/1024921548 >> v2:172.21.15.156:3300/0 conn(0x7f8c0c0b2360 0x7f8c0c15d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:34.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.208+0000 7f8c113eb640 1 -- 172.21.15.116:0/1024921548 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183892120 0 0) 0x7f8c0c15e6b0 con 0x7f8c0c06bf70 2023-11-28T17:10:34.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.208+0000 7f8c113eb640 1 -- 172.21.15.116:0/1024921548 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bf8003520 con 0x7f8c0c06bf70 2023-11-28T17:10:34.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.208+0000 7f8c123ed640 1 -- 172.21.15.116:0/1024921548 >> v2:172.21.15.156:3300/0 conn(0x7f8c0c0b2360 msgr2=0x7f8c0c15d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:34.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.208+0000 7f8c123ed640 1 --2- 172.21.15.116:0/1024921548 >> v2:172.21.15.156:3300/0 conn(0x7f8c0c0b2360 0x7f8c0c15d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.208+0000 7f8c123ed640 1 -- 172.21.15.116:0/1024921548 >> v1:172.21.15.116:6789/0 conn(0x7f8c0c06bf70 legacy=0x7f8c0c0b1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:34.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.208+0000 7f8c123ed640 1 -- 172.21.15.116:0/1024921548 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c0c06caf0 con 0x7f8c0c15e2d0 2023-11-28T17:10:34.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.208+0000 7f8c12bee640 1 --2- 172.21.15.116:0/1024921548 >> v2:172.21.15.156:3300/0 conn(0x7f8c0c0b2360 0x7f8c0c15d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:10:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.209+0000 7f8c123ed640 1 --2- 172.21.15.116:0/1024921548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0c15e2d0 0x7f8c0c06d5d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0b08e0 tx=0x7f8c0802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f971e3e1655cae server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.210+0000 7f8c113eb640 1 -- 172.21.15.116:0/1024921548 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c08031830 con 0x7f8c0c15e2d0 2023-11-28T17:10:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.210+0000 7f8c113eb640 1 -- 172.21.15.116:0/1024921548 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c08031990 con 0x7f8c0c15e2d0 2023-11-28T17:10:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.210+0000 7f8c113eb640 1 -- 172.21.15.116:0/1024921548 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c08031d20 con 0x7f8c0c15e2d0 2023-11-28T17:10:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.210+0000 7f8c133ef640 1 -- 172.21.15.116:0/1024921548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0c15e2d0 msgr2=0x7f8c0c06d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.210+0000 7f8c133ef640 1 --2- 172.21.15.116:0/1024921548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0c15e2d0 0x7f8c0c06d5d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0b08e0 tx=0x7f8c0802b8d0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.210+0000 7f8c133ef640 1 -- 172.21.15.116:0/1024921548 shutdown_connections 2023-11-28T17:10:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.210+0000 7f8c133ef640 1 --2- 172.21.15.116:0/1024921548 >> v2:172.21.15.156:3300/0 conn(0x7f8c0c0b2360 0x7f8c0c15d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.210+0000 7f8c133ef640 1 --2- 172.21.15.116:0/1024921548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0c15e2d0 0x7f8c0c06d5d0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.210+0000 7f8c133ef640 1 -- 172.21.15.116:0/1024921548 >> 172.21.15.116:0/1024921548 conn(0x7f8c0c0b0950 msgr2=0x7f8c0c0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:34.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.211+0000 7f8c133ef640 1 -- 172.21.15.116:0/1024921548 shutdown_connections 2023-11-28T17:10:34.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.211+0000 7f8c133ef640 1 -- 172.21.15.116:0/1024921548 wait complete. 2023-11-28T17:10:34.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.212+0000 7f8c133ef640 1 Processor -- start 2023-11-28T17:10:34.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.212+0000 7f8c133ef640 1 -- start start 2023-11-28T17:10:34.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.213+0000 7f8c133ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0c06bf70 0x7f8c0c09c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:34.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.213+0000 7f8c133ef640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c0c0b2360 0x7f8c0c0714e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:34.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.213+0000 7f8c133ef640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8c0c15e2d0 0x7f8c0c09ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:34.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.213+0000 7f8c133ef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c09c2a0 con 0x7f8c0c06bf70 2023-11-28T17:10:34.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.213+0000 7f8c133ef640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c09b690 con 0x7f8c0c0b2360 2023-11-28T17:10:34.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.213+0000 7f8c133ef640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c09b970 con 0x7f8c0c15e2d0 2023-11-28T17:10:34.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.214+0000 7f8c123ed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0c06bf70 0x7f8c0c09c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:34.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.214+0000 7f8c123ed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0c06bf70 0x7f8c0c09c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56990/0 (socket says 172.21.15.116:56990) 2023-11-28T17:10:34.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.214+0000 7f8c123ed640 1 -- 172.21.15.116:0/2029627580 learned_addr learned my addr 172.21.15.116:0/2029627580 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:34.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.214+0000 7f8c11bec640 1 --2- 172.21.15.116:0/2029627580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c0c0b2360 0x7f8c0c0714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:34.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.214+0000 7f8c123ed640 1 -- 172.21.15.116:0/2029627580 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8c0c15e2d0 msgr2=0x7f8c0c09ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:34.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.214+0000 7f8c123ed640 1 --2- 172.21.15.116:0/2029627580 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8c0c15e2d0 0x7f8c0c09ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.214+0000 7f8c123ed640 1 -- 172.21.15.116:0/2029627580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c0c0b2360 msgr2=0x7f8c0c0714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:34.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.214+0000 7f8c123ed640 1 --2- 172.21.15.116:0/2029627580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c0c0b2360 0x7f8c0c0714e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.214+0000 7f8c123ed640 1 -- 172.21.15.116:0/2029627580 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c08009400 con 0x7f8c0c06bf70 2023-11-28T17:10:34.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.214+0000 7f8c11bec640 1 --2- 172.21.15.116:0/2029627580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c0c0b2360 0x7f8c0c0714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:34.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.214+0000 7f8c123ed640 1 --2- 172.21.15.116:0/2029627580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0c06bf70 0x7f8c0c09c730 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8c08000c00 tx=0x7f8c080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.215+0000 7f8c02ffd640 1 -- 172.21.15.116:0/2029627580 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0803a8f0 con 0x7f8c0c06bf70 2023-11-28T17:10:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.215+0000 7f8c02ffd640 1 -- 172.21.15.116:0/2029627580 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0803aa70 con 0x7f8c0c06bf70 2023-11-28T17:10:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.215+0000 7f8c02ffd640 1 -- 172.21.15.116:0/2029627580 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0803ae20 con 0x7f8c0c06bf70 2023-11-28T17:10:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.215+0000 7f8c133ef640 1 -- 172.21.15.116:0/2029627580 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0c09bc50 con 0x7f8c0c06bf70 2023-11-28T17:10:34.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.215+0000 7f8c133ef640 1 -- 172.21.15.116:0/2029627580 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0c09cc70 con 0x7f8c0c06bf70 2023-11-28T17:10:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.216+0000 7f8c133ef640 1 -- 172.21.15.116:0/2029627580 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0c09d0b0 con 0x7f8c0c06bf70 2023-11-28T17:10:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.217+0000 7f8c02ffd640 1 -- 172.21.15.116:0/2029627580 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f8c0804d450 con 0x7f8c0c06bf70 2023-11-28T17:10:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.218+0000 7f8c02ffd640 1 --2- 172.21.15.116:0/2029627580 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8bf006b2a0 0x7f8bf006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.218+0000 7f8c02ffd640 1 -- 172.21.15.116:0/2029627580 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c080abc40 con 0x7f8c0c06bf70 2023-11-28T17:10:34.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.218+0000 7f8c11bec640 1 --2- 172.21.15.116:0/2029627580 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8bf006b2a0 0x7f8bf006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:34.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.219+0000 7f8c11bec640 1 --2- 172.21.15.116:0/2029627580 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8bf006b2a0 0x7f8bf006d760 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c071eb0 tx=0x7f8bfc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:34.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.222+0000 7f8c02ffd640 1 -- 172.21.15.116:0/2029627580 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8c08076ab0 con 0x7f8c0c06bf70 2023-11-28T17:10:34.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.411+0000 7f8c133ef640 1 -- 172.21.15.116:0/2029627580 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8c0c04e960 con 0x7f8c0c06bf70 2023-11-28T17:10:34.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.411+0000 7f8c02ffd640 1 -- 172.21.15.116:0/2029627580 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8c08035020 con 0x7f8c0c06bf70 2023-11-28T17:10:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:10:34.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.414+0000 7f8c133ef640 1 -- 172.21.15.116:0/2029627580 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8bf006b2a0 msgr2=0x7f8bf006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:34.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.414+0000 7f8c133ef640 1 --2- 172.21.15.116:0/2029627580 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8bf006b2a0 0x7f8bf006d760 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c071eb0 tx=0x7f8bfc009040 comp rx=0 tx=0).stop 2023-11-28T17:10:34.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.414+0000 7f8c133ef640 1 -- 172.21.15.116:0/2029627580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0c06bf70 msgr2=0x7f8c0c09c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:34.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.414+0000 7f8c133ef640 1 --2- 172.21.15.116:0/2029627580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0c06bf70 0x7f8c0c09c730 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8c08000c00 tx=0x7f8c080046c0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.414+0000 7f8c133ef640 1 -- 172.21.15.116:0/2029627580 shutdown_connections 2023-11-28T17:10:34.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.414+0000 7f8c133ef640 1 --2- 172.21.15.116:0/2029627580 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8bf006b2a0 0x7f8bf006d760 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.414+0000 7f8c133ef640 1 --2- 172.21.15.116:0/2029627580 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8c0c15e2d0 0x7f8c0c09ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.414+0000 7f8c133ef640 1 --2- 172.21.15.116:0/2029627580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c0c0b2360 0x7f8c0c0714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.414+0000 7f8c133ef640 1 --2- 172.21.15.116:0/2029627580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c0c06bf70 0x7f8c0c09c730 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.414+0000 7f8c133ef640 1 -- 172.21.15.116:0/2029627580 >> 172.21.15.116:0/2029627580 conn(0x7f8c0c0b0950 msgr2=0x7f8c0c15d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:34.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.415+0000 7f8c133ef640 1 -- 172.21.15.116:0/2029627580 shutdown_connections 2023-11-28T17:10:34.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.415+0000 7f8c133ef640 1 -- 172.21.15.116:0/2029627580 wait complete. 2023-11-28T17:10:34.434 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:10:34.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.653+0000 7fb362213640 1 Processor -- start 2023-11-28T17:10:34.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.654+0000 7fb362213640 1 -- start start 2023-11-28T17:10:34.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.655+0000 7fb362213640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb35c152050 0x7fb35c15bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:34.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.655+0000 7fb362213640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb35c0abd30 0x7fb35c0cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:34.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.655+0000 7fb362213640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35c0b2b50 con 0x7fb35c0b2fe0 2023-11-28T17:10:34.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.655+0000 7fb362213640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35c15c520 con 0x7fb35c0abd30 2023-11-28T17:10:34.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.655+0000 7fb362213640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c0ac2d0 con 0x7fb35c152050 2023-11-28T17:10:34.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.656+0000 7fb361a12640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb35c0abd30 0x7fb35c0cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:34.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.656+0000 7fb361a12640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb35c0abd30 0x7fb35c0cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57002/0 (socket says 172.21.15.116:57002) 2023-11-28T17:10:34.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.656+0000 7fb361a12640 1 -- 172.21.15.116:0/1386752532 learned_addr learned my addr 172.21.15.116:0/1386752532 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:34.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.656+0000 7fb361211640 1 --2- 172.21.15.116:0/1386752532 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb35c152050 0x7fb35c15bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:34.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.656+0000 7fb361a12640 1 -- 172.21.15.116:0/1386752532 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb35c152050 msgr2=0x7fb35c15bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:34.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.656+0000 7fb361a12640 1 --2- 172.21.15.116:0/1386752532 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb35c152050 0x7fb35c15bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.656+0000 7fb361a12640 1 -- 172.21.15.116:0/1386752532 >> v1:172.21.15.156:6789/0 conn(0x7fb35c0b2fe0 legacy=0x7fb35c0ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:34.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.656+0000 7fb361a12640 1 -- 172.21.15.116:0/1386752532 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c0b2220 con 0x7fb35c0abd30 2023-11-28T17:10:34.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.657+0000 7fb361a12640 1 --2- 172.21.15.116:0/1386752532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c0abd30 0x7fb35c0cdc60 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb354009e50 tx=0x7fb35402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=148193cfbc1880d0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:34.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.658+0000 7fb353fff640 1 -- 172.21.15.116:0/1386752532 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35402caf0 con 0x7fb35c0abd30 2023-11-28T17:10:34.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.658+0000 7fb353fff640 1 -- 172.21.15.116:0/1386752532 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35402cc50 con 0x7fb35c0abd30 2023-11-28T17:10:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.658+0000 7fb353fff640 1 -- 172.21.15.116:0/1386752532 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3540343f0 con 0x7fb35c0abd30 2023-11-28T17:10:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.658+0000 7fb362213640 1 -- 172.21.15.116:0/1386752532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c0abd30 msgr2=0x7fb35c0cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.658+0000 7fb362213640 1 --2- 172.21.15.116:0/1386752532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c0abd30 0x7fb35c0cdc60 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb354009e50 tx=0x7fb35402bf50 comp rx=0 tx=0).stop 2023-11-28T17:10:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.658+0000 7fb362213640 1 -- 172.21.15.116:0/1386752532 shutdown_connections 2023-11-28T17:10:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.658+0000 7fb362213640 1 --2- 172.21.15.116:0/1386752532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c0abd30 0x7fb35c0cdc60 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.658+0000 7fb362213640 1 --2- 172.21.15.116:0/1386752532 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb35c152050 0x7fb35c15bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.658+0000 7fb362213640 1 -- 172.21.15.116:0/1386752532 >> 172.21.15.116:0/1386752532 conn(0x7fb35c0b0950 msgr2=0x7fb35c0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.659+0000 7fb362213640 1 -- 172.21.15.116:0/1386752532 shutdown_connections 2023-11-28T17:10:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.659+0000 7fb362213640 1 -- 172.21.15.116:0/1386752532 wait complete. 2023-11-28T17:10:34.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.660+0000 7fb362213640 1 Processor -- start 2023-11-28T17:10:34.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.660+0000 7fb362213640 1 -- start start 2023-11-28T17:10:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.661+0000 7fb362213640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb35c0abd30 0x7fb35c146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.661+0000 7fb362213640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c0b2fe0 0x7fb35c099a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.661+0000 7fb362213640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb35c152050 0x7fb35c099f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.661+0000 7fb362213640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c146f30 con 0x7fb35c0b2fe0 2023-11-28T17:10:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.661+0000 7fb362213640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c146d50 con 0x7fb35c152050 2023-11-28T17:10:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.661+0000 7fb362213640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c147180 con 0x7fb35c0abd30 2023-11-28T17:10:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.662+0000 7fb360a10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c0b2fe0 0x7fb35c099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.662+0000 7fb361211640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb35c0abd30 0x7fb35c146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:34.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.662+0000 7fb361211640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb35c0abd30 0x7fb35c146810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:56150/0 (socket says 172.21.15.116:56150) 2023-11-28T17:10:34.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.662+0000 7fb361211640 1 -- 172.21.15.116:0/3129744550 learned_addr learned my addr 172.21.15.116:0/3129744550 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:34.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.662+0000 7fb361a12640 1 --2- 172.21.15.116:0/3129744550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb35c152050 0x7fb35c099f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:34.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.662+0000 7fb361211640 1 -- 172.21.15.116:0/3129744550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb35c152050 msgr2=0x7fb35c099f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:34.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.662+0000 7fb361211640 1 --2- 172.21.15.116:0/3129744550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb35c152050 0x7fb35c099f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.662+0000 7fb361211640 1 -- 172.21.15.116:0/3129744550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c0b2fe0 msgr2=0x7fb35c099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:34.665 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.662+0000 7fb361211640 1 --2- 172.21.15.116:0/3129744550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c0b2fe0 0x7fb35c099a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.662+0000 7fb361211640 1 -- 172.21.15.116:0/3129744550 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb354009b30 con 0x7fb35c0abd30 2023-11-28T17:10:34.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.662+0000 7fb360a10640 1 --2- 172.21.15.116:0/3129744550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c0b2fe0 0x7fb35c099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:10:34.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.663+0000 7fb361211640 1 --2- 172.21.15.116:0/3129744550 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb35c0abd30 0x7fb35c146810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb35800e870 tx=0x7fb35800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:34.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.663+0000 7fb351ffb640 1 -- 172.21.15.116:0/3129744550 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3580094f0 con 0x7fb35c0abd30 2023-11-28T17:10:34.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.663+0000 7fb362213640 1 -- 172.21.15.116:0/3129744550 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35c147460 con 0x7fb35c0abd30 2023-11-28T17:10:34.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.663+0000 7fb351ffb640 1 -- 172.21.15.116:0/3129744550 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35800c940 con 0x7fb35c0abd30 2023-11-28T17:10:34.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.663+0000 7fb351ffb640 1 -- 172.21.15.116:0/3129744550 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35800ccf0 con 0x7fb35c0abd30 2023-11-28T17:10:34.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.663+0000 7fb362213640 1 -- 172.21.15.116:0/3129744550 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35c09a5d0 con 0x7fb35c0abd30 2023-11-28T17:10:34.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.664+0000 7fb362213640 1 -- 172.21.15.116:0/3129744550 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35c09d0b0 con 0x7fb35c0abd30 2023-11-28T17:10:34.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.666+0000 7fb351ffb640 1 -- 172.21.15.116:0/3129744550 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb358005990 con 0x7fb35c0abd30 2023-11-28T17:10:34.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.667+0000 7fb351ffb640 1 --2- 172.21.15.116:0/3129744550 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb34006b2a0 0x7fb34006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:34.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.667+0000 7fb351ffb640 1 -- 172.21.15.116:0/3129744550 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb358089ff0 con 0x7fb35c0abd30 2023-11-28T17:10:34.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.667+0000 7fb360a10640 1 --2- 172.21.15.116:0/3129744550 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb34006b2a0 0x7fb34006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:34.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.667+0000 7fb360a10640 1 --2- 172.21.15.116:0/3129744550 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb34006b2a0 0x7fb34006d760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb35c1f6790 tx=0x7fb34c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:34.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.678+0000 7fb351ffb640 1 -- 172.21.15.116:0/3129744550 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb358054f10 con 0x7fb35c0abd30 2023-11-28T17:10:34.870 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.868+0000 7fb362213640 1 -- 172.21.15.116:0/3129744550 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb35c04e960 con 0x7fb35c0abd30 2023-11-28T17:10:34.871 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.870+0000 7fb351ffb640 1 -- 172.21.15.116:0/3129744550 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb3580170a0 con 0x7fb35c0abd30 2023-11-28T17:10:34.871 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:10:34.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.872+0000 7fb362213640 1 -- 172.21.15.116:0/3129744550 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb34006b2a0 msgr2=0x7fb34006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:34.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.872+0000 7fb362213640 1 --2- 172.21.15.116:0/3129744550 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb34006b2a0 0x7fb34006d760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb35c1f6790 tx=0x7fb34c009040 comp rx=0 tx=0).stop 2023-11-28T17:10:34.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.872+0000 7fb362213640 1 -- 172.21.15.116:0/3129744550 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb35c0abd30 msgr2=0x7fb35c146810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:34.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.872+0000 7fb362213640 1 --2- 172.21.15.116:0/3129744550 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb35c0abd30 0x7fb35c146810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb35800e870 tx=0x7fb35800ed30 comp rx=0 tx=0).stop 2023-11-28T17:10:34.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.873+0000 7fb362213640 1 -- 172.21.15.116:0/3129744550 shutdown_connections 2023-11-28T17:10:34.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.873+0000 7fb362213640 1 --2- 172.21.15.116:0/3129744550 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb34006b2a0 0x7fb34006d760 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.873+0000 7fb362213640 1 --2- 172.21.15.116:0/3129744550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb35c152050 0x7fb35c099f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.873+0000 7fb362213640 1 --2- 172.21.15.116:0/3129744550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb35c0b2fe0 0x7fb35c099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.873+0000 7fb362213640 1 --2- 172.21.15.116:0/3129744550 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb35c0abd30 0x7fb35c146810 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:34.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.873+0000 7fb362213640 1 -- 172.21.15.116:0/3129744550 >> 172.21.15.116:0/3129744550 conn(0x7fb35c0b0950 msgr2=0x7fb35c06cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:34.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.873+0000 7fb362213640 1 -- 172.21.15.116:0/3129744550 shutdown_connections 2023-11-28T17:10:34.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:34.873+0000 7fb362213640 1 -- 172.21.15.116:0/3129744550 wait complete. 2023-11-28T17:10:34.892 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:10:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.103+0000 7f72871c9640 1 Processor -- start 2023-11-28T17:10:35.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.104+0000 7f72871c9640 1 -- start start 2023-11-28T17:10:35.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.105+0000 7f72871c9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f728013cee0 0x7f728006c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.105+0000 7f72871c9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7280140200 0x7f728006c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.105+0000 7f72871c9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f728015a490 0x7f728015a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.105+0000 7f72871c9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72800aea20 con 0x7f728015a490 2023-11-28T17:10:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.105+0000 7f72871c9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f728013d2c0 con 0x7f7280140200 2023-11-28T17:10:35.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.105+0000 7f72871c9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f728015b3a0 con 0x7f728013cee0 2023-11-28T17:10:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.106+0000 7f7285766640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f728015a490 0x7f728015a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:35.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.106+0000 7f7285766640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f728015a490 0x7f728015a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57028/0 (socket says 172.21.15.116:57028) 2023-11-28T17:10:35.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.106+0000 7f7285766640 1 -- 172.21.15.116:0/401169894 learned_addr learned my addr 172.21.15.116:0/401169894 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:35.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.106+0000 7f727ffff640 1 --2- 172.21.15.116:0/401169894 >> v2:172.21.15.156:3300/0 conn(0x7f7280140200 0x7f728006c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:35.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.106+0000 7f7284f65640 1 --2- 172.21.15.116:0/401169894 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f728013cee0 0x7f728006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:35.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.106+0000 7f7285766640 1 -- 172.21.15.116:0/401169894 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f728013cee0 msgr2=0x7f728006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:35.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.106+0000 7f7285766640 1 --2- 172.21.15.116:0/401169894 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f728013cee0 0x7f728006c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.106+0000 7f7285766640 1 -- 172.21.15.116:0/401169894 >> v2:172.21.15.156:3300/0 conn(0x7f7280140200 msgr2=0x7f728006c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:35.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.106+0000 7f7285766640 1 --2- 172.21.15.116:0/401169894 >> v2:172.21.15.156:3300/0 conn(0x7f7280140200 0x7f728006c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.106+0000 7f7285766640 1 -- 172.21.15.116:0/401169894 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728015ae60 con 0x7f728015a490 2023-11-28T17:10:35.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.106+0000 7f727ffff640 1 --2- 172.21.15.116:0/401169894 >> v2:172.21.15.156:3300/0 conn(0x7f7280140200 0x7f728006c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:10:35.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.107+0000 7f7285766640 1 --2- 172.21.15.116:0/401169894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f728015a490 0x7f728015a920 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f72800728a0 tx=0x7f727802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ceebc32c751418b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.108+0000 7f727f7fe640 1 -- 172.21.15.116:0/401169894 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727802c9f0 con 0x7f728015a490 2023-11-28T17:10:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.108+0000 7f727f7fe640 1 -- 172.21.15.116:0/401169894 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f727802cb50 con 0x7f728015a490 2023-11-28T17:10:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.108+0000 7f727f7fe640 1 -- 172.21.15.116:0/401169894 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72780343f0 con 0x7f728015a490 2023-11-28T17:10:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.108+0000 7f72871c9640 1 -- 172.21.15.116:0/401169894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f728015a490 msgr2=0x7f728015a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.108+0000 7f72871c9640 1 --2- 172.21.15.116:0/401169894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f728015a490 0x7f728015a920 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f72800728a0 tx=0x7f727802be50 comp rx=0 tx=0).stop 2023-11-28T17:10:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.109+0000 7f72871c9640 1 -- 172.21.15.116:0/401169894 shutdown_connections 2023-11-28T17:10:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.109+0000 7f72871c9640 1 --2- 172.21.15.116:0/401169894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f728015a490 0x7f728015a920 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.109+0000 7f72871c9640 1 --2- 172.21.15.116:0/401169894 >> v2:172.21.15.156:3300/0 conn(0x7f7280140200 0x7f728006c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.109+0000 7f72871c9640 1 --2- 172.21.15.116:0/401169894 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f728013cee0 0x7f728006c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.109+0000 7f72871c9640 1 -- 172.21.15.116:0/401169894 >> 172.21.15.116:0/401169894 conn(0x7f728006a4c0 msgr2=0x7f728006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.109+0000 7f72871c9640 1 -- 172.21.15.116:0/401169894 shutdown_connections 2023-11-28T17:10:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.109+0000 7f72871c9640 1 -- 172.21.15.116:0/401169894 wait complete. 2023-11-28T17:10:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.111+0000 7f72871c9640 1 Processor -- start 2023-11-28T17:10:35.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.111+0000 7f72871c9640 1 -- start start 2023-11-28T17:10:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.112+0000 7f72871c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f728013cee0 0x7f7280138950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.112+0000 7f72871c9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7280140200 0x7f728013ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.112+0000 7f72871c9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f728015a490 0x7f728013dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.112+0000 7f72871c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72800b3c10 con 0x7f728013cee0 2023-11-28T17:10:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.112+0000 7f72871c9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72800b43e0 con 0x7f728015a490 2023-11-28T17:10:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.112+0000 7f72871c9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f728013e2e0 con 0x7f7280140200 2023-11-28T17:10:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.112+0000 7f7284f65640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f728013cee0 0x7f7280138950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.112+0000 7f7284f65640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f728013cee0 0x7f7280138950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57040/0 (socket says 172.21.15.116:57040) 2023-11-28T17:10:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.112+0000 7f7284f65640 1 -- 172.21.15.116:0/548063617 learned_addr learned my addr 172.21.15.116:0/548063617 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.112+0000 7f7284f65640 1 -- 172.21.15.116:0/548063617 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7280140200 msgr2=0x7f728013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.112+0000 7f7284f65640 1 --2- 172.21.15.116:0/548063617 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7280140200 0x7f728013ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.112+0000 7f7285766640 1 --2- 172.21.15.116:0/548063617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f728015a490 0x7f728013dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.112+0000 7f7284f65640 1 -- 172.21.15.116:0/548063617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f728015a490 msgr2=0x7f728013dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.113+0000 7f7284f65640 1 --2- 172.21.15.116:0/548063617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f728015a490 0x7f728013dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.113+0000 7f7284f65640 1 -- 172.21.15.116:0/548063617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7278009b30 con 0x7f728013cee0 2023-11-28T17:10:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.113+0000 7f7285766640 1 --2- 172.21.15.116:0/548063617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f728015a490 0x7f728013dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.113+0000 7f7284f65640 1 --2- 172.21.15.116:0/548063617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f728013cee0 0x7f7280138950 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f727400d5f0 tx=0x7f727400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.113+0000 7f727d7fa640 1 -- 172.21.15.116:0/548063617 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72740125f0 con 0x7f728013cee0 2023-11-28T17:10:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.113+0000 7f727d7fa640 1 -- 172.21.15.116:0/548063617 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7274012750 con 0x7f728013cee0 2023-11-28T17:10:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.113+0000 7f727d7fa640 1 -- 172.21.15.116:0/548063617 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7274012b00 con 0x7f728013cee0 2023-11-28T17:10:35.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.113+0000 7f72871c9640 1 -- 172.21.15.116:0/548063617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72800b3e20 con 0x7f728013cee0 2023-11-28T17:10:35.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.113+0000 7f72871c9640 1 -- 172.21.15.116:0/548063617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f728013d5a0 con 0x7f728013cee0 2023-11-28T17:10:35.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.114+0000 7f72871c9640 1 -- 172.21.15.116:0/548063617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72800b4580 con 0x7f728013cee0 2023-11-28T17:10:35.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.115+0000 7f727d7fa640 1 -- 172.21.15.116:0/548063617 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f7274012c60 con 0x7f728013cee0 2023-11-28T17:10:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.116+0000 7f727d7fa640 1 --2- 172.21.15.116:0/548063617 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f726406b2a0 0x7f726406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.116+0000 7f727d7fa640 1 -- 172.21.15.116:0/548063617 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f727408a160 con 0x7f728013cee0 2023-11-28T17:10:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.116+0000 7f727ffff640 1 --2- 172.21.15.116:0/548063617 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f726406b2a0 0x7f726406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.117+0000 7f727ffff640 1 --2- 172.21.15.116:0/548063617 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f726406b2a0 0x7f726406d760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7280072450 tx=0x7f726c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:35.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.120+0000 7f727d7fa640 1 -- 172.21.15.116:0/548063617 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7274054f30 con 0x7f728013cee0 2023-11-28T17:10:35.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.326+0000 7f72871c9640 1 -- 172.21.15.116:0/548063617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f728013d400 con 0x7f728013cee0 2023-11-28T17:10:35.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.327+0000 7f727d7fa640 1 -- 172.21.15.116:0/548063617 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f728013d400 con 0x7f728013cee0 2023-11-28T17:10:35.328 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-28T17:10:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.329+0000 7f72871c9640 1 -- 172.21.15.116:0/548063617 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f726406b2a0 msgr2=0x7f726406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.329+0000 7f72871c9640 1 --2- 172.21.15.116:0/548063617 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f726406b2a0 0x7f726406d760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7280072450 tx=0x7f726c008040 comp rx=0 tx=0).stop 2023-11-28T17:10:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.329+0000 7f72871c9640 1 -- 172.21.15.116:0/548063617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f728013cee0 msgr2=0x7f7280138950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.329+0000 7f72871c9640 1 --2- 172.21.15.116:0/548063617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f728013cee0 0x7f7280138950 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f727400d5f0 tx=0x7f727400dab0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.329+0000 7f72871c9640 1 -- 172.21.15.116:0/548063617 shutdown_connections 2023-11-28T17:10:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.329+0000 7f72871c9640 1 --2- 172.21.15.116:0/548063617 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f726406b2a0 0x7f726406d760 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.329+0000 7f72871c9640 1 --2- 172.21.15.116:0/548063617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f728015a490 0x7f728013dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.329+0000 7f72871c9640 1 --2- 172.21.15.116:0/548063617 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7280140200 0x7f728013ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.329+0000 7f72871c9640 1 --2- 172.21.15.116:0/548063617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f728013cee0 0x7f7280138950 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.329+0000 7f72871c9640 1 -- 172.21.15.116:0/548063617 >> 172.21.15.116:0/548063617 conn(0x7f728006a4c0 msgr2=0x7f72800ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:35.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.329+0000 7f72871c9640 1 -- 172.21.15.116:0/548063617 shutdown_connections 2023-11-28T17:10:35.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.329+0000 7f72871c9640 1 -- 172.21.15.116:0/548063617 wait complete. 2023-11-28T17:10:35.348 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:10:35.349 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:10:35.396 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:10:35.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.648+0000 7fcfb7778640 1 Processor -- start 2023-11-28T17:10:35.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.649+0000 7fcfb7778640 1 -- start start 2023-11-28T17:10:35.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.650+0000 7fcfb7778640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fcfb01401c0 0x7fcfb0071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:35.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.650+0000 7fcfb7778640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcfb0071710 0x7fcfb006c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:35.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.650+0000 7fcfb7778640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcfb006c790 0x7fcfb00703f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:35.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.650+0000 7fcfb7778640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb006cc00 con 0x7fcfb006c790 2023-11-28T17:10:35.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.650+0000 7fcfb7778640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb01405a0 con 0x7fcfb01401c0 2023-11-28T17:10:35.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.650+0000 7fcfb7778640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb0140880 con 0x7fcfb0071710 2023-11-28T17:10:35.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.651+0000 7fcfb5d15640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcfb006c790 0x7fcfb00703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:35.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.651+0000 7fcfb5d15640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcfb006c790 0x7fcfb00703f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57046/0 (socket says 172.21.15.116:57046) 2023-11-28T17:10:35.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.651+0000 7fcfb5d15640 1 -- 172.21.15.116:0/196382789 learned_addr learned my addr 172.21.15.116:0/196382789 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:35.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.651+0000 7fcfb5514640 1 --2- 172.21.15.116:0/196382789 >> v2:172.21.15.156:3300/0 conn(0x7fcfb01401c0 0x7fcfb0071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:35.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.651+0000 7fcfb5d15640 1 -- 172.21.15.116:0/196382789 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcfb0071710 msgr2=0x7fcfb006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:35.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.651+0000 7fcfb5d15640 1 --2- 172.21.15.116:0/196382789 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcfb0071710 0x7fcfb006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.651+0000 7fcfb5d15640 1 -- 172.21.15.116:0/196382789 >> v2:172.21.15.156:3300/0 conn(0x7fcfb01401c0 msgr2=0x7fcfb0071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.651+0000 7fcfb5d15640 1 --2- 172.21.15.116:0/196382789 >> v2:172.21.15.156:3300/0 conn(0x7fcfb01401c0 0x7fcfb0071eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.651+0000 7fcfb5d15640 1 -- 172.21.15.116:0/196382789 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb015db00 con 0x7fcfb006c790 2023-11-28T17:10:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.651+0000 7fcfb5514640 1 --2- 172.21.15.116:0/196382789 >> v2:172.21.15.156:3300/0 conn(0x7fcfb01401c0 0x7fcfb0071eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:35.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.652+0000 7fcfb5d15640 1 --2- 172.21.15.116:0/196382789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb006c790 0x7fcfb00703f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fcfb006bdd0 tx=0x7fcfa802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f833ea1bcfd1d0d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:35.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.652+0000 7fcfa7fff640 1 -- 172.21.15.116:0/196382789 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa802ca80 con 0x7fcfb006c790 2023-11-28T17:10:35.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.653+0000 7fcfa7fff640 1 -- 172.21.15.116:0/196382789 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa802cbe0 con 0x7fcfb006c790 2023-11-28T17:10:35.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.653+0000 7fcfa7fff640 1 -- 172.21.15.116:0/196382789 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa80343f0 con 0x7fcfb006c790 2023-11-28T17:10:35.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.653+0000 7fcfb7778640 1 -- 172.21.15.116:0/196382789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb006c790 msgr2=0x7fcfb00703f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:35.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.653+0000 7fcfb7778640 1 --2- 172.21.15.116:0/196382789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb006c790 0x7fcfb00703f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fcfb006bdd0 tx=0x7fcfa802bee0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.653+0000 7fcfb7778640 1 -- 172.21.15.116:0/196382789 shutdown_connections 2023-11-28T17:10:35.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.653+0000 7fcfb7778640 1 --2- 172.21.15.116:0/196382789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb006c790 0x7fcfb00703f0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.653+0000 7fcfb7778640 1 --2- 172.21.15.116:0/196382789 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcfb0071710 0x7fcfb006c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.653+0000 7fcfb7778640 1 --2- 172.21.15.116:0/196382789 >> v2:172.21.15.156:3300/0 conn(0x7fcfb01401c0 0x7fcfb0071eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.653+0000 7fcfb7778640 1 -- 172.21.15.116:0/196382789 >> 172.21.15.116:0/196382789 conn(0x7fcfb006a4c0 msgr2=0x7fcfb006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:35.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.653+0000 7fcfb7778640 1 -- 172.21.15.116:0/196382789 shutdown_connections 2023-11-28T17:10:35.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.653+0000 7fcfb7778640 1 -- 172.21.15.116:0/196382789 wait complete. 2023-11-28T17:10:35.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.655+0000 7fcfb7778640 1 Processor -- start 2023-11-28T17:10:35.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.655+0000 7fcfb7778640 1 -- start start 2023-11-28T17:10:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.656+0000 7fcfb7778640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb006c790 0x7fcfb015ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.656+0000 7fcfb7778640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfb0071710 0x7fcfb0159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.656+0000 7fcfb7778640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcfb01401c0 0x7fcfb013c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.656+0000 7fcfb7778640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb0159fd0 con 0x7fcfb006c790 2023-11-28T17:10:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.656+0000 7fcfb7778640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb0140a80 con 0x7fcfb0071710 2023-11-28T17:10:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.656+0000 7fcfb7778640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb013c8f0 con 0x7fcfb01401c0 2023-11-28T17:10:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.656+0000 7fcfb5514640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb006c790 0x7fcfb015ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.656+0000 7fcfb5514640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb006c790 0x7fcfb015ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57048/0 (socket says 172.21.15.116:57048) 2023-11-28T17:10:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.656+0000 7fcfb5514640 1 -- 172.21.15.116:0/3721664417 learned_addr learned my addr 172.21.15.116:0/3721664417 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.657+0000 7fcfb5514640 1 -- 172.21.15.116:0/3721664417 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcfb01401c0 msgr2=0x7fcfb013c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.657+0000 7fcfb5514640 1 --2- 172.21.15.116:0/3721664417 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcfb01401c0 0x7fcfb013c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.657+0000 7fcfb5514640 1 -- 172.21.15.116:0/3721664417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfb0071710 msgr2=0x7fcfb0159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.657+0000 7fcfb4d13640 1 --2- 172.21.15.116:0/3721664417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfb0071710 0x7fcfb0159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.657+0000 7fcfb5514640 1 --2- 172.21.15.116:0/3721664417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfb0071710 0x7fcfb0159a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.657+0000 7fcfb5514640 1 -- 172.21.15.116:0/3721664417 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa8009b30 con 0x7fcfb006c790 2023-11-28T17:10:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.657+0000 7fcfb4d13640 1 --2- 172.21.15.116:0/3721664417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfb0071710 0x7fcfb0159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:35.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.657+0000 7fcfb5514640 1 --2- 172.21.15.116:0/3721664417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb006c790 0x7fcfb015ac40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fcfac00c7f0 tx=0x7fcfac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:35.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.657+0000 7fcfa5ffb640 1 -- 172.21.15.116:0/3721664417 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfac007820 con 0x7fcfb006c790 2023-11-28T17:10:35.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.657+0000 7fcfa5ffb640 1 -- 172.21.15.116:0/3721664417 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfac007980 con 0x7fcfb006c790 2023-11-28T17:10:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.657+0000 7fcfa5ffb640 1 -- 172.21.15.116:0/3721664417 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfac007cf0 con 0x7fcfb006c790 2023-11-28T17:10:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.657+0000 7fcfb7778640 1 -- 172.21.15.116:0/3721664417 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb015a230 con 0x7fcfb006c790 2023-11-28T17:10:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.657+0000 7fcfb7778640 1 -- 172.21.15.116:0/3721664417 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb0141000 con 0x7fcfb006c790 2023-11-28T17:10:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.659+0000 7fcfb7778640 1 -- 172.21.15.116:0/3721664417 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb00693d0 con 0x7fcfb006c790 2023-11-28T17:10:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.659+0000 7fcfa5ffb640 1 -- 172.21.15.116:0/3721664417 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fcfac0113f0 con 0x7fcfb006c790 2023-11-28T17:10:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.660+0000 7fcfa5ffb640 1 --2- 172.21.15.116:0/3721664417 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcf9406b2a0 0x7fcf9406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:35.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.660+0000 7fcfa5ffb640 1 -- 172.21.15.116:0/3721664417 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcfac089ef0 con 0x7fcfb006c790 2023-11-28T17:10:35.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.661+0000 7fcfb4d13640 1 --2- 172.21.15.116:0/3721664417 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcf9406b2a0 0x7fcf9406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:35.662 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.661+0000 7fcfb4d13640 1 --2- 172.21.15.116:0/3721664417 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcf9406b2a0 0x7fcf9406d760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00ae970 tx=0x7fcfa0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:35.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.665+0000 7fcfa5ffb640 1 -- 172.21.15.116:0/3721664417 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcfac054d50 con 0x7fcfb006c790 2023-11-28T17:10:35.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.855+0000 7fcfb7778640 1 -- 172.21.15.116:0/3721664417 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfb006b4d0 con 0x7fcfb006c790 2023-11-28T17:10:35.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.858+0000 7fcfa5ffb640 1 -- 172.21.15.116:0/3721664417 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16521 (secure 0 0 0) 0x7fcfac01f090 con 0x7fcfb006c790 2023-11-28T17:10:35.860 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:35.860 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":34,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:10:21.316243+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:10:17.338444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-28T17:10:18.766338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:10:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.860+0000 7fcfb7778640 1 -- 172.21.15.116:0/3721664417 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcf9406b2a0 msgr2=0x7fcf9406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.860+0000 7fcfb7778640 1 --2- 172.21.15.116:0/3721664417 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcf9406b2a0 0x7fcf9406d760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00ae970 tx=0x7fcfa0008040 comp rx=0 tx=0).stop 2023-11-28T17:10:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.860+0000 7fcfb7778640 1 -- 172.21.15.116:0/3721664417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb006c790 msgr2=0x7fcfb015ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.860+0000 7fcfb7778640 1 --2- 172.21.15.116:0/3721664417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb006c790 0x7fcfb015ac40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fcfac00c7f0 tx=0x7fcfac00ccb0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.860+0000 7fcfb7778640 1 -- 172.21.15.116:0/3721664417 shutdown_connections 2023-11-28T17:10:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.860+0000 7fcfb7778640 1 --2- 172.21.15.116:0/3721664417 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcf9406b2a0 0x7fcf9406d760 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.860+0000 7fcfb7778640 1 --2- 172.21.15.116:0/3721664417 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcfb01401c0 0x7fcfb013c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.860+0000 7fcfb7778640 1 --2- 172.21.15.116:0/3721664417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfb0071710 0x7fcfb0159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.860+0000 7fcfb7778640 1 --2- 172.21.15.116:0/3721664417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb006c790 0x7fcfb015ac40 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.860+0000 7fcfb7778640 1 -- 172.21.15.116:0/3721664417 >> 172.21.15.116:0/3721664417 conn(0x7fcfb006a4c0 msgr2=0x7fcfb013f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.861+0000 7fcfb7778640 1 -- 172.21.15.116:0/3721664417 shutdown_connections 2023-11-28T17:10:35.864 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:35.861+0000 7fcfb7778640 1 -- 172.21.15.116:0/3721664417 wait complete. 2023-11-28T17:10:35.881 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:10:17.338444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:10:18.766338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:10:35.881 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.096+0000 7fe136c75640 1 Processor -- start 2023-11-28T17:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.096+0000 7fe136c75640 1 -- start start 2023-11-28T17:10:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.098+0000 7fe136c75640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe13015b0c0 0x7fe13015ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.098+0000 7fe136c75640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe130072250 0x7fe1300cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.098+0000 7fe136c75640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1300cd320 con 0x7fe1300ce940 2023-11-28T17:10:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.098+0000 7fe136c75640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1300ced80 con 0x7fe13015b0c0 2023-11-28T17:10:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.098+0000 7fe136c75640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1300cbee0 con 0x7fe130072250 2023-11-28T17:10:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.098+0000 7fe12ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe1300ce940 0x7fe130071b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57254/0 (socket says 172.21.15.116:57254) 2023-11-28T17:10:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.098+0000 7fe12ffff640 1 -- 172.21.15.116:0/166362865 learned_addr learned my addr 172.21.15.116:0/166362865 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.098+0000 7fe134a11640 1 --2- 172.21.15.116:0/166362865 >> v2:172.21.15.116:3300/0 conn(0x7fe13015b0c0 0x7fe13015ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.099+0000 7fe135212640 1 --2- 172.21.15.116:0/166362865 >> v2:172.21.15.156:3300/0 conn(0x7fe130072250 0x7fe1300cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.099+0000 7fe12f7fe640 1 -- 172.21.15.116:0/166362865 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059418827 0 0) 0x7fe1300cd320 con 0x7fe1300ce940 2023-11-28T17:10:36.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.099+0000 7fe134a11640 1 -- 172.21.15.116:0/166362865 >> v2:172.21.15.156:3300/0 conn(0x7fe130072250 msgr2=0x7fe1300cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:36.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.099+0000 7fe134a11640 1 --2- 172.21.15.116:0/166362865 >> v2:172.21.15.156:3300/0 conn(0x7fe130072250 0x7fe1300cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.099+0000 7fe134a11640 1 -- 172.21.15.116:0/166362865 >> v1:172.21.15.116:6789/0 conn(0x7fe1300ce940 legacy=0x7fe130071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.099+0000 7fe134a11640 1 -- 172.21.15.116:0/166362865 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1300cc710 con 0x7fe13015b0c0 2023-11-28T17:10:36.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.100+0000 7fe134a11640 1 --2- 172.21.15.116:0/166362865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13015b0c0 0x7fe13015ed10 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe13006e850 tx=0x7fe12402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6cfdc59b607ec8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:36.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.100+0000 7fe12f7fe640 1 -- 172.21.15.116:0/166362865 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12402e070 con 0x7fe13015b0c0 2023-11-28T17:10:36.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.100+0000 7fe12f7fe640 1 -- 172.21.15.116:0/166362865 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1240315a0 con 0x7fe13015b0c0 2023-11-28T17:10:36.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.100+0000 7fe12f7fe640 1 -- 172.21.15.116:0/166362865 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe124031930 con 0x7fe13015b0c0 2023-11-28T17:10:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.101+0000 7fe136c75640 1 -- 172.21.15.116:0/166362865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13015b0c0 msgr2=0x7fe13015ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.101+0000 7fe136c75640 1 --2- 172.21.15.116:0/166362865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13015b0c0 0x7fe13015ed10 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe13006e850 tx=0x7fe12402b840 comp rx=0 tx=0).stop 2023-11-28T17:10:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.101+0000 7fe136c75640 1 -- 172.21.15.116:0/166362865 shutdown_connections 2023-11-28T17:10:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.101+0000 7fe136c75640 1 --2- 172.21.15.116:0/166362865 >> v2:172.21.15.156:3300/0 conn(0x7fe130072250 0x7fe1300cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.101+0000 7fe136c75640 1 --2- 172.21.15.116:0/166362865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13015b0c0 0x7fe13015ed10 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.101+0000 7fe136c75640 1 -- 172.21.15.116:0/166362865 >> 172.21.15.116:0/166362865 conn(0x7fe1300b0cf0 msgr2=0x7fe1300b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.101+0000 7fe136c75640 1 -- 172.21.15.116:0/166362865 shutdown_connections 2023-11-28T17:10:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.101+0000 7fe136c75640 1 -- 172.21.15.116:0/166362865 wait complete. 2023-11-28T17:10:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.103+0000 7fe136c75640 1 Processor -- start 2023-11-28T17:10:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.103+0000 7fe136c75640 1 -- start start 2023-11-28T17:10:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe136c75640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe130072250 0x7fe1301f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe136c75640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1300ce940 0x7fe130153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe136c75640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe13015b0c0 0x7fe130154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe136c75640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1301e6360 con 0x7fe1300ce940 2023-11-28T17:10:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe136c75640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1301f2a90 con 0x7fe13015b0c0 2023-11-28T17:10:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe136c75640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe130153160 con 0x7fe130072250 2023-11-28T17:10:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe12ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1300ce940 0x7fe130153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe12ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1300ce940 0x7fe130153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57070/0 (socket says 172.21.15.116:57070) 2023-11-28T17:10:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe12ffff640 1 -- 172.21.15.116:0/990540629 learned_addr learned my addr 172.21.15.116:0/990540629 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe134a11640 1 --2- 172.21.15.116:0/990540629 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe130072250 0x7fe1301f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe135212640 1 --2- 172.21.15.116:0/990540629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe13015b0c0 0x7fe130154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe12ffff640 1 -- 172.21.15.116:0/990540629 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe130072250 msgr2=0x7fe1301f2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe12ffff640 1 --2- 172.21.15.116:0/990540629 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe130072250 0x7fe1301f2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe12ffff640 1 -- 172.21.15.116:0/990540629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe13015b0c0 msgr2=0x7fe130154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe12ffff640 1 --2- 172.21.15.116:0/990540629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe13015b0c0 0x7fe130154380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.104+0000 7fe12ffff640 1 -- 172.21.15.116:0/990540629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe124009400 con 0x7fe1300ce940 2023-11-28T17:10:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.105+0000 7fe12ffff640 1 --2- 172.21.15.116:0/990540629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1300ce940 0x7fe130153e40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fe11c00c4d0 tx=0x7fe11c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.105+0000 7fe12d7fa640 1 -- 172.21.15.116:0/990540629 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c0074e0 con 0x7fe1300ce940 2023-11-28T17:10:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.105+0000 7fe12d7fa640 1 -- 172.21.15.116:0/990540629 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11c007660 con 0x7fe1300ce940 2023-11-28T17:10:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.105+0000 7fe12d7fa640 1 -- 172.21.15.116:0/990540629 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c004a20 con 0x7fe1300ce940 2023-11-28T17:10:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.105+0000 7fe136c75640 1 -- 172.21.15.116:0/990540629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1301e65b0 con 0x7fe1300ce940 2023-11-28T17:10:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.105+0000 7fe136c75640 1 -- 172.21.15.116:0/990540629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1301e6b70 con 0x7fe1300ce940 2023-11-28T17:10:36.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.106+0000 7fe136c75640 1 -- 172.21.15.116:0/990540629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1301e6d90 con 0x7fe1300ce940 2023-11-28T17:10:36.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.107+0000 7fe12d7fa640 1 -- 172.21.15.116:0/990540629 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe11c004b80 con 0x7fe1300ce940 2023-11-28T17:10:36.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.108+0000 7fe12d7fa640 1 --2- 172.21.15.116:0/990540629 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe11406b2a0 0x7fe11406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.108+0000 7fe12d7fa640 1 -- 172.21.15.116:0/990540629 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe11c088e50 con 0x7fe1300ce940 2023-11-28T17:10:36.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.108+0000 7fe134a11640 1 --2- 172.21.15.116:0/990540629 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe11406b2a0 0x7fe11406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.109+0000 7fe134a11640 1 --2- 172.21.15.116:0/990540629 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe11406b2a0 0x7fe11406d760 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe124036000 tx=0x7fe12403d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:36.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.113+0000 7fe12d7fa640 1 -- 172.21.15.116:0/990540629 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe11c053c30 con 0x7fe1300ce940 2023-11-28T17:10:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.303+0000 7fe136c75640 1 -- 172.21.15.116:0/990540629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe130069360 con 0x7fe1300ce940 2023-11-28T17:10:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.304+0000 7fe12d7fa640 1 -- 172.21.15.116:0/990540629 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe11c018070 con 0x7fe1300ce940 2023-11-28T17:10:36.306 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:10:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.307+0000 7fe136c75640 1 -- 172.21.15.116:0/990540629 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe11406b2a0 msgr2=0x7fe11406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.307+0000 7fe136c75640 1 --2- 172.21.15.116:0/990540629 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe11406b2a0 0x7fe11406d760 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe124036000 tx=0x7fe12403d040 comp rx=0 tx=0).stop 2023-11-28T17:10:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.307+0000 7fe136c75640 1 -- 172.21.15.116:0/990540629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1300ce940 msgr2=0x7fe130153e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.307+0000 7fe136c75640 1 --2- 172.21.15.116:0/990540629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1300ce940 0x7fe130153e40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fe11c00c4d0 tx=0x7fe11c00c990 comp rx=0 tx=0).stop 2023-11-28T17:10:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.308+0000 7fe136c75640 1 -- 172.21.15.116:0/990540629 shutdown_connections 2023-11-28T17:10:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.308+0000 7fe136c75640 1 --2- 172.21.15.116:0/990540629 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe11406b2a0 0x7fe11406d760 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.308+0000 7fe136c75640 1 --2- 172.21.15.116:0/990540629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe13015b0c0 0x7fe130154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.308+0000 7fe136c75640 1 --2- 172.21.15.116:0/990540629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1300ce940 0x7fe130153e40 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.308+0000 7fe136c75640 1 --2- 172.21.15.116:0/990540629 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe130072250 0x7fe1301f2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.308+0000 7fe136c75640 1 -- 172.21.15.116:0/990540629 >> 172.21.15.116:0/990540629 conn(0x7fe1300b0cf0 msgr2=0x7fe13015d730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.308+0000 7fe136c75640 1 -- 172.21.15.116:0/990540629 shutdown_connections 2023-11-28T17:10:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.308+0000 7fe136c75640 1 -- 172.21.15.116:0/990540629 wait complete. 2023-11-28T17:10:36.326 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:10:36.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.526+0000 7fc9e4d73640 1 Processor -- start 2023-11-28T17:10:36.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.527+0000 7fc9e4d73640 1 -- start start 2023-11-28T17:10:36.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.528+0000 7fc9e4d73640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9e0152050 0x7fc9e015bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.528+0000 7fc9e4d73640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e00b2b50 con 0x7fc9e00abd30 2023-11-28T17:10:36.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.528+0000 7fc9e4d73640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e015c520 con 0x7fc9e00b2fe0 2023-11-28T17:10:36.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.528+0000 7fc9e4d73640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e00ac2d0 con 0x7fc9e0152050 2023-11-28T17:10:36.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.529+0000 7fc9dffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc9e00abd30 0x7fc9e00cdc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57262/0 (socket says 172.21.15.116:57262) 2023-11-28T17:10:36.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.529+0000 7fc9dffff640 1 -- 172.21.15.116:0/2844230309 learned_addr learned my addr 172.21.15.116:0/2844230309 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:36.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.529+0000 7fc9df7fe640 1 --2- 172.21.15.116:0/2844230309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9e0152050 0x7fc9e015bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.529+0000 7fc9deffd640 1 -- 172.21.15.116:0/2844230309 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71426301 0 0) 0x7fc9e00b2b50 con 0x7fc9e00abd30 2023-11-28T17:10:36.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.529+0000 7fc9deffd640 1 -- 172.21.15.116:0/2844230309 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9cc003520 con 0x7fc9e00abd30 2023-11-28T17:10:36.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9deffd640 1 -- 172.21.15.116:0/2844230309 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3184527567 0 0) 0x7fc9cc003520 con 0x7fc9e00abd30 2023-11-28T17:10:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9deffd640 1 -- 172.21.15.116:0/2844230309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9e0152050 msgr2=0x7fc9e015bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9deffd640 1 --2- 172.21.15.116:0/2844230309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9e0152050 0x7fc9e015bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9deffd640 1 -- 172.21.15.116:0/2844230309 >> v1:172.21.15.156:6789/0 conn(0x7fc9e00b2fe0 legacy=0x7fc9e00ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9deffd640 1 -- 172.21.15.116:0/2844230309 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e00b2220 con 0x7fc9e00abd30 2023-11-28T17:10:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9deffd640 1 -- 172.21.15.116:0/2844230309 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fc9d80019f0 con 0x7fc9e00abd30 2023-11-28T17:10:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9deffd640 1 -- 172.21.15.116:0/2844230309 >> v1:172.21.15.116:6789/0 conn(0x7fc9e00abd30 legacy=0x7fc9e00cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9deffd640 1 --2- 172.21.15.116:0/2844230309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9cc0039a0 0x7fc9cc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9deffd640 1 --2- 172.21.15.116:0/2844230309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9e0152050 0x7fc9cc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9deffd640 1 --2- 172.21.15.116:0/2844230309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9cc004d00 0x7fc9cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9deffd640 1 -- 172.21.15.116:0/2844230309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e00ac2d0 con 0x7fc9cc0039a0 2023-11-28T17:10:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9deffd640 1 -- 172.21.15.116:0/2844230309 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc00ba60 con 0x7fc9cc004d00 2023-11-28T17:10:36.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9deffd640 1 -- 172.21.15.116:0/2844230309 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc00bc20 con 0x7fc9e0152050 2023-11-28T17:10:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9df7fe640 1 --2- 172.21.15.116:0/2844230309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9cc0039a0 0x7fc9cc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9df7fe640 1 -- 172.21.15.116:0/2844230309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9e0152050 msgr2=0x7fc9cc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9df7fe640 1 --2- 172.21.15.116:0/2844230309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9e0152050 0x7fc9cc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9dffff640 1 --2- 172.21.15.116:0/2844230309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9cc004d00 0x7fc9cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9df7fe640 1 -- 172.21.15.116:0/2844230309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9cc004d00 msgr2=0x7fc9cc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9df7fe640 1 --2- 172.21.15.116:0/2844230309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9cc004d00 0x7fc9cc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9df7fe640 1 -- 172.21.15.116:0/2844230309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9cc003520 con 0x7fc9cc0039a0 2023-11-28T17:10:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.530+0000 7fc9dffff640 1 --2- 172.21.15.116:0/2844230309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9cc004d00 0x7fc9cc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:36.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.532+0000 7fc9df7fe640 1 --2- 172.21.15.116:0/2844230309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9cc0039a0 0x7fc9cc003dc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009a90 tx=0x7fc9cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a3b5d67f60ffe7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.532+0000 7fc9deffd640 1 -- 172.21.15.116:0/2844230309 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d002c6b0 con 0x7fc9cc0039a0 2023-11-28T17:10:36.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.532+0000 7fc9deffd640 1 -- 172.21.15.116:0/2844230309 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d002c810 con 0x7fc9cc0039a0 2023-11-28T17:10:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.532+0000 7fc9deffd640 1 -- 172.21.15.116:0/2844230309 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d002cba0 con 0x7fc9cc0039a0 2023-11-28T17:10:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.532+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2844230309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9cc0039a0 msgr2=0x7fc9cc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.532+0000 7fc9e4d73640 1 --2- 172.21.15.116:0/2844230309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9cc0039a0 0x7fc9cc003dc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009a90 tx=0x7fc9cc0057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.532+0000 7fc9dffff640 1 -- 172.21.15.116:0/2844230309 reap_dead start 2023-11-28T17:10:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.532+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2844230309 shutdown_connections 2023-11-28T17:10:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.532+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2844230309 >> 172.21.15.116:0/2844230309 conn(0x7fc9e00b0950 msgr2=0x7fc9e00b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:36.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.533+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2844230309 shutdown_connections 2023-11-28T17:10:36.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.533+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2844230309 wait complete. 2023-11-28T17:10:36.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.534+0000 7fc9e4d73640 1 Processor -- start 2023-11-28T17:10:36.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.534+0000 7fc9e4d73640 1 -- start start 2023-11-28T17:10:36.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.535+0000 7fc9e4d73640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9cc0039a0 0x7fc9e009ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.535+0000 7fc9e4d73640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9e0154a50 0x7fc9e01e5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.535+0000 7fc9e4d73640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e01e58a0 0x7fc9e01f46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.535+0000 7fc9e4d73640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e0154630 con 0x7fc9cc0039a0 2023-11-28T17:10:36.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.535+0000 7fc9e4d73640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e0146ee0 con 0x7fc9e01e58a0 2023-11-28T17:10:36.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.535+0000 7fc9e4d73640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e01e5d20 con 0x7fc9e0154a50 2023-11-28T17:10:36.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.536+0000 7fc9df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9cc0039a0 0x7fc9e009ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.536+0000 7fc9df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9cc0039a0 0x7fc9e009ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57094/0 (socket says 172.21.15.116:57094) 2023-11-28T17:10:36.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.536+0000 7fc9df7fe640 1 -- 172.21.15.116:0/2064934488 learned_addr learned my addr 172.21.15.116:0/2064934488 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:36.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.536+0000 7fc9d7fff640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9e0154a50 0x7fc9e01e5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.536+0000 7fc9dffff640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e01e58a0 0x7fc9e01f46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.536+0000 7fc9df7fe640 1 -- 172.21.15.116:0/2064934488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9e0154a50 msgr2=0x7fc9e01e5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.536+0000 7fc9df7fe640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9e0154a50 0x7fc9e01e5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.536+0000 7fc9df7fe640 1 -- 172.21.15.116:0/2064934488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e01e58a0 msgr2=0x7fc9e01f46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.536+0000 7fc9df7fe640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e01e58a0 0x7fc9e01f46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.536+0000 7fc9df7fe640 1 -- 172.21.15.116:0/2064934488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e01f5510 con 0x7fc9cc0039a0 2023-11-28T17:10:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.536+0000 7fc9d7fff640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9e0154a50 0x7fc9e01e5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.536+0000 7fc9df7fe640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9cc0039a0 0x7fc9e009ae00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009a90 tx=0x7fc9d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.537+0000 7fc9dcff9640 1 -- 172.21.15.116:0/2064934488 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0003810 con 0x7fc9cc0039a0 2023-11-28T17:10:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.537+0000 7fc9dcff9640 1 -- 172.21.15.116:0/2064934488 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d0003970 con 0x7fc9cc0039a0 2023-11-28T17:10:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.537+0000 7fc9dcff9640 1 -- 172.21.15.116:0/2064934488 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0003d20 con 0x7fc9cc0039a0 2023-11-28T17:10:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.537+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2064934488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e0153c90 con 0x7fc9cc0039a0 2023-11-28T17:10:36.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.537+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2064934488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e0154250 con 0x7fc9cc0039a0 2023-11-28T17:10:36.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.538+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2064934488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9e009d0b0 con 0x7fc9cc0039a0 2023-11-28T17:10:36.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.539+0000 7fc9dcff9640 1 -- 172.21.15.116:0/2064934488 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fc9d0040450 con 0x7fc9cc0039a0 2023-11-28T17:10:36.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.540+0000 7fc9dcff9640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc9c006b2a0 0x7fc9c006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.540+0000 7fc9dcff9640 1 -- 172.21.15.116:0/2064934488 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9d00aba20 con 0x7fc9cc0039a0 2023-11-28T17:10:36.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.540+0000 7fc9d7fff640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc9c006b2a0 0x7fc9c006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.541+0000 7fc9d7fff640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc9c006b2a0 0x7fc9c006d760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0151880 tx=0x7fc9c8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:36.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.545+0000 7fc9dcff9640 1 -- 172.21.15.116:0/2064934488 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc9d0076920 con 0x7fc9cc0039a0 2023-11-28T17:10:36.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.732+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2064934488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc9e004e960 con 0x7fc9cc0039a0 2023-11-28T17:10:36.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.733+0000 7fc9dcff9640 1 -- 172.21.15.116:0/2064934488 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc9d00044e0 con 0x7fc9cc0039a0 2023-11-28T17:10:36.734 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:10:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.736+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2064934488 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc9c006b2a0 msgr2=0x7fc9c006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.736+0000 7fc9e4d73640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc9c006b2a0 0x7fc9c006d760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0151880 tx=0x7fc9c8009040 comp rx=0 tx=0).stop 2023-11-28T17:10:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.736+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2064934488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9cc0039a0 msgr2=0x7fc9e009ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.736+0000 7fc9e4d73640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9cc0039a0 0x7fc9e009ae00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009a90 tx=0x7fc9d0004730 comp rx=0 tx=0).stop 2023-11-28T17:10:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.736+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2064934488 shutdown_connections 2023-11-28T17:10:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.736+0000 7fc9e4d73640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc9c006b2a0 0x7fc9c006d760 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.736+0000 7fc9e4d73640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9e01e58a0 0x7fc9e01f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.736+0000 7fc9e4d73640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc9e0154a50 0x7fc9e01e5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.736+0000 7fc9e4d73640 1 --2- 172.21.15.116:0/2064934488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9cc0039a0 0x7fc9e009ae00 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.736+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2064934488 >> 172.21.15.116:0/2064934488 conn(0x7fc9e00b0950 msgr2=0x7fc9e015de30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.736+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2064934488 shutdown_connections 2023-11-28T17:10:36.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.736+0000 7fc9e4d73640 1 -- 172.21.15.116:0/2064934488 wait complete. 2023-11-28T17:10:36.754 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:10:36.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.961+0000 7fe2bb59e640 1 Processor -- start 2023-11-28T17:10:36.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.961+0000 7fe2bb59e640 1 -- start start 2023-11-28T17:10:36.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.962+0000 7fe2bb59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2bc15a270 0x7fe2bc0ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.962+0000 7fe2bb59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2bc06d910 0x7fe2bc0b2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.962+0000 7fe2bb59e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe2bc0b2dc0 0x7fe2bc0ac7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.962+0000 7fe2bb59e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2bc15a650 con 0x7fe2bc15a270 2023-11-28T17:10:36.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.962+0000 7fe2bb59e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2bc0b3250 con 0x7fe2bc0b2dc0 2023-11-28T17:10:36.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.962+0000 7fe2bb59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc06de10 con 0x7fe2bc06d910 2023-11-28T17:10:36.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.963+0000 7fe2ba59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2bc15a270 0x7fe2bc0ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.963+0000 7fe2ba59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2bc15a270 0x7fe2bc0ae2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57096/0 (socket says 172.21.15.116:57096) 2023-11-28T17:10:36.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.963+0000 7fe2ba59c640 1 -- 172.21.15.116:0/1967665982 learned_addr learned my addr 172.21.15.116:0/1967665982 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:36.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.963+0000 7fe2b1d9b640 1 --2- 172.21.15.116:0/1967665982 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2bc06d910 0x7fe2bc0b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.963+0000 7fe2bad9d640 1 --2- 172.21.15.116:0/1967665982 >> v2:172.21.15.156:3300/0 conn(0x7fe2bc0b2dc0 0x7fe2bc0ac7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.963+0000 7fe2ba59c640 1 -- 172.21.15.116:0/1967665982 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2bc06d910 msgr2=0x7fe2bc0b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:36.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.963+0000 7fe2ba59c640 1 --2- 172.21.15.116:0/1967665982 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2bc06d910 0x7fe2bc0b2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.963+0000 7fe2ba59c640 1 -- 172.21.15.116:0/1967665982 >> v2:172.21.15.156:3300/0 conn(0x7fe2bc0b2dc0 msgr2=0x7fe2bc0ac7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:36.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.963+0000 7fe2ba59c640 1 --2- 172.21.15.116:0/1967665982 >> v2:172.21.15.156:3300/0 conn(0x7fe2bc0b2dc0 0x7fe2bc0ac7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.963+0000 7fe2ba59c640 1 -- 172.21.15.116:0/1967665982 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc0b34e0 con 0x7fe2bc15a270 2023-11-28T17:10:36.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.963+0000 7fe2b1d9b640 1 --2- 172.21.15.116:0/1967665982 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2bc06d910 0x7fe2bc0b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:36.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.963+0000 7fe2bad9d640 1 --2- 172.21.15.116:0/1967665982 >> v2:172.21.15.156:3300/0 conn(0x7fe2bc0b2dc0 0x7fe2bc0ac7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:36.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.964+0000 7fe2ba59c640 1 --2- 172.21.15.116:0/1967665982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2bc15a270 0x7fe2bc0ae2d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc0b08e0 tx=0x7fe2ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d5a8660eef8a9e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:36.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.964+0000 7fe2b9d9b640 1 -- 172.21.15.116:0/1967665982 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac031870 con 0x7fe2bc15a270 2023-11-28T17:10:36.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.965+0000 7fe2b9d9b640 1 -- 172.21.15.116:0/1967665982 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac0319d0 con 0x7fe2bc15a270 2023-11-28T17:10:36.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.965+0000 7fe2b9d9b640 1 -- 172.21.15.116:0/1967665982 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac031d60 con 0x7fe2bc15a270 2023-11-28T17:10:36.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.965+0000 7fe2bb59e640 1 -- 172.21.15.116:0/1967665982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2bc15a270 msgr2=0x7fe2bc0ae2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.965+0000 7fe2bb59e640 1 --2- 172.21.15.116:0/1967665982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2bc15a270 0x7fe2bc0ae2d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc0b08e0 tx=0x7fe2ac02b910 comp rx=0 tx=0).stop 2023-11-28T17:10:36.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.965+0000 7fe2bb59e640 1 -- 172.21.15.116:0/1967665982 shutdown_connections 2023-11-28T17:10:36.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.965+0000 7fe2bb59e640 1 --2- 172.21.15.116:0/1967665982 >> v2:172.21.15.156:3300/0 conn(0x7fe2bc0b2dc0 0x7fe2bc0ac7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.965+0000 7fe2bb59e640 1 --2- 172.21.15.116:0/1967665982 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2bc06d910 0x7fe2bc0b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.965+0000 7fe2bb59e640 1 --2- 172.21.15.116:0/1967665982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2bc15a270 0x7fe2bc0ae2d0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.965+0000 7fe2bb59e640 1 -- 172.21.15.116:0/1967665982 >> 172.21.15.116:0/1967665982 conn(0x7fe2bc0b0950 msgr2=0x7fe2bc0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:36.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.965+0000 7fe2bb59e640 1 -- 172.21.15.116:0/1967665982 shutdown_connections 2023-11-28T17:10:36.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.965+0000 7fe2bb59e640 1 -- 172.21.15.116:0/1967665982 wait complete. 2023-11-28T17:10:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.967+0000 7fe2bb59e640 1 Processor -- start 2023-11-28T17:10:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.967+0000 7fe2bb59e640 1 -- start start 2023-11-28T17:10:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.968+0000 7fe2bb59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2bc06d910 0x7fe2bc14bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.968+0000 7fe2bb59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2bc0b2dc0 0x7fe2bc148630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.968+0000 7fe2bb59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2bc15a270 0x7fe2bc157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.968+0000 7fe2bb59e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc15ace0 con 0x7fe2bc15a270 2023-11-28T17:10:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.968+0000 7fe2bb59e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc15afc0 con 0x7fe2bc06d910 2023-11-28T17:10:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.968+0000 7fe2bb59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc15b2a0 con 0x7fe2bc0b2dc0 2023-11-28T17:10:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.968+0000 7fe2bad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2bc15a270 0x7fe2bc157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.968+0000 7fe2bad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2bc15a270 0x7fe2bc157630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57108/0 (socket says 172.21.15.116:57108) 2023-11-28T17:10:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.968+0000 7fe2bad9d640 1 -- 172.21.15.116:0/25939875 learned_addr learned my addr 172.21.15.116:0/25939875 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.969+0000 7fe2ba59c640 1 --2- 172.21.15.116:0/25939875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2bc06d910 0x7fe2bc14bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.969+0000 7fe2b1d9b640 1 --2- 172.21.15.116:0/25939875 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2bc0b2dc0 0x7fe2bc148630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.969+0000 7fe2bad9d640 1 -- 172.21.15.116:0/25939875 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2bc0b2dc0 msgr2=0x7fe2bc148630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.969+0000 7fe2bad9d640 1 --2- 172.21.15.116:0/25939875 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2bc0b2dc0 0x7fe2bc148630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.969+0000 7fe2bad9d640 1 -- 172.21.15.116:0/25939875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2bc06d910 msgr2=0x7fe2bc14bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.969+0000 7fe2bad9d640 1 --2- 172.21.15.116:0/25939875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2bc06d910 0x7fe2bc14bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.969+0000 7fe2bad9d640 1 -- 172.21.15.116:0/25939875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ac0094d0 con 0x7fe2bc15a270 2023-11-28T17:10:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.969+0000 7fe2bad9d640 1 --2- 172.21.15.116:0/25939875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2bc15a270 0x7fe2bc157630 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400cd50 tx=0x7fe2b400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.969+0000 7fe2b37fe640 1 -- 172.21.15.116:0/25939875 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b400f680 con 0x7fe2bc15a270 2023-11-28T17:10:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.969+0000 7fe2b37fe640 1 -- 172.21.15.116:0/25939875 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b400f800 con 0x7fe2bc15a270 2023-11-28T17:10:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.969+0000 7fe2b37fe640 1 -- 172.21.15.116:0/25939875 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b400fbb0 con 0x7fe2bc15a270 2023-11-28T17:10:36.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.969+0000 7fe2bb59e640 1 -- 172.21.15.116:0/25939875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2bc15b580 con 0x7fe2bc15a270 2023-11-28T17:10:36.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.970+0000 7fe2bb59e640 1 -- 172.21.15.116:0/25939875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2bc14c420 con 0x7fe2bc15a270 2023-11-28T17:10:36.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.970+0000 7fe2bb59e640 1 -- 172.21.15.116:0/25939875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2bc06e510 con 0x7fe2bc15a270 2023-11-28T17:10:36.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.971+0000 7fe2b37fe640 1 -- 172.21.15.116:0/25939875 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe2b400b040 con 0x7fe2bc15a270 2023-11-28T17:10:36.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.972+0000 7fe2b37fe640 1 --2- 172.21.15.116:0/25939875 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe29c06b2a0 0x7fe29c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:36.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.972+0000 7fe2b37fe640 1 -- 172.21.15.116:0/25939875 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2b408a5f0 con 0x7fe2bc15a270 2023-11-28T17:10:36.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.973+0000 7fe2ba59c640 1 --2- 172.21.15.116:0/25939875 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe29c06b2a0 0x7fe29c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:36.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.973+0000 7fe2ba59c640 1 --2- 172.21.15.116:0/25939875 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe29c06b2a0 0x7fe29c06d760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac036000 tx=0x7fe2ac03c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:36.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:36.976+0000 7fe2b37fe640 1 -- 172.21.15.116:0/25939875 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe2b4055450 con 0x7fe2bc15a270 2023-11-28T17:10:37.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.168+0000 7fe2bb59e640 1 -- 172.21.15.116:0/25939875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe2bc15baa0 con 0x7fe2bc15a270 2023-11-28T17:10:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.169+0000 7fe2b37fe640 1 -- 172.21.15.116:0/25939875 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe2bc15baa0 con 0x7fe2bc15a270 2023-11-28T17:10:37.225 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:10:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.172+0000 7fe2bb59e640 1 -- 172.21.15.116:0/25939875 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe29c06b2a0 msgr2=0x7fe29c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.172+0000 7fe2bb59e640 1 --2- 172.21.15.116:0/25939875 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe29c06b2a0 0x7fe29c06d760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac036000 tx=0x7fe2ac03c040 comp rx=0 tx=0).stop 2023-11-28T17:10:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.172+0000 7fe2bb59e640 1 -- 172.21.15.116:0/25939875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2bc15a270 msgr2=0x7fe2bc157630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.172+0000 7fe2bb59e640 1 --2- 172.21.15.116:0/25939875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2bc15a270 0x7fe2bc157630 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400cd50 tx=0x7fe2b400a650 comp rx=0 tx=0).stop 2023-11-28T17:10:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.172+0000 7fe2bb59e640 1 -- 172.21.15.116:0/25939875 shutdown_connections 2023-11-28T17:10:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.172+0000 7fe2bb59e640 1 --2- 172.21.15.116:0/25939875 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe29c06b2a0 0x7fe29c06d760 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:37.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.172+0000 7fe2bb59e640 1 --2- 172.21.15.116:0/25939875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2bc15a270 0x7fe2bc157630 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:37.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.172+0000 7fe2bb59e640 1 --2- 172.21.15.116:0/25939875 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2bc0b2dc0 0x7fe2bc148630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:37.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.172+0000 7fe2bb59e640 1 --2- 172.21.15.116:0/25939875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2bc06d910 0x7fe2bc14bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:37.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.172+0000 7fe2bb59e640 1 -- 172.21.15.116:0/25939875 >> 172.21.15.116:0/25939875 conn(0x7fe2bc0b0950 msgr2=0x7fe2bc071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:37.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.172+0000 7fe2bb59e640 1 -- 172.21.15.116:0/25939875 shutdown_connections 2023-11-28T17:10:37.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.172+0000 7fe2bb59e640 1 -- 172.21.15.116:0/25939875 wait complete. 2023-11-28T17:10:37.227 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:10:37.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.446+0000 7f0bc3f1d640 1 Processor -- start 2023-11-28T17:10:37.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.446+0000 7f0bc3f1d640 1 -- start start 2023-11-28T17:10:37.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.448+0000 7f0bc3f1d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0bbc13cee0 0x7f0bbc06c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:37.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.448+0000 7f0bc3f1d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0bbc140200 0x7f0bbc06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:37.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.448+0000 7f0bc3f1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc0aea20 con 0x7f0bbc15a490 2023-11-28T17:10:37.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.448+0000 7f0bc3f1d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bbc13d2c0 con 0x7f0bbc13cee0 2023-11-28T17:10:37.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.448+0000 7f0bc3f1d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bbc15b3a0 con 0x7f0bbc140200 2023-11-28T17:10:37.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.448+0000 7f0bc24ba640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0bbc15a490 0x7f0bbc15a900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57274/0 (socket says 172.21.15.116:57274) 2023-11-28T17:10:37.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.448+0000 7f0bc24ba640 1 -- 172.21.15.116:0/620603923 learned_addr learned my addr 172.21.15.116:0/620603923 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:37.450 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.448+0000 7f0bc1cb9640 1 --2- 172.21.15.116:0/620603923 >> v2:172.21.15.116:3300/0 conn(0x7f0bbc13cee0 0x7f0bbc06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:37.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.449+0000 7f0bc1cb9640 1 -- 172.21.15.116:0/620603923 >> v2:172.21.15.156:3300/0 conn(0x7f0bbc140200 msgr2=0x7f0bbc06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:37.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.449+0000 7f0bc14b8640 1 --2- 172.21.15.116:0/620603923 >> v2:172.21.15.156:3300/0 conn(0x7f0bbc140200 0x7f0bbc06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:37.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.449+0000 7f0bc1cb9640 1 --2- 172.21.15.116:0/620603923 >> v2:172.21.15.156:3300/0 conn(0x7f0bbc140200 0x7f0bbc06c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:37.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.449+0000 7f0bc1cb9640 1 -- 172.21.15.116:0/620603923 >> v1:172.21.15.116:6789/0 conn(0x7f0bbc15a490 legacy=0x7f0bbc15a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:37.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.449+0000 7f0bc0cb7640 1 -- 172.21.15.116:0/620603923 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871913717 0 0) 0x7f0bbc0aea20 con 0x7f0bbc15a490 2023-11-28T17:10:37.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.449+0000 7f0bc1cb9640 1 -- 172.21.15.116:0/620603923 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc15b0c0 con 0x7f0bbc13cee0 2023-11-28T17:10:37.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.450+0000 7f0bc1cb9640 1 --2- 172.21.15.116:0/620603923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bbc13cee0 0x7f0bbc06c250 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc067f50 tx=0x7f0bb802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d186be6d84687b90 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:37.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.450+0000 7f0bc0cb7640 1 -- 172.21.15.116:0/620603923 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb802cbe0 con 0x7f0bbc13cee0 2023-11-28T17:10:37.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.450+0000 7f0bc0cb7640 1 -- 172.21.15.116:0/620603923 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb802cd40 con 0x7f0bbc13cee0 2023-11-28T17:10:37.453 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.450+0000 7f0bc0cb7640 1 -- 172.21.15.116:0/620603923 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8034660 con 0x7f0bbc13cee0 2023-11-28T17:10:37.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.451+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/620603923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bbc13cee0 msgr2=0x7f0bbc06c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:37.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.451+0000 7f0bc3f1d640 1 --2- 172.21.15.116:0/620603923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bbc13cee0 0x7f0bbc06c250 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc067f50 tx=0x7f0bb802be30 comp rx=0 tx=0).stop 2023-11-28T17:10:37.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.451+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/620603923 shutdown_connections 2023-11-28T17:10:37.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.451+0000 7f0bc3f1d640 1 --2- 172.21.15.116:0/620603923 >> v2:172.21.15.156:3300/0 conn(0x7f0bbc140200 0x7f0bbc06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:37.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.451+0000 7f0bc3f1d640 1 --2- 172.21.15.116:0/620603923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bbc13cee0 0x7f0bbc06c250 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:37.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.451+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/620603923 >> 172.21.15.116:0/620603923 conn(0x7f0bbc06a4c0 msgr2=0x7f0bbc06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:37.454 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.451+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/620603923 shutdown_connections 2023-11-28T17:10:37.455 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.451+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/620603923 wait complete. 2023-11-28T17:10:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.452+0000 7f0bc3f1d640 1 Processor -- start 2023-11-28T17:10:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.453+0000 7f0bc3f1d640 1 -- start start 2023-11-28T17:10:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.453+0000 7f0bc3f1d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0bbc13cee0 0x7f0bbc156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.453+0000 7f0bc3f1d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bbc140200 0x7f0bbc14f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.453+0000 7f0bc3f1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bbc15a490 0x7f0bbc0b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.453+0000 7f0bc3f1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc138d40 con 0x7f0bbc15a490 2023-11-28T17:10:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.453+0000 7f0bc3f1d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc1391a0 con 0x7f0bbc140200 2023-11-28T17:10:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.453+0000 7f0bc3f1d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc139440 con 0x7f0bbc13cee0 2023-11-28T17:10:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.454+0000 7f0bc24ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bbc15a490 0x7f0bbc0b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.454+0000 7f0bc24ba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bbc15a490 0x7f0bbc0b0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57112/0 (socket says 172.21.15.116:57112) 2023-11-28T17:10:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.454+0000 7f0bc24ba640 1 -- 172.21.15.116:0/796818244 learned_addr learned my addr 172.21.15.116:0/796818244 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:37.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.454+0000 7f0bc24ba640 1 -- 172.21.15.116:0/796818244 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0bbc13cee0 msgr2=0x7f0bbc156090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:37.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.454+0000 7f0bc24ba640 1 --2- 172.21.15.116:0/796818244 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0bbc13cee0 0x7f0bbc156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:37.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.454+0000 7f0bc24ba640 1 -- 172.21.15.116:0/796818244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bbc140200 msgr2=0x7f0bbc14f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:37.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.454+0000 7f0bc14b8640 1 --2- 172.21.15.116:0/796818244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bbc140200 0x7f0bbc14f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:37.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.454+0000 7f0bc24ba640 1 --2- 172.21.15.116:0/796818244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bbc140200 0x7f0bbc14f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:37.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.454+0000 7f0bc24ba640 1 -- 172.21.15.116:0/796818244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb8009a70 con 0x7f0bbc15a490 2023-11-28T17:10:37.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.455+0000 7f0bc24ba640 1 --2- 172.21.15.116:0/796818244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bbc15a490 0x7f0bbc0b0cb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4001eb0 tx=0x7f0bb400c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:37.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.455+0000 7f0bb27fc640 1 -- 172.21.15.116:0/796818244 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb40074e0 con 0x7f0bbc15a490 2023-11-28T17:10:37.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.455+0000 7f0bb27fc640 1 -- 172.21.15.116:0/796818244 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb4007640 con 0x7f0bbc15a490 2023-11-28T17:10:37.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.455+0000 7f0bb27fc640 1 -- 172.21.15.116:0/796818244 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4004a00 con 0x7f0bbc15a490 2023-11-28T17:10:37.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.455+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/796818244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bbc139720 con 0x7f0bbc15a490 2023-11-28T17:10:37.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.455+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/796818244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bbc157470 con 0x7f0bbc15a490 2023-11-28T17:10:37.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.456+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/796818244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bbc0693d0 con 0x7f0bbc15a490 2023-11-28T17:10:37.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.457+0000 7f0bb27fc640 1 -- 172.21.15.116:0/796818244 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0bb4004b60 con 0x7f0bbc15a490 2023-11-28T17:10:37.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.458+0000 7f0bb27fc640 1 --2- 172.21.15.116:0/796818244 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ba006b2a0 0x7f0ba006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:37.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.458+0000 7f0bb27fc640 1 -- 172.21.15.116:0/796818244 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0bb4089a60 con 0x7f0bbc15a490 2023-11-28T17:10:37.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.458+0000 7f0bc1cb9640 1 --2- 172.21.15.116:0/796818244 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ba006b2a0 0x7f0ba006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:37.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.459+0000 7f0bc1cb9640 1 --2- 172.21.15.116:0/796818244 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ba006b2a0 0x7f0ba006d760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8036000 tx=0x7f0bb802e000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:37.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.462+0000 7f0bb27fc640 1 -- 172.21.15.116:0/796818244 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0bb40548d0 con 0x7f0bbc15a490 2023-11-28T17:10:37.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.666+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/796818244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0bbc06d170 con 0x7f0bbc15a490 2023-11-28T17:10:37.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.666+0000 7f0bb27fc640 1 -- 172.21.15.116:0/796818244 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0bb4057ef0 con 0x7f0bbc15a490 2023-11-28T17:10:37.668 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-28T17:10:37.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.669+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/796818244 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ba006b2a0 msgr2=0x7f0ba006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:37.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.669+0000 7f0bc3f1d640 1 --2- 172.21.15.116:0/796818244 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ba006b2a0 0x7f0ba006d760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8036000 tx=0x7f0bb802e000 comp rx=0 tx=0).stop 2023-11-28T17:10:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.669+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/796818244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bbc15a490 msgr2=0x7f0bbc0b0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.669+0000 7f0bc3f1d640 1 --2- 172.21.15.116:0/796818244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bbc15a490 0x7f0bbc0b0cb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4001eb0 tx=0x7f0bb400c870 comp rx=0 tx=0).stop 2023-11-28T17:10:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.669+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/796818244 shutdown_connections 2023-11-28T17:10:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.669+0000 7f0bc3f1d640 1 --2- 172.21.15.116:0/796818244 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ba006b2a0 0x7f0ba006d760 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.669+0000 7f0bc3f1d640 1 --2- 172.21.15.116:0/796818244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bbc15a490 0x7f0bbc0b0cb0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:37.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.669+0000 7f0bc3f1d640 1 --2- 172.21.15.116:0/796818244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0bbc140200 0x7f0bbc14f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:37.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.669+0000 7f0bc3f1d640 1 --2- 172.21.15.116:0/796818244 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0bbc13cee0 0x7f0bbc156090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:37.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.669+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/796818244 >> 172.21.15.116:0/796818244 conn(0x7f0bbc06a4c0 msgr2=0x7f0bbc15d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:37.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.669+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/796818244 shutdown_connections 2023-11-28T17:10:37.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:37.669+0000 7f0bc3f1d640 1 -- 172.21.15.116:0/796818244 wait complete. 2023-11-28T17:10:37.688 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2023-11-28T17:10:37.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:37.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:37.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[102559]: fuse finished with error 0 and tester_r 0 2023-11-28T17:10:37.780 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:10:37.845 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:10:37.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:37.845 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T17:10:37.899 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:10:37.899 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:10:37.959 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:10:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.179+0000 7ffbbf9e3640 1 Processor -- start 2023-11-28T17:10:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.179+0000 7ffbbf9e3640 1 -- start start 2023-11-28T17:10:38.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.181+0000 7ffbbf9e3640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ffbb8166150 0x7ffbb8168540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:38.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.181+0000 7ffbbf9e3640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb80ae710 con 0x7ffbb80ac100 2023-11-28T17:10:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.181+0000 7ffbbf9e3640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb80c8bf0 con 0x7ffbb815d470 2023-11-28T17:10:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.181+0000 7ffbbf9e3640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbb80afe00 con 0x7ffbb8166150 2023-11-28T17:10:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.181+0000 7ffbbcf7e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ffbb80ac100 0x7ffbb80cb0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36486/0 (socket says 172.21.15.116:36486) 2023-11-28T17:10:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.181+0000 7ffbbcf7e640 1 -- 172.21.15.116:0/4101937976 learned_addr learned my addr 172.21.15.116:0/4101937976 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbbdf80640 1 --2- 172.21.15.116:0/4101937976 >> v2:172.21.15.156:3300/0 conn(0x7ffbb8166150 0x7ffbb8168540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 -- 172.21.15.116:0/4101937976 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259918874 0 0) 0x7ffbb80ae710 con 0x7ffbb80ac100 2023-11-28T17:10:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 -- 172.21.15.116:0/4101937976 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba4003520 con 0x7ffbb80ac100 2023-11-28T17:10:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 -- 172.21.15.116:0/4101937976 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2127592119 0 0) 0x7ffba4003520 con 0x7ffbb80ac100 2023-11-28T17:10:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 -- 172.21.15.116:0/4101937976 >> v2:172.21.15.156:3300/0 conn(0x7ffbb8166150 msgr2=0x7ffbb8168540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 --2- 172.21.15.116:0/4101937976 >> v2:172.21.15.156:3300/0 conn(0x7ffbb8166150 0x7ffbb8168540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 -- 172.21.15.116:0/4101937976 >> v1:172.21.15.156:6789/0 conn(0x7ffbb815d470 legacy=0x7ffbb80ca9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 -- 172.21.15.116:0/4101937976 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb80ac570 con 0x7ffbb80ac100 2023-11-28T17:10:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 -- 172.21.15.116:0/4101937976 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7ffba8001670 con 0x7ffbb80ac100 2023-11-28T17:10:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 -- 172.21.15.116:0/4101937976 >> v1:172.21.15.116:6789/0 conn(0x7ffbb80ac100 legacy=0x7ffbb80cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 --2- 172.21.15.116:0/4101937976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003970 0x7ffba4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 --2- 172.21.15.116:0/4101937976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffba40047c0 0x7ffba4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 --2- 172.21.15.116:0/4101937976 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffba4005180 0x7ffba400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 -- 172.21.15.116:0/4101937976 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb80afe00 con 0x7ffba4003970 2023-11-28T17:10:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 -- 172.21.15.116:0/4101937976 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba400c020 con 0x7ffba40047c0 2023-11-28T17:10:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.182+0000 7ffbaffff640 1 -- 172.21.15.116:0/4101937976 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba400c1e0 con 0x7ffba4005180 2023-11-28T17:10:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.183+0000 7ffbbd77f640 1 --2- 172.21.15.116:0/4101937976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003970 0x7ffba4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.183+0000 7ffbbd77f640 1 -- 172.21.15.116:0/4101937976 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffba4005180 msgr2=0x7ffba400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.183+0000 7ffbbd77f640 1 --2- 172.21.15.116:0/4101937976 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffba4005180 0x7ffba400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.183+0000 7ffbbd77f640 1 -- 172.21.15.116:0/4101937976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffba40047c0 msgr2=0x7ffba4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:38.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.183+0000 7ffbbd77f640 1 --2- 172.21.15.116:0/4101937976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffba40047c0 0x7ffba4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.183+0000 7ffbbd77f640 1 -- 172.21.15.116:0/4101937976 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba4003520 con 0x7ffba4003970 2023-11-28T17:10:38.187 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.183+0000 7ffbbdf80640 1 -- 172.21.15.116:0/4101937976 reap_dead start 2023-11-28T17:10:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.184+0000 7ffbbd77f640 1 --2- 172.21.15.116:0/4101937976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003970 0x7ffba4003d90 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40015b0 tx=0x7ffba40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=651224a0fffea975 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.184+0000 7ffbaffff640 1 -- 172.21.15.116:0/4101937976 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb40052d0 con 0x7ffba4003970 2023-11-28T17:10:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.184+0000 7ffbaffff640 1 -- 172.21.15.116:0/4101937976 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb402e050 con 0x7ffba4003970 2023-11-28T17:10:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.184+0000 7ffbaffff640 1 -- 172.21.15.116:0/4101937976 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4003830 con 0x7ffba4003970 2023-11-28T17:10:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.185+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/4101937976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003970 msgr2=0x7ffba4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.185+0000 7ffbbf9e3640 1 --2- 172.21.15.116:0/4101937976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003970 0x7ffba4003d90 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40015b0 tx=0x7ffba40057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.185+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/4101937976 shutdown_connections 2023-11-28T17:10:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.185+0000 7ffbbf9e3640 1 --2- 172.21.15.116:0/4101937976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003970 0x7ffba4003d90 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.185+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/4101937976 >> 172.21.15.116:0/4101937976 conn(0x7ffbb8071be0 msgr2=0x7ffbb8071ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.185+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/4101937976 shutdown_connections 2023-11-28T17:10:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.185+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/4101937976 wait complete. 2023-11-28T17:10:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.187+0000 7ffbbf9e3640 1 Processor -- start 2023-11-28T17:10:38.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.187+0000 7ffbbf9e3640 1 -- start start 2023-11-28T17:10:38.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.188+0000 7ffbbf9e3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003970 0x7ffbb815ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:38.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.188+0000 7ffbbf9e3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffbb8154460 0x7ffbb81559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:38.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.188+0000 7ffbbf9e3640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffbb81538c0 0x7ffbb8153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:38.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.188+0000 7ffbbf9e3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb8154280 con 0x7ffba4003970 2023-11-28T17:10:38.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.188+0000 7ffbbf9e3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb815f830 con 0x7ffbb8154460 2023-11-28T17:10:38.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.188+0000 7ffbbf9e3640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb815fb10 con 0x7ffbb81538c0 2023-11-28T17:10:38.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.188+0000 7ffbbd77f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003970 0x7ffbb815ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:38.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.188+0000 7ffbbd77f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003970 0x7ffbb815ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42686/0 (socket says 172.21.15.116:42686) 2023-11-28T17:10:38.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.188+0000 7ffbbd77f640 1 -- 172.21.15.116:0/3260044102 learned_addr learned my addr 172.21.15.116:0/3260044102 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:38.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.188+0000 7ffbbd77f640 1 -- 172.21.15.116:0/3260044102 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffbb81538c0 msgr2=0x7ffbb8153d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:38.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.188+0000 7ffbbd77f640 1 --2- 172.21.15.116:0/3260044102 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffbb81538c0 0x7ffbb8153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.188+0000 7ffbbd77f640 1 -- 172.21.15.116:0/3260044102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffbb8154460 msgr2=0x7ffbb81559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:38.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.188+0000 7ffbbd77f640 1 --2- 172.21.15.116:0/3260044102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffbb8154460 0x7ffbb81559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.188+0000 7ffbbd77f640 1 -- 172.21.15.116:0/3260044102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb815fdf0 con 0x7ffba4003970 2023-11-28T17:10:38.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.189+0000 7ffbbd77f640 1 --2- 172.21.15.116:0/3260044102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003970 0x7ffbb815ef00 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ffbb402ba60 tx=0x7ffbb4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:38.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.189+0000 7ffbadffb640 1 -- 172.21.15.116:0/3260044102 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4007a30 con 0x7ffba4003970 2023-11-28T17:10:38.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.189+0000 7ffbadffb640 1 -- 172.21.15.116:0/3260044102 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb4007640 con 0x7ffba4003970 2023-11-28T17:10:38.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.189+0000 7ffbadffb640 1 -- 172.21.15.116:0/3260044102 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4006be0 con 0x7ffba4003970 2023-11-28T17:10:38.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.189+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/3260044102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb81600d0 con 0x7ffba4003970 2023-11-28T17:10:38.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.189+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/3260044102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb815f440 con 0x7ffba4003970 2023-11-28T17:10:38.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.190+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/3260044102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb80693d0 con 0x7ffba4003970 2023-11-28T17:10:38.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.191+0000 7ffbadffb640 1 -- 172.21.15.116:0/3260044102 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ffbb4006d40 con 0x7ffba4003970 2023-11-28T17:10:38.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.192+0000 7ffbadffb640 1 --2- 172.21.15.116:0/3260044102 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ffb9c06b2a0 0x7ffb9c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:38.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.192+0000 7ffbadffb640 1 -- 172.21.15.116:0/3260044102 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffbb40ab8a0 con 0x7ffba4003970 2023-11-28T17:10:38.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.193+0000 7ffbbcf7e640 1 --2- 172.21.15.116:0/3260044102 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ffb9c06b2a0 0x7ffb9c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:38.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.193+0000 7ffbbcf7e640 1 --2- 172.21.15.116:0/3260044102 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ffb9c06b2a0 0x7ffb9c06d760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ffba80011e0 tx=0x7ffba80014e0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:38.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.196+0000 7ffbadffb640 1 -- 172.21.15.116:0/3260044102 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffbb40766f0 con 0x7ffba4003970 2023-11-28T17:10:38.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.387+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/3260044102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbb80c8bf0 con 0x7ffba4003970 2023-11-28T17:10:38.391 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.390+0000 7ffbadffb640 1 -- 172.21.15.116:0/3260044102 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16521 (secure 0 0 0) 0x7ffbb80c8bf0 con 0x7ffba4003970 2023-11-28T17:10:38.392 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:38.392 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":34,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:10:21.316243+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:10:17.338444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-28T17:10:18.766338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:10:38.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.393+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/3260044102 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ffb9c06b2a0 msgr2=0x7ffb9c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:38.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.393+0000 7ffbbf9e3640 1 --2- 172.21.15.116:0/3260044102 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ffb9c06b2a0 0x7ffb9c06d760 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ffba80011e0 tx=0x7ffba80014e0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.393+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/3260044102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003970 msgr2=0x7ffbb815ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:38.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.393+0000 7ffbbf9e3640 1 --2- 172.21.15.116:0/3260044102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003970 0x7ffbb815ef00 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ffbb402ba60 tx=0x7ffbb4001120 comp rx=0 tx=0).stop 2023-11-28T17:10:38.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.393+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/3260044102 shutdown_connections 2023-11-28T17:10:38.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.393+0000 7ffbbf9e3640 1 --2- 172.21.15.116:0/3260044102 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ffb9c06b2a0 0x7ffb9c06d760 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.393+0000 7ffbbf9e3640 1 --2- 172.21.15.116:0/3260044102 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffbb81538c0 0x7ffbb8153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.393+0000 7ffbbf9e3640 1 --2- 172.21.15.116:0/3260044102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffbb8154460 0x7ffbb81559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.393+0000 7ffbbf9e3640 1 --2- 172.21.15.116:0/3260044102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003970 0x7ffbb815ef00 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.393+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/3260044102 >> 172.21.15.116:0/3260044102 conn(0x7ffbb8071be0 msgr2=0x7ffbb80c9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:38.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.393+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/3260044102 shutdown_connections 2023-11-28T17:10:38.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.393+0000 7ffbbf9e3640 1 -- 172.21.15.116:0/3260044102 wait complete. 2023-11-28T17:10:38.412 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:10:17.338444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:10:18.766338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:10:38.412 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:10:38.621 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.620+0000 7f2cef1de640 1 Processor -- start 2023-11-28T17:10:38.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.620+0000 7f2cef1de640 1 -- start start 2023-11-28T17:10:38.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.622+0000 7f2cef1de640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2ce8145ee0 0x7f2ce80ac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:38.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.622+0000 7f2cef1de640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ce8161520 0x7f2ce8071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:38.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.622+0000 7f2cef1de640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce80b5110 con 0x7f2ce8146540 2023-11-28T17:10:38.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.622+0000 7f2cef1de640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce8161dc0 con 0x7f2ce8145ee0 2023-11-28T17:10:38.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.622+0000 7f2cef1de640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80b2190 con 0x7f2ce8161520 2023-11-28T17:10:38.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.622+0000 7f2cecf7a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2ce8145ee0 0x7f2ce80ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:38.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.623+0000 7f2ce7fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2ce8146540 0x7f2ce8160e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36496/0 (socket says 172.21.15.116:36496) 2023-11-28T17:10:38.624 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.623+0000 7f2ce7fff640 1 -- 172.21.15.116:0/1037331140 learned_addr learned my addr 172.21.15.116:0/1037331140 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:38.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.623+0000 7f2cecf7a640 1 -- 172.21.15.116:0/1037331140 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ce8161520 msgr2=0x7f2ce8071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:38.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.623+0000 7f2cecf7a640 1 --2- 172.21.15.116:0/1037331140 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ce8161520 0x7f2ce8071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.623+0000 7f2cecf7a640 1 -- 172.21.15.116:0/1037331140 >> v1:172.21.15.116:6789/0 conn(0x7f2ce8146540 legacy=0x7f2ce8160e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:38.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.623+0000 7f2cecf7a640 1 -- 172.21.15.116:0/1037331140 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce8161ae0 con 0x7f2ce8145ee0 2023-11-28T17:10:38.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.623+0000 7f2ce77fe640 1 -- 172.21.15.116:0/1037331140 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356173706 0 0) 0x7f2ce80b5110 con 0x7f2ce8146540 2023-11-28T17:10:38.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.625+0000 7f2cecf7a640 1 --2- 172.21.15.116:0/1037331140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ce8145ee0 0x7f2ce80ac870 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc0097a0 tx=0x7f2cdc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0c0dbd2485595f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:38.626 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.625+0000 7f2ce77fe640 1 -- 172.21.15.116:0/1037331140 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc031830 con 0x7f2ce8145ee0 2023-11-28T17:10:38.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.625+0000 7f2ce77fe640 1 -- 172.21.15.116:0/1037331140 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cdc031990 con 0x7f2ce8145ee0 2023-11-28T17:10:38.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.625+0000 7f2ce77fe640 1 -- 172.21.15.116:0/1037331140 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc031d20 con 0x7f2ce8145ee0 2023-11-28T17:10:38.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.626+0000 7f2cef1de640 1 -- 172.21.15.116:0/1037331140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ce8145ee0 msgr2=0x7f2ce80ac870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:38.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.626+0000 7f2cef1de640 1 --2- 172.21.15.116:0/1037331140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ce8145ee0 0x7f2ce80ac870 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc0097a0 tx=0x7f2cdc02b8d0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.626+0000 7f2cef1de640 1 -- 172.21.15.116:0/1037331140 shutdown_connections 2023-11-28T17:10:38.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.626+0000 7f2cef1de640 1 --2- 172.21.15.116:0/1037331140 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ce8161520 0x7f2ce8071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.626+0000 7f2cef1de640 1 --2- 172.21.15.116:0/1037331140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ce8145ee0 0x7f2ce80ac870 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.626+0000 7f2cef1de640 1 -- 172.21.15.116:0/1037331140 >> 172.21.15.116:0/1037331140 conn(0x7f2ce80b0cf0 msgr2=0x7f2ce80b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:38.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.626+0000 7f2cef1de640 1 -- 172.21.15.116:0/1037331140 shutdown_connections 2023-11-28T17:10:38.628 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.626+0000 7f2cef1de640 1 -- 172.21.15.116:0/1037331140 wait complete. 2023-11-28T17:10:38.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.628+0000 7f2cef1de640 1 Processor -- start 2023-11-28T17:10:38.629 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.628+0000 7f2cef1de640 1 -- start start 2023-11-28T17:10:38.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.629+0000 7f2cef1de640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ce8145ee0 0x7f2ce815cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.629+0000 7f2cef1de640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ce8146540 0x7f2ce80b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.629+0000 7f2cef1de640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ce8161520 0x7f2ce80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.629+0000 7f2cef1de640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce815f340 con 0x7f2ce8145ee0 2023-11-28T17:10:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.629+0000 7f2cef1de640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce815e9c0 con 0x7f2ce8146540 2023-11-28T17:10:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.629+0000 7f2cef1de640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce815ec80 con 0x7f2ce8161520 2023-11-28T17:10:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.629+0000 7f2cecf7a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ce8145ee0 0x7f2ce815cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.629+0000 7f2cecf7a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ce8145ee0 0x7f2ce815cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42700/0 (socket says 172.21.15.116:42700) 2023-11-28T17:10:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.629+0000 7f2cecf7a640 1 -- 172.21.15.116:0/1625764023 learned_addr learned my addr 172.21.15.116:0/1625764023 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.630+0000 7f2cecf7a640 1 -- 172.21.15.116:0/1625764023 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ce8161520 msgr2=0x7f2ce80b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.630+0000 7f2cecf7a640 1 --2- 172.21.15.116:0/1625764023 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ce8161520 0x7f2ce80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.630+0000 7f2cecf7a640 1 -- 172.21.15.116:0/1625764023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ce8146540 msgr2=0x7f2ce80b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:38.632 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.630+0000 7f2ce7fff640 1 --2- 172.21.15.116:0/1625764023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ce8146540 0x7f2ce80b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:38.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.630+0000 7f2cecf7a640 1 --2- 172.21.15.116:0/1625764023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ce8146540 0x7f2ce80b3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.630+0000 7f2cecf7a640 1 -- 172.21.15.116:0/1625764023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc009400 con 0x7f2ce8145ee0 2023-11-28T17:10:38.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.630+0000 7f2cecf7a640 1 --2- 172.21.15.116:0/1625764023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ce8145ee0 0x7f2ce815cac0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc0097a0 tx=0x7f2cdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:38.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.630+0000 7f2ce57fa640 1 -- 172.21.15.116:0/1625764023 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc03aa60 con 0x7f2ce8145ee0 2023-11-28T17:10:38.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.630+0000 7f2ce57fa640 1 -- 172.21.15.116:0/1625764023 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cdc03abc0 con 0x7f2ce8145ee0 2023-11-28T17:10:38.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.630+0000 7f2ce57fa640 1 -- 172.21.15.116:0/1625764023 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc0443f0 con 0x7f2ce8145ee0 2023-11-28T17:10:38.633 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.630+0000 7f2cef1de640 1 -- 172.21.15.116:0/1625764023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce815ef60 con 0x7f2ce8145ee0 2023-11-28T17:10:38.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.631+0000 7f2cef1de640 1 -- 172.21.15.116:0/1625764023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce815f760 con 0x7f2ce8145ee0 2023-11-28T17:10:38.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.632+0000 7f2cef1de640 1 -- 172.21.15.116:0/1625764023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce815f480 con 0x7f2ce8145ee0 2023-11-28T17:10:38.634 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.632+0000 7f2ce57fa640 1 -- 172.21.15.116:0/1625764023 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2cdc04d450 con 0x7f2ce8145ee0 2023-11-28T17:10:38.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.633+0000 7f2ce57fa640 1 --2- 172.21.15.116:0/1625764023 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ccc06b2a0 0x7f2ccc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:38.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.634+0000 7f2ce57fa640 1 -- 172.21.15.116:0/1625764023 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2cdc0abe30 con 0x7f2ce8145ee0 2023-11-28T17:10:38.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.634+0000 7f2ce7fff640 1 --2- 172.21.15.116:0/1625764023 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ccc06b2a0 0x7f2ccc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:38.635 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.634+0000 7f2ce7fff640 1 --2- 172.21.15.116:0/1625764023 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ccc06b2a0 0x7f2ccc06d760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4007fd0 tx=0x7f2cd4009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:38.639 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.638+0000 7f2ce57fa640 1 -- 172.21.15.116:0/1625764023 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2cdc076c90 con 0x7f2ce8145ee0 2023-11-28T17:10:38.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.826+0000 7f2cef1de640 1 -- 172.21.15.116:0/1625764023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2ce80acd30 con 0x7f2ce8145ee0 2023-11-28T17:10:38.829 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.828+0000 7f2ce57fa640 1 -- 172.21.15.116:0/1625764023 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2cdc07a2b0 con 0x7f2ce8145ee0 2023-11-28T17:10:38.829 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:10:38.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.830+0000 7f2cef1de640 1 -- 172.21.15.116:0/1625764023 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ccc06b2a0 msgr2=0x7f2ccc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:38.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.830+0000 7f2cef1de640 1 --2- 172.21.15.116:0/1625764023 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ccc06b2a0 0x7f2ccc06d760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4007fd0 tx=0x7f2cd4009040 comp rx=0 tx=0).stop 2023-11-28T17:10:38.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.830+0000 7f2cef1de640 1 -- 172.21.15.116:0/1625764023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ce8145ee0 msgr2=0x7f2ce815cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:38.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.830+0000 7f2cef1de640 1 --2- 172.21.15.116:0/1625764023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ce8145ee0 0x7f2ce815cac0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc0097a0 tx=0x7f2cdc0046c0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.831+0000 7f2cef1de640 1 -- 172.21.15.116:0/1625764023 shutdown_connections 2023-11-28T17:10:38.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.831+0000 7f2cef1de640 1 --2- 172.21.15.116:0/1625764023 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ccc06b2a0 0x7f2ccc06d760 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.831+0000 7f2cef1de640 1 --2- 172.21.15.116:0/1625764023 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ce8161520 0x7f2ce80b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.831+0000 7f2cef1de640 1 --2- 172.21.15.116:0/1625764023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ce8146540 0x7f2ce80b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.831+0000 7f2cef1de640 1 --2- 172.21.15.116:0/1625764023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ce8145ee0 0x7f2ce815cac0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:38.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.831+0000 7f2cef1de640 1 -- 172.21.15.116:0/1625764023 >> 172.21.15.116:0/1625764023 conn(0x7f2ce80b0cf0 msgr2=0x7f2ce80733f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:38.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.831+0000 7f2cef1de640 1 -- 172.21.15.116:0/1625764023 shutdown_connections 2023-11-28T17:10:38.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:38.831+0000 7f2cef1de640 1 -- 172.21.15.116:0/1625764023 wait complete. 2023-11-28T17:10:38.851 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:10:39.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.068+0000 7f3105cc2640 1 Processor -- start 2023-11-28T17:10:39.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.068+0000 7f3105cc2640 1 -- start start 2023-11-28T17:10:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.071+0000 7f3105cc2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f310015b0c0 0x7f310015ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.071+0000 7f3105cc2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f31000ce940 0x7f31000cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.071+0000 7f3105cc2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3100071b40 0x7f3100071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.071+0000 7f3105cc2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3100072510 con 0x7f3100071b40 2023-11-28T17:10:39.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.071+0000 7f3105cc2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31000cd320 con 0x7f31000ce940 2023-11-28T17:10:39.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.071+0000 7f3105cc2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31000ced80 con 0x7f310015b0c0 2023-11-28T17:10:39.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.071+0000 7f30fffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3100071b40 0x7f3100071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.071+0000 7f30fffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3100071b40 0x7f3100071fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42712/0 (socket says 172.21.15.116:42712) 2023-11-28T17:10:39.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.071+0000 7f30fffff640 1 -- 172.21.15.116:0/1677484369 learned_addr learned my addr 172.21.15.116:0/1677484369 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:39.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.071+0000 7f30ff7fe640 1 --2- 172.21.15.116:0/1677484369 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f310015b0c0 0x7f310015ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.071+0000 7f30fffff640 1 -- 172.21.15.116:0/1677484369 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f310015b0c0 msgr2=0x7f310015ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:39.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.071+0000 7f30fffff640 1 --2- 172.21.15.116:0/1677484369 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f310015b0c0 0x7f310015ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.071+0000 7f30f7fff640 1 --2- 172.21.15.116:0/1677484369 >> v2:172.21.15.156:3300/0 conn(0x7f31000ce940 0x7f31000cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.071+0000 7f30fffff640 1 -- 172.21.15.116:0/1677484369 >> v2:172.21.15.156:3300/0 conn(0x7f31000ce940 msgr2=0x7f31000cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:39.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.072+0000 7f30fffff640 1 --2- 172.21.15.116:0/1677484369 >> v2:172.21.15.156:3300/0 conn(0x7f31000ce940 0x7f31000cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.072+0000 7f30fffff640 1 -- 172.21.15.116:0/1677484369 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31000cbee0 con 0x7f3100071b40 2023-11-28T17:10:39.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.072+0000 7f30ff7fe640 1 --2- 172.21.15.116:0/1677484369 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f310015b0c0 0x7f310015ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:39.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.073+0000 7f30fffff640 1 --2- 172.21.15.116:0/1677484369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3100071b40 0x7f3100071fd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f31000b1db0 tx=0x7f30f802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac599803aae3f7b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:39.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.073+0000 7f30feffd640 1 -- 172.21.15.116:0/1677484369 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f802c9f0 con 0x7f3100071b40 2023-11-28T17:10:39.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.073+0000 7f30feffd640 1 -- 172.21.15.116:0/1677484369 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f802cb50 con 0x7f3100071b40 2023-11-28T17:10:39.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.073+0000 7f30feffd640 1 -- 172.21.15.116:0/1677484369 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f80343f0 con 0x7f3100071b40 2023-11-28T17:10:39.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.074+0000 7f3105cc2640 1 -- 172.21.15.116:0/1677484369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3100071b40 msgr2=0x7f3100071fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.074+0000 7f3105cc2640 1 --2- 172.21.15.116:0/1677484369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3100071b40 0x7f3100071fd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f31000b1db0 tx=0x7f30f802be50 comp rx=0 tx=0).stop 2023-11-28T17:10:39.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.074+0000 7f3105cc2640 1 -- 172.21.15.116:0/1677484369 shutdown_connections 2023-11-28T17:10:39.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.074+0000 7f3105cc2640 1 --2- 172.21.15.116:0/1677484369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3100071b40 0x7f3100071fd0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.074+0000 7f3105cc2640 1 --2- 172.21.15.116:0/1677484369 >> v2:172.21.15.156:3300/0 conn(0x7f31000ce940 0x7f31000cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.074+0000 7f3105cc2640 1 --2- 172.21.15.116:0/1677484369 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f310015b0c0 0x7f310015ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.074+0000 7f3105cc2640 1 -- 172.21.15.116:0/1677484369 >> 172.21.15.116:0/1677484369 conn(0x7f31000b0cf0 msgr2=0x7f31000b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:39.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.074+0000 7f3105cc2640 1 -- 172.21.15.116:0/1677484369 shutdown_connections 2023-11-28T17:10:39.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.074+0000 7f3105cc2640 1 -- 172.21.15.116:0/1677484369 wait complete. 2023-11-28T17:10:39.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.076+0000 7f3105cc2640 1 Processor -- start 2023-11-28T17:10:39.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.076+0000 7f3105cc2640 1 -- start start 2023-11-28T17:10:39.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.077+0000 7f3105cc2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3100071b40 0x7f31001e40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.083 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.077+0000 7f3105cc2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31000ce940 0x7f310015a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.077+0000 7f3105cc2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310015b0c0 0x7f31001452b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.077+0000 7f3105cc2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310015d040 con 0x7f310015b0c0 2023-11-28T17:10:39.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.077+0000 7f3105cc2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310015a960 con 0x7f31000ce940 2023-11-28T17:10:39.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.077+0000 7f3105cc2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3100145b50 con 0x7f3100071b40 2023-11-28T17:10:39.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.077+0000 7f30fffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310015b0c0 0x7f31001452b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.077+0000 7f30fffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310015b0c0 0x7f31001452b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42718/0 (socket says 172.21.15.116:42718) 2023-11-28T17:10:39.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.077+0000 7f30fffff640 1 -- 172.21.15.116:0/1249288038 learned_addr learned my addr 172.21.15.116:0/1249288038 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:39.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.078+0000 7f30ff7fe640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3100071b40 0x7f31001e40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.078+0000 7f30f7fff640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31000ce940 0x7f310015a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.078+0000 7f30fffff640 1 -- 172.21.15.116:0/1249288038 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3100071b40 msgr2=0x7f31001e40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.078+0000 7f30fffff640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3100071b40 0x7f31001e40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.078+0000 7f30fffff640 1 -- 172.21.15.116:0/1249288038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31000ce940 msgr2=0x7f310015a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.078+0000 7f30fffff640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31000ce940 0x7f310015a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.078+0000 7f30fffff640 1 -- 172.21.15.116:0/1249288038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f8009b30 con 0x7f310015b0c0 2023-11-28T17:10:39.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.078+0000 7f30ff7fe640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3100071b40 0x7f31001e40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:10:39.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.078+0000 7f30fffff640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310015b0c0 0x7f31001452b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f30f8009f40 tx=0x7f30f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:39.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.078+0000 7f30fcff9640 1 -- 172.21.15.116:0/1249288038 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8009510 con 0x7f310015b0c0 2023-11-28T17:10:39.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.079+0000 7f30fcff9640 1 -- 172.21.15.116:0/1249288038 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f803e450 con 0x7f310015b0c0 2023-11-28T17:10:39.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.079+0000 7f30fcff9640 1 -- 172.21.15.116:0/1249288038 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f803e7e0 con 0x7f310015b0c0 2023-11-28T17:10:39.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.079+0000 7f3105cc2640 1 -- 172.21.15.116:0/1249288038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3100145e30 con 0x7f310015b0c0 2023-11-28T17:10:39.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.079+0000 7f3105cc2640 1 -- 172.21.15.116:0/1249288038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310015cc60 con 0x7f310015b0c0 2023-11-28T17:10:39.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.080+0000 7f3105cc2640 1 -- 172.21.15.116:0/1249288038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310015d180 con 0x7f310015b0c0 2023-11-28T17:10:39.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.081+0000 7f30fcff9640 1 -- 172.21.15.116:0/1249288038 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f30f803e940 con 0x7f310015b0c0 2023-11-28T17:10:39.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.082+0000 7f30fcff9640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f30e006b2a0 0x7f30e006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.082+0000 7f30fcff9640 1 -- 172.21.15.116:0/1249288038 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30f80acab0 con 0x7f310015b0c0 2023-11-28T17:10:39.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.082+0000 7f30ff7fe640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f30e006b2a0 0x7f30e006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.083+0000 7f30ff7fe640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f30e006b2a0 0x7f30e006d760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3100146110 tx=0x7f30f0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:39.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.086+0000 7f30fcff9640 1 -- 172.21.15.116:0/1249288038 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f310015d180 con 0x7f310015b0c0 2023-11-28T17:10:39.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.274+0000 7f3105cc2640 1 -- 172.21.15.116:0/1249288038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f310015ce20 con 0x7f310015b0c0 2023-11-28T17:10:39.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.275+0000 7f30fcff9640 1 -- 172.21.15.116:0/1249288038 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7f30f8077a40 con 0x7f310015b0c0 2023-11-28T17:10:39.276 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:10:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.277+0000 7f3105cc2640 1 -- 172.21.15.116:0/1249288038 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f30e006b2a0 msgr2=0x7f30e006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.277+0000 7f3105cc2640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f30e006b2a0 0x7f30e006d760 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3100146110 tx=0x7f30f0008040 comp rx=0 tx=0).stop 2023-11-28T17:10:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.277+0000 7f3105cc2640 1 -- 172.21.15.116:0/1249288038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310015b0c0 msgr2=0x7f31001452b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.277+0000 7f3105cc2640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310015b0c0 0x7f31001452b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f30f8009f40 tx=0x7f30f8004a40 comp rx=0 tx=0).stop 2023-11-28T17:10:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.277+0000 7f3105cc2640 1 -- 172.21.15.116:0/1249288038 shutdown_connections 2023-11-28T17:10:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.277+0000 7f3105cc2640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f30e006b2a0 0x7f30e006d760 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.277+0000 7f3105cc2640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f310015b0c0 0x7f31001452b0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.277+0000 7f3105cc2640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31000ce940 0x7f310015a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.277+0000 7f3105cc2640 1 --2- 172.21.15.116:0/1249288038 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3100071b40 0x7f31001e40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.277+0000 7f3105cc2640 1 -- 172.21.15.116:0/1249288038 >> 172.21.15.116:0/1249288038 conn(0x7f31000b0cf0 msgr2=0x7f3100156ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:39.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.278+0000 7f3105cc2640 1 -- 172.21.15.116:0/1249288038 shutdown_connections 2023-11-28T17:10:39.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.278+0000 7f3105cc2640 1 -- 172.21.15.116:0/1249288038 wait complete. 2023-11-28T17:10:39.298 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:10:39.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.516+0000 7f6eb3a00640 1 Processor -- start 2023-11-28T17:10:39.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.516+0000 7f6eb3a00640 1 -- start start 2023-11-28T17:10:39.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.518+0000 7f6eb3a00640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6eac14ac30 0x7f6eac0cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.518+0000 7f6eb3a00640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eac070420 con 0x7f6eac0b4600 2023-11-28T17:10:39.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.518+0000 7f6eb3a00640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eac0b2fc0 con 0x7f6eac14a740 2023-11-28T17:10:39.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.518+0000 7f6eb3a00640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eac070aa0 con 0x7f6eac14ac30 2023-11-28T17:10:39.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.518+0000 7f6eb1f9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6eac14ac30 0x7f6eac0cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.518+0000 7f6eb1f9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6eac14ac30 0x7f6eac0cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42724/0 (socket says 172.21.15.116:42724) 2023-11-28T17:10:39.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.518+0000 7f6eb1f9d640 1 -- 172.21.15.116:0/1398129405 learned_addr learned my addr 172.21.15.116:0/1398129405 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:39.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.519+0000 7f6eb1f9d640 1 -- 172.21.15.116:0/1398129405 >> v1:172.21.15.156:6789/0 conn(0x7f6eac14a740 legacy=0x7f6eac0b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.519+0000 7f6eb1f9d640 1 -- 172.21.15.116:0/1398129405 >> v1:172.21.15.116:6789/0 conn(0x7f6eac0b4600 legacy=0x7f6eac070ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.519+0000 7f6eb1f9d640 1 -- 172.21.15.116:0/1398129405 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eac0706c0 con 0x7f6eac14ac30 2023-11-28T17:10:39.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.519+0000 7f6ea3fff640 1 -- 172.21.15.116:0/1398129405 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373297762 0 0) 0x7f6eac070420 con 0x7f6eac0b4600 2023-11-28T17:10:39.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.520+0000 7f6eb1f9d640 1 --2- 172.21.15.116:0/1398129405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6eac14ac30 0x7f6eac0cc7a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009e50 tx=0x7f6ea402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2661c85fa1cd74c6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:39.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.520+0000 7f6ea3fff640 1 -- 172.21.15.116:0/1398129405 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea402cb20 con 0x7f6eac14ac30 2023-11-28T17:10:39.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.520+0000 7f6ea3fff640 1 -- 172.21.15.116:0/1398129405 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea402cc80 con 0x7f6eac14ac30 2023-11-28T17:10:39.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.520+0000 7f6ea3fff640 1 -- 172.21.15.116:0/1398129405 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea40343f0 con 0x7f6eac14ac30 2023-11-28T17:10:39.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.521+0000 7f6eb3a00640 1 -- 172.21.15.116:0/1398129405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6eac14ac30 msgr2=0x7f6eac0cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.521+0000 7f6eb3a00640 1 --2- 172.21.15.116:0/1398129405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6eac14ac30 0x7f6eac0cc7a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009e50 tx=0x7f6ea402bf80 comp rx=0 tx=0).stop 2023-11-28T17:10:39.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.521+0000 7f6eb3a00640 1 -- 172.21.15.116:0/1398129405 shutdown_connections 2023-11-28T17:10:39.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.521+0000 7f6eb3a00640 1 --2- 172.21.15.116:0/1398129405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6eac14ac30 0x7f6eac0cc7a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.521+0000 7f6eb3a00640 1 -- 172.21.15.116:0/1398129405 >> 172.21.15.116:0/1398129405 conn(0x7f6eac0b0cf0 msgr2=0x7f6eac0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:39.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.521+0000 7f6eb3a00640 1 -- 172.21.15.116:0/1398129405 shutdown_connections 2023-11-28T17:10:39.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.521+0000 7f6eb3a00640 1 -- 172.21.15.116:0/1398129405 wait complete. 2023-11-28T17:10:39.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.523+0000 7f6eb3a00640 1 Processor -- start 2023-11-28T17:10:39.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.523+0000 7f6eb3a00640 1 -- start start 2023-11-28T17:10:39.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.524+0000 7f6eb3a00640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6eac0b4600 0x7f6eac160df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.524+0000 7f6eb3a00640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6eac14a740 0x7f6eac0a9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.524+0000 7f6eb3a00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6eac14ac30 0x7f6eac0aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.524+0000 7f6eb3a00640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac0a9f50 con 0x7f6eac14ac30 2023-11-28T17:10:39.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.524+0000 7f6eb3a00640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac0aa230 con 0x7f6eac14a740 2023-11-28T17:10:39.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.524+0000 7f6eb3a00640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac0aa510 con 0x7f6eac0b4600 2023-11-28T17:10:39.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.524+0000 7f6eb1f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6eac14ac30 0x7f6eac0aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.524+0000 7f6eb1f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6eac14ac30 0x7f6eac0aec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42726/0 (socket says 172.21.15.116:42726) 2023-11-28T17:10:39.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.524+0000 7f6eb1f9d640 1 -- 172.21.15.116:0/2103077732 learned_addr learned my addr 172.21.15.116:0/2103077732 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:39.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.524+0000 7f6eb1f9d640 1 -- 172.21.15.116:0/2103077732 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6eac0b4600 msgr2=0x7f6eac160df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.524+0000 7f6eb1f9d640 1 --2- 172.21.15.116:0/2103077732 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6eac0b4600 0x7f6eac160df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.524+0000 7f6eb1f9d640 1 -- 172.21.15.116:0/2103077732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6eac14a740 msgr2=0x7f6eac0a9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.524+0000 7f6eb1f9d640 1 --2- 172.21.15.116:0/2103077732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6eac14a740 0x7f6eac0a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.524+0000 7f6eb1f9d640 1 -- 172.21.15.116:0/2103077732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea4009b30 con 0x7f6eac14ac30 2023-11-28T17:10:39.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.525+0000 7f6eb1f9d640 1 --2- 172.21.15.116:0/2103077732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6eac14ac30 0x7f6eac0aec10 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009e50 tx=0x7f6ea4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:39.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.525+0000 7f6ea1ffb640 1 -- 172.21.15.116:0/2103077732 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4009510 con 0x7f6eac14ac30 2023-11-28T17:10:39.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.525+0000 7f6ea1ffb640 1 -- 172.21.15.116:0/2103077732 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea403e450 con 0x7f6eac14ac30 2023-11-28T17:10:39.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.525+0000 7f6eb3a00640 1 -- 172.21.15.116:0/2103077732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eac0aa7f0 con 0x7f6eac14ac30 2023-11-28T17:10:39.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.525+0000 7f6ea1ffb640 1 -- 172.21.15.116:0/2103077732 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea403e600 con 0x7f6eac14ac30 2023-11-28T17:10:39.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.525+0000 7f6eb3a00640 1 -- 172.21.15.116:0/2103077732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eac161330 con 0x7f6eac14ac30 2023-11-28T17:10:39.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.526+0000 7f6eb3a00640 1 -- 172.21.15.116:0/2103077732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eac0cbe70 con 0x7f6eac14ac30 2023-11-28T17:10:39.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.527+0000 7f6ea1ffb640 1 -- 172.21.15.116:0/2103077732 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6ea403e760 con 0x7f6eac14ac30 2023-11-28T17:10:39.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.529+0000 7f6ea1ffb640 1 --2- 172.21.15.116:0/2103077732 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6e9006b2a0 0x7f6e9006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.529+0000 7f6ea1ffb640 1 -- 172.21.15.116:0/2103077732 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ea40ac500 con 0x7f6eac14ac30 2023-11-28T17:10:39.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.529+0000 7f6eb179c640 1 --2- 172.21.15.116:0/2103077732 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6e9006b2a0 0x7f6e9006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.530+0000 7f6eb179c640 1 --2- 172.21.15.116:0/2103077732 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6e9006b2a0 0x7f6e9006d760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8006fd0 tx=0x7f6ea8006bc0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:39.534 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.533+0000 7f6ea1ffb640 1 -- 172.21.15.116:0/2103077732 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ea4077390 con 0x7f6eac14ac30 2023-11-28T17:10:39.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.723+0000 7f6eb3a00640 1 -- 172.21.15.116:0/2103077732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6eac0aad10 con 0x7f6eac14ac30 2023-11-28T17:10:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.724+0000 7f6ea1ffb640 1 -- 172.21.15.116:0/2103077732 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6ea407a9b0 con 0x7f6eac14ac30 2023-11-28T17:10:39.725 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:10:39.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.727+0000 7f6eb3a00640 1 -- 172.21.15.116:0/2103077732 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6e9006b2a0 msgr2=0x7f6e9006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.727+0000 7f6eb3a00640 1 --2- 172.21.15.116:0/2103077732 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6e9006b2a0 0x7f6e9006d760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8006fd0 tx=0x7f6ea8006bc0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.727+0000 7f6eb3a00640 1 -- 172.21.15.116:0/2103077732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6eac14ac30 msgr2=0x7f6eac0aec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.727+0000 7f6eb3a00640 1 --2- 172.21.15.116:0/2103077732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6eac14ac30 0x7f6eac0aec10 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4009e50 tx=0x7f6ea4004a40 comp rx=0 tx=0).stop 2023-11-28T17:10:39.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.727+0000 7f6eb3a00640 1 -- 172.21.15.116:0/2103077732 shutdown_connections 2023-11-28T17:10:39.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.727+0000 7f6eb3a00640 1 --2- 172.21.15.116:0/2103077732 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6e9006b2a0 0x7f6e9006d760 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.727+0000 7f6eb3a00640 1 --2- 172.21.15.116:0/2103077732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6eac14ac30 0x7f6eac0aec10 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.727+0000 7f6eb3a00640 1 --2- 172.21.15.116:0/2103077732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6eac14a740 0x7f6eac0a9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.727+0000 7f6eb3a00640 1 --2- 172.21.15.116:0/2103077732 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6eac0b4600 0x7f6eac160df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.727+0000 7f6eb3a00640 1 -- 172.21.15.116:0/2103077732 >> 172.21.15.116:0/2103077732 conn(0x7f6eac0b0cf0 msgr2=0x7f6eac0ce4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:39.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.727+0000 7f6eb3a00640 1 -- 172.21.15.116:0/2103077732 shutdown_connections 2023-11-28T17:10:39.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.727+0000 7f6eb3a00640 1 -- 172.21.15.116:0/2103077732 wait complete. 2023-11-28T17:10:39.746 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:10:39.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.960+0000 7f640fcee640 1 Processor -- start 2023-11-28T17:10:39.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.961+0000 7f640fcee640 1 -- start start 2023-11-28T17:10:39.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.962+0000 7f640fcee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f640815f7b0 0x7f6408071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.962+0000 7f640fcee640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f64080721b0 0x7f640806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.962+0000 7f640fcee640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f640815fc10 con 0x7f6408144430 2023-11-28T17:10:39.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.962+0000 7f640fcee640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6408144810 con 0x7f64080721b0 2023-11-28T17:10:39.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.962+0000 7f640fcee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6408070bd0 con 0x7f640815f7b0 2023-11-28T17:10:39.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.962+0000 7f640da8a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6408144430 0x7f64080703f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36516/0 (socket says 172.21.15.116:36516) 2023-11-28T17:10:39.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.962+0000 7f640da8a640 1 -- 172.21.15.116:0/426341114 learned_addr learned my addr 172.21.15.116:0/426341114 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:39.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.963+0000 7f640d289640 1 --2- 172.21.15.116:0/426341114 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f640815f7b0 0x7f6408071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.963+0000 7f640ca88640 1 -- 172.21.15.116:0/426341114 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463615884 0 0) 0x7f640815fc10 con 0x7f6408144430 2023-11-28T17:10:39.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.963+0000 7f640e28b640 1 --2- 172.21.15.116:0/426341114 >> v2:172.21.15.156:3300/0 conn(0x7f64080721b0 0x7f640806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.963+0000 7f640ca88640 1 -- 172.21.15.116:0/426341114 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f4003520 con 0x7f6408144430 2023-11-28T17:10:39.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.963+0000 7f640ca88640 1 -- 172.21.15.116:0/426341114 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2563981174 0 0) 0x7f63f4003520 con 0x7f6408144430 2023-11-28T17:10:39.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.963+0000 7f640ca88640 1 -- 172.21.15.116:0/426341114 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f640815f7b0 msgr2=0x7f6408071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:39.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.963+0000 7f640ca88640 1 --2- 172.21.15.116:0/426341114 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f640815f7b0 0x7f6408071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.963+0000 7f640ca88640 1 -- 172.21.15.116:0/426341114 >> v2:172.21.15.156:3300/0 conn(0x7f64080721b0 msgr2=0x7f640806c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.963+0000 7f640ca88640 1 --2- 172.21.15.116:0/426341114 >> v2:172.21.15.156:3300/0 conn(0x7f64080721b0 0x7f640806c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.963+0000 7f640ca88640 1 -- 172.21.15.116:0/426341114 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6408070eb0 con 0x7f6408144430 2023-11-28T17:10:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.963+0000 7f640ca88640 1 -- 172.21.15.116:0/426341114 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f6404001670 con 0x7f6408144430 2023-11-28T17:10:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.963+0000 7f640ca88640 1 -- 172.21.15.116:0/426341114 >> v1:172.21.15.116:6789/0 conn(0x7f6408144430 legacy=0x7f64080703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.963+0000 7f640ca88640 1 --2- 172.21.15.116:0/426341114 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f640815f7b0 0x7f63f4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.964+0000 7f640ca88640 1 --2- 172.21.15.116:0/426341114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63f4003fa0 0x7f63f40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.964+0000 7f640ca88640 1 --2- 172.21.15.116:0/426341114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63f4004d00 0x7f63f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.964+0000 7f640ca88640 1 -- 172.21.15.116:0/426341114 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6408144810 con 0x7f63f4003fa0 2023-11-28T17:10:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.964+0000 7f640ca88640 1 -- 172.21.15.116:0/426341114 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6408070bd0 con 0x7f63f4004d00 2023-11-28T17:10:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.964+0000 7f640ca88640 1 -- 172.21.15.116:0/426341114 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f400ba80 con 0x7f640815f7b0 2023-11-28T17:10:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.964+0000 7f640d289640 1 --2- 172.21.15.116:0/426341114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63f4003fa0 0x7f63f40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.964+0000 7f640da8a640 1 --2- 172.21.15.116:0/426341114 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f640815f7b0 0x7f63f4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.964+0000 7f640e28b640 1 --2- 172.21.15.116:0/426341114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63f4004d00 0x7f63f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.964+0000 7f640d289640 1 -- 172.21.15.116:0/426341114 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f640815f7b0 msgr2=0x7f63f4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.964+0000 7f640d289640 1 --2- 172.21.15.116:0/426341114 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f640815f7b0 0x7f63f4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.964+0000 7f640d289640 1 -- 172.21.15.116:0/426341114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63f4004d00 msgr2=0x7f63f4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.964+0000 7f640d289640 1 --2- 172.21.15.116:0/426341114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63f4004d00 0x7f63f4005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.964+0000 7f640d289640 1 -- 172.21.15.116:0/426341114 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f4003520 con 0x7f63f4003fa0 2023-11-28T17:10:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.965+0000 7f640d289640 1 --2- 172.21.15.116:0/426341114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63f4003fa0 0x7f63f40047c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f63f800ba30 tx=0x7f63f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a12ad40600e30e3e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.966+0000 7f640ca88640 1 -- 172.21.15.116:0/426341114 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8032640 con 0x7f63f4003fa0 2023-11-28T17:10:39.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.966+0000 7f640ca88640 1 -- 172.21.15.116:0/426341114 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f80327a0 con 0x7f63f4003fa0 2023-11-28T17:10:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.966+0000 7f640ca88640 1 -- 172.21.15.116:0/426341114 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8032b30 con 0x7f63f4003fa0 2023-11-28T17:10:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.966+0000 7f640fcee640 1 -- 172.21.15.116:0/426341114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63f4003fa0 msgr2=0x7f63f40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.966+0000 7f640fcee640 1 --2- 172.21.15.116:0/426341114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63f4003fa0 0x7f63f40047c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f63f800ba30 tx=0x7f63f40057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.966+0000 7f640e28b640 1 -- 172.21.15.116:0/426341114 reap_dead start 2023-11-28T17:10:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.966+0000 7f640fcee640 1 -- 172.21.15.116:0/426341114 shutdown_connections 2023-11-28T17:10:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.966+0000 7f640fcee640 1 -- 172.21.15.116:0/426341114 >> 172.21.15.116:0/426341114 conn(0x7f640806a4c0 msgr2=0x7f640806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.966+0000 7f640fcee640 1 -- 172.21.15.116:0/426341114 shutdown_connections 2023-11-28T17:10:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.966+0000 7f640fcee640 1 -- 172.21.15.116:0/426341114 wait complete. 2023-11-28T17:10:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.968+0000 7f640fcee640 1 Processor -- start 2023-11-28T17:10:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.968+0000 7f640fcee640 1 -- start start 2023-11-28T17:10:39.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.969+0000 7f640fcee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f63f4003fa0 0x7f640815d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.969+0000 7f640fcee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64080cb740 0x7f64081403d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.969+0000 7f640fcee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f640815c410 0x7f6408140910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.969+0000 7f640fcee640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640815d520 con 0x7f640815c410 2023-11-28T17:10:39.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.969+0000 7f640fcee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640815c910 con 0x7f64080cb740 2023-11-28T17:10:39.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.969+0000 7f640fcee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640815cbf0 con 0x7f63f4003fa0 2023-11-28T17:10:39.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.970+0000 7f640e28b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f640815c410 0x7f6408140910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.970+0000 7f640e28b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f640815c410 0x7f6408140910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42748/0 (socket says 172.21.15.116:42748) 2023-11-28T17:10:39.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.970+0000 7f640e28b640 1 -- 172.21.15.116:0/543111869 learned_addr learned my addr 172.21.15.116:0/543111869 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:39.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.970+0000 7f640da8a640 1 --2- 172.21.15.116:0/543111869 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f63f4003fa0 0x7f640815d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.970+0000 7f640d289640 1 --2- 172.21.15.116:0/543111869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64080cb740 0x7f64081403d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.970+0000 7f640e28b640 1 -- 172.21.15.116:0/543111869 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f63f4003fa0 msgr2=0x7f640815d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.970+0000 7f640e28b640 1 --2- 172.21.15.116:0/543111869 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f63f4003fa0 0x7f640815d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.970+0000 7f640e28b640 1 -- 172.21.15.116:0/543111869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64080cb740 msgr2=0x7f64081403d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:39.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.970+0000 7f640e28b640 1 --2- 172.21.15.116:0/543111869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64080cb740 0x7f64081403d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:39.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.970+0000 7f640e28b640 1 -- 172.21.15.116:0/543111869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640815ced0 con 0x7f640815c410 2023-11-28T17:10:39.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.970+0000 7f640e28b640 1 --2- 172.21.15.116:0/543111869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f640815c410 0x7f6408140910 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f64080cc1d0 tx=0x7f6400009d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:39.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.971+0000 7f63fe7fc640 1 -- 172.21.15.116:0/543111869 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6400007cf0 con 0x7f640815c410 2023-11-28T17:10:39.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.971+0000 7f63fe7fc640 1 -- 172.21.15.116:0/543111869 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6400007e70 con 0x7f640815c410 2023-11-28T17:10:39.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.971+0000 7f63fe7fc640 1 -- 172.21.15.116:0/543111869 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6400014620 con 0x7f640815c410 2023-11-28T17:10:39.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.971+0000 7f640fcee640 1 -- 172.21.15.116:0/543111869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640815d1b0 con 0x7f640815c410 2023-11-28T17:10:39.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.971+0000 7f640fcee640 1 -- 172.21.15.116:0/543111869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640815e0d0 con 0x7f640815c410 2023-11-28T17:10:39.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.972+0000 7f640fcee640 1 -- 172.21.15.116:0/543111869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640815d340 con 0x7f640815c410 2023-11-28T17:10:39.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.973+0000 7f63fe7fc640 1 -- 172.21.15.116:0/543111869 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f640001d3b0 con 0x7f640815c410 2023-11-28T17:10:39.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.974+0000 7f63fe7fc640 1 --2- 172.21.15.116:0/543111869 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f63ec06b2a0 0x7f63ec06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:39.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.974+0000 7f63fe7fc640 1 -- 172.21.15.116:0/543111869 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6400089a40 con 0x7f640815c410 2023-11-28T17:10:39.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.974+0000 7f640da8a640 1 --2- 172.21.15.116:0/543111869 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f63ec06b2a0 0x7f63ec06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:39.976 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.975+0000 7f640da8a640 1 --2- 172.21.15.116:0/543111869 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f63ec06b2a0 0x7f63ec06d760 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6404001050 tx=0x7f6404009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:39.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:39.978+0000 7f63fe7fc640 1 -- 172.21.15.116:0/543111869 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6400054810 con 0x7f640815c410 2023-11-28T17:10:40.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.175+0000 7f640fcee640 1 -- 172.21.15.116:0/543111869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f64080693d0 con 0x7f640815c410 2023-11-28T17:10:40.177 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.176+0000 7f63fe7fc640 1 -- 172.21.15.116:0/543111869 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6400057e30 con 0x7f640815c410 2023-11-28T17:10:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-28T17:10:40.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.179+0000 7f640fcee640 1 -- 172.21.15.116:0/543111869 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f63ec06b2a0 msgr2=0x7f63ec06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:40.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.179+0000 7f640fcee640 1 --2- 172.21.15.116:0/543111869 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f63ec06b2a0 0x7f63ec06d760 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6404001050 tx=0x7f6404009040 comp rx=0 tx=0).stop 2023-11-28T17:10:40.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.179+0000 7f640fcee640 1 -- 172.21.15.116:0/543111869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f640815c410 msgr2=0x7f6408140910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:40.180 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.179+0000 7f640fcee640 1 --2- 172.21.15.116:0/543111869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f640815c410 0x7f6408140910 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f64080cc1d0 tx=0x7f6400009d40 comp rx=0 tx=0).stop 2023-11-28T17:10:40.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.179+0000 7f640fcee640 1 -- 172.21.15.116:0/543111869 shutdown_connections 2023-11-28T17:10:40.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.179+0000 7f640fcee640 1 --2- 172.21.15.116:0/543111869 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f63ec06b2a0 0x7f63ec06d760 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.179+0000 7f640fcee640 1 --2- 172.21.15.116:0/543111869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f640815c410 0x7f6408140910 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.179+0000 7f640fcee640 1 --2- 172.21.15.116:0/543111869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64080cb740 0x7f64081403d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.179+0000 7f640fcee640 1 --2- 172.21.15.116:0/543111869 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f63f4003fa0 0x7f640815d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.179+0000 7f640fcee640 1 -- 172.21.15.116:0/543111869 >> 172.21.15.116:0/543111869 conn(0x7f640806a4c0 msgr2=0x7f64080b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:40.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.179+0000 7f640fcee640 1 -- 172.21.15.116:0/543111869 shutdown_connections 2023-11-28T17:10:40.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.179+0000 7f640fcee640 1 -- 172.21.15.116:0/543111869 wait complete. 2023-11-28T17:10:40.199 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:10:40.199 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:10:40.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:10:40.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.479+0000 7f34c0eee640 1 Processor -- start 2023-11-28T17:10:40.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.479+0000 7f34c0eee640 1 -- start start 2023-11-28T17:10:40.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.481+0000 7f34c0eee640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f34bc0b4b50 0x7f34bc06caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:40.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.481+0000 7f34c0eee640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34bc06bae0 con 0x7f34bc145df0 2023-11-28T17:10:40.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.481+0000 7f34c0eee640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34bc06c660 con 0x7f34bc0b3000 2023-11-28T17:10:40.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.481+0000 7f34c0eee640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34bc06bd30 con 0x7f34bc0b4b50 2023-11-28T17:10:40.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.481+0000 7f34ba59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f34bc145df0 0x7f34bc1465c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36530/0 (socket says 172.21.15.116:36530) 2023-11-28T17:10:40.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.481+0000 7f34ba59c640 1 -- 172.21.15.116:0/4248964370 learned_addr learned my addr 172.21.15.116:0/4248964370 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:40.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 -- 172.21.15.116:0/4248964370 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128280534 0 0) 0x7f34bc06bae0 con 0x7f34bc145df0 2023-11-28T17:10:40.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 -- 172.21.15.116:0/4248964370 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34a8003520 con 0x7f34bc145df0 2023-11-28T17:10:40.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b9d9b640 1 --2- 172.21.15.116:0/4248964370 >> v2:172.21.15.156:3300/0 conn(0x7f34bc0b4b50 0x7f34bc06caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:40.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 -- 172.21.15.116:0/4248964370 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1735743281 0 0) 0x7f34a8003520 con 0x7f34bc145df0 2023-11-28T17:10:40.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 -- 172.21.15.116:0/4248964370 >> v2:172.21.15.156:3300/0 conn(0x7f34bc0b4b50 msgr2=0x7f34bc06caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:40.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 --2- 172.21.15.116:0/4248964370 >> v2:172.21.15.156:3300/0 conn(0x7f34bc0b4b50 0x7f34bc06caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 -- 172.21.15.116:0/4248964370 >> v1:172.21.15.156:6789/0 conn(0x7f34bc0b3000 legacy=0x7f34bc15d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:40.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 -- 172.21.15.116:0/4248964370 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc06c010 con 0x7f34bc145df0 2023-11-28T17:10:40.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 -- 172.21.15.116:0/4248964370 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f34a4001670 con 0x7f34bc145df0 2023-11-28T17:10:40.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 -- 172.21.15.116:0/4248964370 >> v1:172.21.15.116:6789/0 conn(0x7f34bc145df0 legacy=0x7f34bc1465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:40.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 --2- 172.21.15.116:0/4248964370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34a8003970 0x7f34a8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:40.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 --2- 172.21.15.116:0/4248964370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a80047c0 0x7f34a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:40.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 --2- 172.21.15.116:0/4248964370 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34a8005180 0x7f34a800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:40.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 -- 172.21.15.116:0/4248964370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc06bd30 con 0x7f34a80047c0 2023-11-28T17:10:40.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 -- 172.21.15.116:0/4248964370 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a800c0d0 con 0x7f34a8003970 2023-11-28T17:10:40.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.482+0000 7f34b959a640 1 -- 172.21.15.116:0/4248964370 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a800c290 con 0x7f34a8005180 2023-11-28T17:10:40.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.483+0000 7f34b9d9b640 1 --2- 172.21.15.116:0/4248964370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a80047c0 0x7f34a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:40.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.483+0000 7f34ba59c640 1 --2- 172.21.15.116:0/4248964370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34a8003970 0x7f34a8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:40.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.483+0000 7f34bad9d640 1 --2- 172.21.15.116:0/4248964370 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34a8005180 0x7f34a800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:40.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.483+0000 7f34b9d9b640 1 -- 172.21.15.116:0/4248964370 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34a8005180 msgr2=0x7f34a800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:40.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.483+0000 7f34b9d9b640 1 --2- 172.21.15.116:0/4248964370 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34a8005180 0x7f34a800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.483+0000 7f34b9d9b640 1 -- 172.21.15.116:0/4248964370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34a8003970 msgr2=0x7f34a8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:40.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.483+0000 7f34b9d9b640 1 --2- 172.21.15.116:0/4248964370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34a8003970 0x7f34a8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.483+0000 7f34b9d9b640 1 -- 172.21.15.116:0/4248964370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a8003520 con 0x7f34a80047c0 2023-11-28T17:10:40.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.483+0000 7f34bad9d640 1 --2- 172.21.15.116:0/4248964370 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34a8005180 0x7f34a800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:40.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.483+0000 7f34ba59c640 1 --2- 172.21.15.116:0/4248964370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34a8003970 0x7f34a8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:10:40.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.483+0000 7f34bad9d640 1 -- 172.21.15.116:0/4248964370 reap_dead start 2023-11-28T17:10:40.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.484+0000 7f34b9d9b640 1 --2- 172.21.15.116:0/4248964370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a80047c0 0x7f34a8004c40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f34ac007360 tx=0x7f34a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8a49ef19d20d5b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:40.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.485+0000 7f34b959a640 1 -- 172.21.15.116:0/4248964370 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac02ccb0 con 0x7f34a80047c0 2023-11-28T17:10:40.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.485+0000 7f34b959a640 1 -- 172.21.15.116:0/4248964370 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ac02ce10 con 0x7f34a80047c0 2023-11-28T17:10:40.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.485+0000 7f34b959a640 1 -- 172.21.15.116:0/4248964370 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac034660 con 0x7f34a80047c0 2023-11-28T17:10:40.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.485+0000 7f34c0eee640 1 -- 172.21.15.116:0/4248964370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a80047c0 msgr2=0x7f34a8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:40.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.485+0000 7f34c0eee640 1 --2- 172.21.15.116:0/4248964370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a80047c0 0x7f34a8004c40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f34ac007360 tx=0x7f34a80057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.485+0000 7f34c0eee640 1 -- 172.21.15.116:0/4248964370 shutdown_connections 2023-11-28T17:10:40.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.485+0000 7f34c0eee640 1 --2- 172.21.15.116:0/4248964370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a80047c0 0x7f34a8004c40 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.485+0000 7f34c0eee640 1 -- 172.21.15.116:0/4248964370 >> 172.21.15.116:0/4248964370 conn(0x7f34bc06a8b0 msgr2=0x7f34bc06acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:40.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.485+0000 7f34c0eee640 1 -- 172.21.15.116:0/4248964370 shutdown_connections 2023-11-28T17:10:40.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.485+0000 7f34c0eee640 1 -- 172.21.15.116:0/4248964370 wait complete. 2023-11-28T17:10:40.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.487+0000 7f34c0eee640 1 Processor -- start 2023-11-28T17:10:40.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.487+0000 7f34c0eee640 1 -- start start 2023-11-28T17:10:40.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.488+0000 7f34c0eee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34a80047c0 0x7f34bc13cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:40.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.488+0000 7f34c0eee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34bc15c8f0 0x7f34bc143f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:40.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.488+0000 7f34c0eee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34bc144440 0x7f34bc13d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:40.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.488+0000 7f34c0eee640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc1448c0 con 0x7f34bc15c8f0 2023-11-28T17:10:40.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.488+0000 7f34c0eee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc1e6920 con 0x7f34a80047c0 2023-11-28T17:10:40.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.488+0000 7f34c0eee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc15c6d0 con 0x7f34bc144440 2023-11-28T17:10:40.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.489+0000 7f34b9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34bc15c8f0 0x7f34bc143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:40.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.489+0000 7f34b9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34bc15c8f0 0x7f34bc143f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42754/0 (socket says 172.21.15.116:42754) 2023-11-28T17:10:40.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.489+0000 7f34b9d9b640 1 -- 172.21.15.116:0/3649393383 learned_addr learned my addr 172.21.15.116:0/3649393383 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:40.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.489+0000 7f34ba59c640 1 --2- 172.21.15.116:0/3649393383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34a80047c0 0x7f34bc13cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:40.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.489+0000 7f34bad9d640 1 --2- 172.21.15.116:0/3649393383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34bc144440 0x7f34bc13d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:40.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.489+0000 7f34b9d9b640 1 -- 172.21.15.116:0/3649393383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34bc144440 msgr2=0x7f34bc13d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:40.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.489+0000 7f34b9d9b640 1 --2- 172.21.15.116:0/3649393383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34bc144440 0x7f34bc13d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.489+0000 7f34b9d9b640 1 -- 172.21.15.116:0/3649393383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34a80047c0 msgr2=0x7f34bc13cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:40.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.489+0000 7f34b9d9b640 1 --2- 172.21.15.116:0/3649393383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34a80047c0 0x7f34bc13cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.489+0000 7f34b9d9b640 1 -- 172.21.15.116:0/3649393383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc15bb50 con 0x7f34bc15c8f0 2023-11-28T17:10:40.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.489+0000 7f34b9d9b640 1 --2- 172.21.15.116:0/3649393383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34bc15c8f0 0x7f34bc143f00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f34ac007480 tx=0x7f34ac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:40.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.490+0000 7f34b2ffd640 1 -- 172.21.15.116:0/3649393383 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac044070 con 0x7f34bc15c8f0 2023-11-28T17:10:40.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.490+0000 7f34b2ffd640 1 -- 172.21.15.116:0/3649393383 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ac003990 con 0x7f34bc15c8f0 2023-11-28T17:10:40.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.490+0000 7f34c0eee640 1 -- 172.21.15.116:0/3649393383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34bc15be30 con 0x7f34bc15c8f0 2023-11-28T17:10:40.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.490+0000 7f34b2ffd640 1 -- 172.21.15.116:0/3649393383 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac003d40 con 0x7f34bc15c8f0 2023-11-28T17:10:40.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.490+0000 7f34c0eee640 1 -- 172.21.15.116:0/3649393383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34bc15c3f0 con 0x7f34bc15c8f0 2023-11-28T17:10:40.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.491+0000 7f34c0eee640 1 -- 172.21.15.116:0/3649393383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34bc144820 con 0x7f34bc15c8f0 2023-11-28T17:10:40.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.492+0000 7f34b2ffd640 1 -- 172.21.15.116:0/3649393383 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f34ac003ea0 con 0x7f34bc15c8f0 2023-11-28T17:10:40.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.493+0000 7f34b2ffd640 1 --2- 172.21.15.116:0/3649393383 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f349c06b2a0 0x7f349c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:40.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.493+0000 7f34b2ffd640 1 -- 172.21.15.116:0/3649393383 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34ac0abb70 con 0x7f34bc15c8f0 2023-11-28T17:10:40.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.493+0000 7f34ba59c640 1 --2- 172.21.15.116:0/3649393383 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f349c06b2a0 0x7f349c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:40.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.494+0000 7f34ba59c640 1 --2- 172.21.15.116:0/3649393383 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f349c06b2a0 0x7f349c06d760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f34a4001b40 tx=0x7f34a4001ab0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:40.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.498+0000 7f34b2ffd640 1 -- 172.21.15.116:0/3649393383 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34ac076940 con 0x7f34bc15c8f0 2023-11-28T17:10:40.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.687+0000 7f34c0eee640 1 -- 172.21.15.116:0/3649393383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34bc067460 con 0x7f34bc15c8f0 2023-11-28T17:10:40.691 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:40.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.689+0000 7f34b2ffd640 1 -- 172.21.15.116:0/3649393383 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16521 (secure 0 0 0) 0x7f34ac079f60 con 0x7f34bc15c8f0 2023-11-28T17:10:40.692 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":34,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:10:21.316243+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:10:17.338444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-28T17:10:18.766338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:10:40.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.692+0000 7f34c0eee640 1 -- 172.21.15.116:0/3649393383 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f349c06b2a0 msgr2=0x7f349c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:40.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.692+0000 7f34c0eee640 1 --2- 172.21.15.116:0/3649393383 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f349c06b2a0 0x7f349c06d760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f34a4001b40 tx=0x7f34a4001ab0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.692+0000 7f34c0eee640 1 -- 172.21.15.116:0/3649393383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34bc15c8f0 msgr2=0x7f34bc143f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:40.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.692+0000 7f34c0eee640 1 --2- 172.21.15.116:0/3649393383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34bc15c8f0 0x7f34bc143f00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f34ac007480 tx=0x7f34ac004ab0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.692+0000 7f34c0eee640 1 -- 172.21.15.116:0/3649393383 shutdown_connections 2023-11-28T17:10:40.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.692+0000 7f34c0eee640 1 --2- 172.21.15.116:0/3649393383 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f349c06b2a0 0x7f349c06d760 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.692+0000 7f34c0eee640 1 --2- 172.21.15.116:0/3649393383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34bc144440 0x7f34bc13d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.692+0000 7f34c0eee640 1 --2- 172.21.15.116:0/3649393383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34bc15c8f0 0x7f34bc143f00 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.692+0000 7f34c0eee640 1 --2- 172.21.15.116:0/3649393383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34a80047c0 0x7f34bc13cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.692+0000 7f34c0eee640 1 -- 172.21.15.116:0/3649393383 >> 172.21.15.116:0/3649393383 conn(0x7f34bc06a8b0 msgr2=0x7f34bc143480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:40.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.692+0000 7f34c0eee640 1 -- 172.21.15.116:0/3649393383 shutdown_connections 2023-11-28T17:10:40.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.692+0000 7f34c0eee640 1 -- 172.21.15.116:0/3649393383 wait complete. 2023-11-28T17:10:40.712 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:10:17.338444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:10:18.766338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:10:40.712 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:10:40.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.917+0000 7ff8f0db0640 1 Processor -- start 2023-11-28T17:10:40.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.917+0000 7ff8f0db0640 1 -- start start 2023-11-28T17:10:40.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.919+0000 7ff8f0db0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff8ec15b0c0 0x7ff8ec15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:40.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.919+0000 7ff8f0db0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff8ec0ce940 0x7ff8ec0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:40.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.919+0000 7ff8f0db0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec0cd320 con 0x7ff8ec071b40 2023-11-28T17:10:40.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.919+0000 7ff8f0db0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ec0ced80 con 0x7ff8ec0ce940 2023-11-28T17:10:40.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.919+0000 7ff8f0db0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ec0cbee0 con 0x7ff8ec15b0c0 2023-11-28T17:10:40.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.919+0000 7ff8e3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff8ec0ce940 0x7ff8ec0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:40.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.919+0000 7ff8e3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff8ec0ce940 0x7ff8ec0cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42770/0 (socket says 172.21.15.116:42770) 2023-11-28T17:10:40.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.919+0000 7ff8e3fff640 1 -- 172.21.15.116:0/3947065930 learned_addr learned my addr 172.21.15.116:0/3947065930 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:40.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.920+0000 7ff8ea59c640 1 --2- 172.21.15.116:0/3947065930 >> v2:172.21.15.156:3300/0 conn(0x7ff8ec15b0c0 0x7ff8ec15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:40.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.920+0000 7ff8e3fff640 1 -- 172.21.15.116:0/3947065930 >> v2:172.21.15.156:3300/0 conn(0x7ff8ec15b0c0 msgr2=0x7ff8ec15ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:40.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.920+0000 7ff8e3fff640 1 --2- 172.21.15.116:0/3947065930 >> v2:172.21.15.156:3300/0 conn(0x7ff8ec15b0c0 0x7ff8ec15ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.920+0000 7ff8e3fff640 1 -- 172.21.15.116:0/3947065930 >> v1:172.21.15.156:6789/0 conn(0x7ff8ec071b40 legacy=0x7ff8ec15b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:40.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.920+0000 7ff8e3fff640 1 -- 172.21.15.116:0/3947065930 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec0cc710 con 0x7ff8ec0ce940 2023-11-28T17:10:40.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.921+0000 7ff8e3fff640 1 --2- 172.21.15.116:0/3947065930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8ec0ce940 0x7ff8ec0cb9a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec06b470 tx=0x7ff8d402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2508bde12db1af53 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:40.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.921+0000 7ff8e9d9b640 1 -- 172.21.15.116:0/3947065930 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d40317e0 con 0x7ff8ec0ce940 2023-11-28T17:10:40.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.921+0000 7ff8e9d9b640 1 -- 172.21.15.116:0/3947065930 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d4031940 con 0x7ff8ec0ce940 2023-11-28T17:10:40.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.921+0000 7ff8e9d9b640 1 -- 172.21.15.116:0/3947065930 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4031cd0 con 0x7ff8ec0ce940 2023-11-28T17:10:40.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.922+0000 7ff8f0db0640 1 -- 172.21.15.116:0/3947065930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8ec0ce940 msgr2=0x7ff8ec0cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:40.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.922+0000 7ff8f0db0640 1 --2- 172.21.15.116:0/3947065930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8ec0ce940 0x7ff8ec0cb9a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec06b470 tx=0x7ff8d402b880 comp rx=0 tx=0).stop 2023-11-28T17:10:40.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.922+0000 7ff8f0db0640 1 -- 172.21.15.116:0/3947065930 shutdown_connections 2023-11-28T17:10:40.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.922+0000 7ff8f0db0640 1 --2- 172.21.15.116:0/3947065930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8ec0ce940 0x7ff8ec0cb9a0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.922+0000 7ff8f0db0640 1 --2- 172.21.15.116:0/3947065930 >> v2:172.21.15.156:3300/0 conn(0x7ff8ec15b0c0 0x7ff8ec15ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.922+0000 7ff8f0db0640 1 -- 172.21.15.116:0/3947065930 >> 172.21.15.116:0/3947065930 conn(0x7ff8ec0b0cf0 msgr2=0x7ff8ec0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:40.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.922+0000 7ff8f0db0640 1 -- 172.21.15.116:0/3947065930 shutdown_connections 2023-11-28T17:10:40.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.922+0000 7ff8f0db0640 1 -- 172.21.15.116:0/3947065930 wait complete. 2023-11-28T17:10:40.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.924+0000 7ff8f0db0640 1 Processor -- start 2023-11-28T17:10:40.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.924+0000 7ff8f0db0640 1 -- start start 2023-11-28T17:10:40.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.925+0000 7ff8f0db0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8ec071b40 0x7ff8ec0adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:40.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.925+0000 7ff8f0db0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8ec0ce940 0x7ff8ec0c4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:40.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.925+0000 7ff8f0db0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8ec15b0c0 0x7ff8ec159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:40.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.925+0000 7ff8f0db0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec0ae350 con 0x7ff8ec071b40 2023-11-28T17:10:40.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.925+0000 7ff8f0db0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec0ae130 con 0x7ff8ec0ce940 2023-11-28T17:10:40.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.925+0000 7ff8f0db0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec0c4660 con 0x7ff8ec15b0c0 2023-11-28T17:10:40.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.925+0000 7ff8ea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8ec071b40 0x7ff8ec0adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:40.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.925+0000 7ff8ea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8ec071b40 0x7ff8ec0adbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42778/0 (socket says 172.21.15.116:42778) 2023-11-28T17:10:40.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.925+0000 7ff8ea59c640 1 -- 172.21.15.116:0/1179730637 learned_addr learned my addr 172.21.15.116:0/1179730637 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:40.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.926+0000 7ff8e3fff640 1 --2- 172.21.15.116:0/1179730637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8ec0ce940 0x7ff8ec0c4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:40.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.926+0000 7ff8ea59c640 1 -- 172.21.15.116:0/1179730637 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8ec15b0c0 msgr2=0x7ff8ec159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:40.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.926+0000 7ff8ead9d640 1 --2- 172.21.15.116:0/1179730637 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8ec15b0c0 0x7ff8ec159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:40.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.926+0000 7ff8ea59c640 1 --2- 172.21.15.116:0/1179730637 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8ec15b0c0 0x7ff8ec159aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.926+0000 7ff8ea59c640 1 -- 172.21.15.116:0/1179730637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8ec0ce940 msgr2=0x7ff8ec0c4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:40.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.926+0000 7ff8ea59c640 1 --2- 172.21.15.116:0/1179730637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8ec0ce940 0x7ff8ec0c4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:40.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.926+0000 7ff8ea59c640 1 -- 172.21.15.116:0/1179730637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d40094d0 con 0x7ff8ec071b40 2023-11-28T17:10:40.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.926+0000 7ff8ea59c640 1 --2- 172.21.15.116:0/1179730637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8ec071b40 0x7ff8ec0adbf0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00d830 tx=0x7ff8dc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:40.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.926+0000 7ff8e2ffd640 1 -- 172.21.15.116:0/1179730637 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc00b660 con 0x7ff8ec071b40 2023-11-28T17:10:40.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.926+0000 7ff8e2ffd640 1 -- 172.21.15.116:0/1179730637 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8dc00b7c0 con 0x7ff8ec071b40 2023-11-28T17:10:40.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.926+0000 7ff8e2ffd640 1 -- 172.21.15.116:0/1179730637 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc00bb50 con 0x7ff8ec071b40 2023-11-28T17:10:40.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.926+0000 7ff8f0db0640 1 -- 172.21.15.116:0/1179730637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ec0b32c0 con 0x7ff8ec071b40 2023-11-28T17:10:40.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.926+0000 7ff8f0db0640 1 -- 172.21.15.116:0/1179730637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ec0b2c70 con 0x7ff8ec071b40 2023-11-28T17:10:40.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.928+0000 7ff8f0db0640 1 -- 172.21.15.116:0/1179730637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ec069360 con 0x7ff8ec071b40 2023-11-28T17:10:40.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.928+0000 7ff8e2ffd640 1 -- 172.21.15.116:0/1179730637 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff8dc01e460 con 0x7ff8ec071b40 2023-11-28T17:10:40.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.929+0000 7ff8e2ffd640 1 --2- 172.21.15.116:0/1179730637 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff8cc06b2a0 0x7ff8cc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:40.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.929+0000 7ff8e2ffd640 1 -- 172.21.15.116:0/1179730637 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8dc08a300 con 0x7ff8ec071b40 2023-11-28T17:10:40.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.930+0000 7ff8e3fff640 1 --2- 172.21.15.116:0/1179730637 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff8cc06b2a0 0x7ff8cc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:40.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.930+0000 7ff8e3fff640 1 --2- 172.21.15.116:0/1179730637 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff8cc06b2a0 0x7ff8cc06d760 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4036000 tx=0x7ff8d403d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:40.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:40.934+0000 7ff8e2ffd640 1 -- 172.21.15.116:0/1179730637 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8dc055160 con 0x7ff8ec071b40 2023-11-28T17:10:41.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.127+0000 7ff8f0db0640 1 -- 172.21.15.116:0/1179730637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff8ec0b1990 con 0x7ff8ec071b40 2023-11-28T17:10:41.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.128+0000 7ff8e2ffd640 1 -- 172.21.15.116:0/1179730637 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff8dc058780 con 0x7ff8ec071b40 2023-11-28T17:10:41.130 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:10:41.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.130+0000 7ff8f0db0640 1 -- 172.21.15.116:0/1179730637 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff8cc06b2a0 msgr2=0x7ff8cc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:41.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.130+0000 7ff8f0db0640 1 --2- 172.21.15.116:0/1179730637 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff8cc06b2a0 0x7ff8cc06d760 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4036000 tx=0x7ff8d403d040 comp rx=0 tx=0).stop 2023-11-28T17:10:41.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.130+0000 7ff8f0db0640 1 -- 172.21.15.116:0/1179730637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8ec071b40 msgr2=0x7ff8ec0adbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:41.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.130+0000 7ff8f0db0640 1 --2- 172.21.15.116:0/1179730637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8ec071b40 0x7ff8ec0adbf0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00d830 tx=0x7ff8dc00dcf0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.130+0000 7ff8f0db0640 1 -- 172.21.15.116:0/1179730637 shutdown_connections 2023-11-28T17:10:41.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.130+0000 7ff8f0db0640 1 --2- 172.21.15.116:0/1179730637 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff8cc06b2a0 0x7ff8cc06d760 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.130+0000 7ff8f0db0640 1 --2- 172.21.15.116:0/1179730637 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8ec15b0c0 0x7ff8ec159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.130+0000 7ff8f0db0640 1 --2- 172.21.15.116:0/1179730637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8ec0ce940 0x7ff8ec0c4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.130+0000 7ff8f0db0640 1 --2- 172.21.15.116:0/1179730637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8ec071b40 0x7ff8ec0adbf0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.130+0000 7ff8f0db0640 1 -- 172.21.15.116:0/1179730637 >> 172.21.15.116:0/1179730637 conn(0x7ff8ec0b0cf0 msgr2=0x7ff8ec070860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:41.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.131+0000 7ff8f0db0640 1 -- 172.21.15.116:0/1179730637 shutdown_connections 2023-11-28T17:10:41.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.131+0000 7ff8f0db0640 1 -- 172.21.15.116:0/1179730637 wait complete. 2023-11-28T17:10:41.149 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:10:41.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.338+0000 7ff1b1c6b640 1 Processor -- start 2023-11-28T17:10:41.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.338+0000 7ff1b1c6b640 1 -- start start 2023-11-28T17:10:41.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.339+0000 7ff1b1c6b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1ac15d860 0x7ff1ac0cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:41.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.339+0000 7ff1b1c6b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff1ac0d0010 0x7ff1ac160270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:41.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.339+0000 7ff1b1c6b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ac15ddc0 con 0x7ff1ac15d480 2023-11-28T17:10:41.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.339+0000 7ff1b1c6b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ac0d04a0 con 0x7ff1ac0d0010 2023-11-28T17:10:41.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.339+0000 7ff1b1c6b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac15e0a0 con 0x7ff1ac15d860 2023-11-28T17:10:41.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.340+0000 7ff1ab7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff1ac15d480 0x7ff1ac0d0dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36546/0 (socket says 172.21.15.116:36546) 2023-11-28T17:10:41.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.340+0000 7ff1ab7fe640 1 -- 172.21.15.116:0/678813836 learned_addr learned my addr 172.21.15.116:0/678813836 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:41.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.340+0000 7ff1abfff640 1 --2- 172.21.15.116:0/678813836 >> v2:172.21.15.116:3300/0 conn(0x7ff1ac0d0010 0x7ff1ac160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:41.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.340+0000 7ff1a3fff640 1 --2- 172.21.15.116:0/678813836 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1ac15d860 0x7ff1ac0cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:41.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.340+0000 7ff1abfff640 1 -- 172.21.15.116:0/678813836 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1ac15d860 msgr2=0x7ff1ac0cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:41.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.340+0000 7ff1abfff640 1 --2- 172.21.15.116:0/678813836 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1ac15d860 0x7ff1ac0cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.340+0000 7ff1abfff640 1 -- 172.21.15.116:0/678813836 >> v1:172.21.15.116:6789/0 conn(0x7ff1ac15d480 legacy=0x7ff1ac0d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:41.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.340+0000 7ff1abfff640 1 -- 172.21.15.116:0/678813836 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac15e380 con 0x7ff1ac0d0010 2023-11-28T17:10:41.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.340+0000 7ff1aaffd640 1 -- 172.21.15.116:0/678813836 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138303868 0 0) 0x7ff1ac15ddc0 con 0x7ff1ac15d480 2023-11-28T17:10:41.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.341+0000 7ff1abfff640 1 --2- 172.21.15.116:0/678813836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac0d0010 0x7ff1ac160270 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009d90 tx=0x7ff1a402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1216de59ee64a839 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:41.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.342+0000 7ff1aaffd640 1 -- 172.21.15.116:0/678813836 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a402cc50 con 0x7ff1ac0d0010 2023-11-28T17:10:41.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.342+0000 7ff1aaffd640 1 -- 172.21.15.116:0/678813836 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a402cdb0 con 0x7ff1ac0d0010 2023-11-28T17:10:41.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.342+0000 7ff1aaffd640 1 -- 172.21.15.116:0/678813836 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4034660 con 0x7ff1ac0d0010 2023-11-28T17:10:41.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.342+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/678813836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac0d0010 msgr2=0x7ff1ac160270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:41.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.342+0000 7ff1b1c6b640 1 --2- 172.21.15.116:0/678813836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac0d0010 0x7ff1ac160270 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009d90 tx=0x7ff1a402bea0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.342+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/678813836 shutdown_connections 2023-11-28T17:10:41.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.342+0000 7ff1b1c6b640 1 --2- 172.21.15.116:0/678813836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac0d0010 0x7ff1ac160270 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.342+0000 7ff1b1c6b640 1 --2- 172.21.15.116:0/678813836 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1ac15d860 0x7ff1ac0cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.342+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/678813836 >> 172.21.15.116:0/678813836 conn(0x7ff1ac09c540 msgr2=0x7ff1ac0d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:41.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.342+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/678813836 shutdown_connections 2023-11-28T17:10:41.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.343+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/678813836 wait complete. 2023-11-28T17:10:41.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.344+0000 7ff1b1c6b640 1 Processor -- start 2023-11-28T17:10:41.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.344+0000 7ff1b1c6b640 1 -- start start 2023-11-28T17:10:41.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.345+0000 7ff1b1c6b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1ac0d0010 0x7ff1ac14c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:41.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.345+0000 7ff1b1c6b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac15d480 0x7ff1ac14f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:41.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.345+0000 7ff1b1c6b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1ac15d860 0x7ff1ac151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:41.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.345+0000 7ff1b1c6b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac150b80 con 0x7ff1ac15d480 2023-11-28T17:10:41.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.345+0000 7ff1b1c6b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac0a0f90 con 0x7ff1ac0d0010 2023-11-28T17:10:41.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.345+0000 7ff1b1c6b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac14ff70 con 0x7ff1ac15d860 2023-11-28T17:10:41.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.346+0000 7ff1a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac15d480 0x7ff1ac14f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:41.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.346+0000 7ff1a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac15d480 0x7ff1ac14f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42802/0 (socket says 172.21.15.116:42802) 2023-11-28T17:10:41.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.346+0000 7ff1a3fff640 1 -- 172.21.15.116:0/4234814541 learned_addr learned my addr 172.21.15.116:0/4234814541 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:41.349 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.346+0000 7ff1abfff640 1 --2- 172.21.15.116:0/4234814541 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1ac15d860 0x7ff1ac151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.346+0000 7ff1a3fff640 1 -- 172.21.15.116:0/4234814541 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1ac15d860 msgr2=0x7ff1ac151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.346+0000 7ff1ab7fe640 1 --2- 172.21.15.116:0/4234814541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1ac0d0010 0x7ff1ac14c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.346+0000 7ff1a3fff640 1 --2- 172.21.15.116:0/4234814541 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1ac15d860 0x7ff1ac151010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.346+0000 7ff1a3fff640 1 -- 172.21.15.116:0/4234814541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1ac0d0010 msgr2=0x7ff1ac14c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.346+0000 7ff1a3fff640 1 --2- 172.21.15.116:0/4234814541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1ac0d0010 0x7ff1ac14c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.346+0000 7ff1a3fff640 1 -- 172.21.15.116:0/4234814541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a4009a70 con 0x7ff1ac15d480 2023-11-28T17:10:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.346+0000 7ff1a3fff640 1 --2- 172.21.15.116:0/4234814541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac15d480 0x7ff1ac14f4f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff19400d5f0 tx=0x7ff19400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.347+0000 7ff1a8ff9640 1 -- 172.21.15.116:0/4234814541 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1940125f0 con 0x7ff1ac15d480 2023-11-28T17:10:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.347+0000 7ff1a8ff9640 1 -- 172.21.15.116:0/4234814541 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff194012750 con 0x7ff1ac15d480 2023-11-28T17:10:41.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.347+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/4234814541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ac150250 con 0x7ff1ac15d480 2023-11-28T17:10:41.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.347+0000 7ff1a8ff9640 1 -- 172.21.15.116:0/4234814541 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff194012b00 con 0x7ff1ac15d480 2023-11-28T17:10:41.351 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.347+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/4234814541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ac150810 con 0x7ff1ac15d480 2023-11-28T17:10:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.348+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/4234814541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ac067980 con 0x7ff1ac15d480 2023-11-28T17:10:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.349+0000 7ff1a8ff9640 1 -- 172.21.15.116:0/4234814541 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff194012c60 con 0x7ff1ac15d480 2023-11-28T17:10:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.350+0000 7ff1a8ff9640 1 --2- 172.21.15.116:0/4234814541 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff18c06b2d0 0x7ff18c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.350+0000 7ff1a8ff9640 1 -- 172.21.15.116:0/4234814541 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff19408a080 con 0x7ff1ac15d480 2023-11-28T17:10:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.350+0000 7ff1ab7fe640 1 --2- 172.21.15.116:0/4234814541 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff18c06b2d0 0x7ff18c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.351+0000 7ff1ab7fe640 1 --2- 172.21.15.116:0/4234814541 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff18c06b2d0 0x7ff18c06d790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff19c007fd0 tx=0x7ff19c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:41.356 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.355+0000 7ff1a8ff9640 1 -- 172.21.15.116:0/4234814541 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1940bce90 con 0x7ff1ac15d480 2023-11-28T17:10:41.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.544+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/4234814541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff1ac069360 con 0x7ff1ac15d480 2023-11-28T17:10:41.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.545+0000 7ff1a8ff9640 1 -- 172.21.15.116:0/4234814541 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff194054ee0 con 0x7ff1ac15d480 2023-11-28T17:10:41.547 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:10:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.547+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/4234814541 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff18c06b2d0 msgr2=0x7ff18c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:41.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.547+0000 7ff1b1c6b640 1 --2- 172.21.15.116:0/4234814541 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff18c06b2d0 0x7ff18c06d790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff19c007fd0 tx=0x7ff19c009040 comp rx=0 tx=0).stop 2023-11-28T17:10:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.547+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/4234814541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac15d480 msgr2=0x7ff1ac14f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.547+0000 7ff1b1c6b640 1 --2- 172.21.15.116:0/4234814541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac15d480 0x7ff1ac14f4f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff19400d5f0 tx=0x7ff19400dab0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.547+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/4234814541 shutdown_connections 2023-11-28T17:10:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.548+0000 7ff1b1c6b640 1 --2- 172.21.15.116:0/4234814541 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff18c06b2d0 0x7ff18c06d790 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.548+0000 7ff1b1c6b640 1 --2- 172.21.15.116:0/4234814541 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1ac15d860 0x7ff1ac151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.548+0000 7ff1b1c6b640 1 --2- 172.21.15.116:0/4234814541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1ac15d480 0x7ff1ac14f4f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.548+0000 7ff1b1c6b640 1 --2- 172.21.15.116:0/4234814541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1ac0d0010 0x7ff1ac14c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.548+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/4234814541 >> 172.21.15.116:0/4234814541 conn(0x7ff1ac09c540 msgr2=0x7ff1ac0b7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:41.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.548+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/4234814541 shutdown_connections 2023-11-28T17:10:41.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.548+0000 7ff1b1c6b640 1 -- 172.21.15.116:0/4234814541 wait complete. 2023-11-28T17:10:41.567 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:10:41.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.767+0000 7f3ab3a87640 1 Processor -- start 2023-11-28T17:10:41.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.767+0000 7f3ab3a87640 1 -- start start 2023-11-28T17:10:41.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.769+0000 7f3ab3a87640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3aac15b0c0 0x7f3aac15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:41.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.769+0000 7f3ab3a87640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3aac0ce940 0x7f3aac0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:41.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.769+0000 7f3ab3a87640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac071b40 0x7f3aac071fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:41.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.769+0000 7f3ab3a87640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aac0724f0 con 0x7f3aac15b0c0 2023-11-28T17:10:41.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.769+0000 7f3ab3a87640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aac0cd320 con 0x7f3aac0ce940 2023-11-28T17:10:41.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.769+0000 7f3ab3a87640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac0ced80 con 0x7f3aac071b40 2023-11-28T17:10:41.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.769+0000 7f3ab1823640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3aac15b0c0 0x7f3aac15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:41.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.769+0000 7f3ab1823640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3aac15b0c0 0x7f3aac15ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42810/0 (socket says 172.21.15.116:42810) 2023-11-28T17:10:41.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.769+0000 7f3ab1823640 1 -- 172.21.15.116:0/3392732552 learned_addr learned my addr 172.21.15.116:0/3392732552 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:41.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.770+0000 7f3ab1823640 1 -- 172.21.15.116:0/3392732552 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac071b40 msgr2=0x7f3aac071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:41.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.770+0000 7f3ab1823640 1 --2- 172.21.15.116:0/3392732552 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac071b40 0x7f3aac071fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.770+0000 7f3ab1823640 1 -- 172.21.15.116:0/3392732552 >> v2:172.21.15.156:3300/0 conn(0x7f3aac0ce940 msgr2=0x7f3aac0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:41.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.770+0000 7f3ab1823640 1 --2- 172.21.15.116:0/3392732552 >> v2:172.21.15.156:3300/0 conn(0x7f3aac0ce940 0x7f3aac0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.770+0000 7f3ab1823640 1 -- 172.21.15.116:0/3392732552 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac0cbee0 con 0x7f3aac15b0c0 2023-11-28T17:10:41.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.771+0000 7f3ab1823640 1 --2- 172.21.15.116:0/3392732552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac15b0c0 0x7f3aac15ed10 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3aac06bf90 tx=0x7f3aa802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e34ac283f5298170 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:41.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.771+0000 7f3ab0821640 1 -- 172.21.15.116:0/3392732552 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa80317e0 con 0x7f3aac15b0c0 2023-11-28T17:10:41.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.771+0000 7f3ab0821640 1 -- 172.21.15.116:0/3392732552 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa8031940 con 0x7f3aac15b0c0 2023-11-28T17:10:41.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.771+0000 7f3ab0821640 1 -- 172.21.15.116:0/3392732552 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8031cd0 con 0x7f3aac15b0c0 2023-11-28T17:10:41.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.772+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3392732552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac15b0c0 msgr2=0x7f3aac15ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:41.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.772+0000 7f3ab3a87640 1 --2- 172.21.15.116:0/3392732552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac15b0c0 0x7f3aac15ed10 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3aac06bf90 tx=0x7f3aa802b880 comp rx=0 tx=0).stop 2023-11-28T17:10:41.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.772+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3392732552 shutdown_connections 2023-11-28T17:10:41.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.772+0000 7f3ab3a87640 1 --2- 172.21.15.116:0/3392732552 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac071b40 0x7f3aac071fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.772+0000 7f3ab3a87640 1 --2- 172.21.15.116:0/3392732552 >> v2:172.21.15.156:3300/0 conn(0x7f3aac0ce940 0x7f3aac0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.772+0000 7f3ab3a87640 1 --2- 172.21.15.116:0/3392732552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac15b0c0 0x7f3aac15ed10 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.772+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3392732552 >> 172.21.15.116:0/3392732552 conn(0x7f3aac0b0cf0 msgr2=0x7f3aac0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:41.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.772+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3392732552 shutdown_connections 2023-11-28T17:10:41.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.772+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3392732552 wait complete. 2023-11-28T17:10:41.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.774+0000 7f3ab3a87640 1 Processor -- start 2023-11-28T17:10:41.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.774+0000 7f3ab3a87640 1 -- start start 2023-11-28T17:10:41.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.775+0000 7f3ab3a87640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac071b40 0x7f3aac159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:41.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.775+0000 7f3ab3a87640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac0ce940 0x7f3aac159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:41.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.775+0000 7f3ab3a87640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aac15b0c0 0x7f3aac0ad230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:41.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.775+0000 7f3ab3a87640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac0a8460 con 0x7f3aac071b40 2023-11-28T17:10:41.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.775+0000 7f3ab3a87640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac15a080 con 0x7f3aac15b0c0 2023-11-28T17:10:41.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.775+0000 7f3ab3a87640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac1f1da0 con 0x7f3aac0ce940 2023-11-28T17:10:41.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.775+0000 7f3ab1823640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac071b40 0x7f3aac159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:41.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.775+0000 7f3ab1823640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac071b40 0x7f3aac159600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42816/0 (socket says 172.21.15.116:42816) 2023-11-28T17:10:41.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.775+0000 7f3ab1823640 1 -- 172.21.15.116:0/3884199542 learned_addr learned my addr 172.21.15.116:0/3884199542 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:41.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.775+0000 7f3ab1823640 1 -- 172.21.15.116:0/3884199542 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac0ce940 msgr2=0x7f3aac159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:41.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.775+0000 7f3ab1823640 1 --2- 172.21.15.116:0/3884199542 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac0ce940 0x7f3aac159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.775+0000 7f3ab1823640 1 -- 172.21.15.116:0/3884199542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aac15b0c0 msgr2=0x7f3aac0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:41.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.775+0000 7f3ab1823640 1 --2- 172.21.15.116:0/3884199542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aac15b0c0 0x7f3aac0ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.775+0000 7f3ab1823640 1 -- 172.21.15.116:0/3884199542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa80094d0 con 0x7f3aac071b40 2023-11-28T17:10:41.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.776+0000 7f3ab1823640 1 --2- 172.21.15.116:0/3884199542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac071b40 0x7f3aac159600 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8000c00 tx=0x7f3aa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:41.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.776+0000 7f3aa27fc640 1 -- 172.21.15.116:0/3884199542 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8032880 con 0x7f3aac071b40 2023-11-28T17:10:41.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.776+0000 7f3aa27fc640 1 -- 172.21.15.116:0/3884199542 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa80329e0 con 0x7f3aac071b40 2023-11-28T17:10:41.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.776+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3884199542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aac1f2080 con 0x7f3aac071b40 2023-11-28T17:10:41.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.776+0000 7f3aa27fc640 1 -- 172.21.15.116:0/3884199542 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa8032d90 con 0x7f3aac071b40 2023-11-28T17:10:41.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.776+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3884199542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aac1f2640 con 0x7f3aac071b40 2023-11-28T17:10:41.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.777+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3884199542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aac069360 con 0x7f3aac071b40 2023-11-28T17:10:41.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.778+0000 7f3aa27fc640 1 -- 172.21.15.116:0/3884199542 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3aa804d3f0 con 0x7f3aac071b40 2023-11-28T17:10:41.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.779+0000 7f3aa27fc640 1 --2- 172.21.15.116:0/3884199542 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3a9006b2a0 0x7f3a9006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:41.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.779+0000 7f3aa27fc640 1 -- 172.21.15.116:0/3884199542 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3aa80abb40 con 0x7f3aac071b40 2023-11-28T17:10:41.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.780+0000 7f3ab1022640 1 --2- 172.21.15.116:0/3884199542 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3a9006b2a0 0x7f3a9006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:41.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.780+0000 7f3ab1022640 1 --2- 172.21.15.116:0/3884199542 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3a9006b2a0 0x7f3a9006d760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3aac0cb8f0 tx=0x7f3a9c006c90 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:41.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.784+0000 7f3aa27fc640 1 -- 172.21.15.116:0/3884199542 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3aa80769a0 con 0x7f3aac071b40 2023-11-28T17:10:41.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.972+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3884199542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3aac0b1990 con 0x7f3aac071b40 2023-11-28T17:10:41.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.973+0000 7f3aa27fc640 1 -- 172.21.15.116:0/3884199542 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3aa8079fc0 con 0x7f3aac071b40 2023-11-28T17:10:41.974 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:10:41.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.975+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3884199542 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3a9006b2a0 msgr2=0x7f3a9006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:41.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.975+0000 7f3ab3a87640 1 --2- 172.21.15.116:0/3884199542 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3a9006b2a0 0x7f3a9006d760 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3aac0cb8f0 tx=0x7f3a9c006c90 comp rx=0 tx=0).stop 2023-11-28T17:10:41.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.975+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3884199542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac071b40 msgr2=0x7f3aac159600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:41.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.975+0000 7f3ab3a87640 1 --2- 172.21.15.116:0/3884199542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac071b40 0x7f3aac159600 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8000c00 tx=0x7f3aa80046c0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.976+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3884199542 shutdown_connections 2023-11-28T17:10:41.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.976+0000 7f3ab3a87640 1 --2- 172.21.15.116:0/3884199542 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3a9006b2a0 0x7f3a9006d760 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.976+0000 7f3ab3a87640 1 --2- 172.21.15.116:0/3884199542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aac15b0c0 0x7f3aac0ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.976+0000 7f3ab3a87640 1 --2- 172.21.15.116:0/3884199542 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac0ce940 0x7f3aac159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.976+0000 7f3ab3a87640 1 --2- 172.21.15.116:0/3884199542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac071b40 0x7f3aac159600 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:41.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.976+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3884199542 >> 172.21.15.116:0/3884199542 conn(0x7f3aac0b0cf0 msgr2=0x7f3aac06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:41.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.976+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3884199542 shutdown_connections 2023-11-28T17:10:41.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:41.976+0000 7f3ab3a87640 1 -- 172.21.15.116:0/3884199542 wait complete. 2023-11-28T17:10:41.994 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:10:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.206+0000 7f92a5b41640 1 Processor -- start 2023-11-28T17:10:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.207+0000 7f92a5b41640 1 -- start start 2023-11-28T17:10:42.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.208+0000 7f92a5b41640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f92a006c250 0x7f92a006c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:42.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.208+0000 7f92a5b41640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a015fc10 con 0x7f92a015f7b0 2023-11-28T17:10:42.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.208+0000 7f92a5b41640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a0070bd0 con 0x7f92a0144430 2023-11-28T17:10:42.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.208+0000 7f92a5b41640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92a0144810 con 0x7f92a006c250 2023-11-28T17:10:42.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.209+0000 7f929effd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f92a015f7b0 0x7f92a0071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36558/0 (socket says 172.21.15.116:36558) 2023-11-28T17:10:42.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.209+0000 7f929effd640 1 -- 172.21.15.116:0/704352619 learned_addr learned my addr 172.21.15.116:0/704352619 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.209+0000 7f929e7fc640 1 -- 172.21.15.116:0/704352619 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986324586 0 0) 0x7f92a015fc10 con 0x7f92a015f7b0 2023-11-28T17:10:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.209+0000 7f929e7fc640 1 -- 172.21.15.116:0/704352619 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9280003520 con 0x7f92a015f7b0 2023-11-28T17:10:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.209+0000 7f929ffff640 1 --2- 172.21.15.116:0/704352619 >> v2:172.21.15.156:3300/0 conn(0x7f92a006c250 0x7f92a006c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.209+0000 7f929e7fc640 1 -- 172.21.15.116:0/704352619 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1475755263 0 0) 0x7f9280003520 con 0x7f92a015f7b0 2023-11-28T17:10:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.209+0000 7f929e7fc640 1 -- 172.21.15.116:0/704352619 >> v2:172.21.15.156:3300/0 conn(0x7f92a006c250 msgr2=0x7f92a006c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.209+0000 7f929e7fc640 1 --2- 172.21.15.116:0/704352619 >> v2:172.21.15.156:3300/0 conn(0x7f92a006c250 0x7f92a006c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.209+0000 7f929e7fc640 1 -- 172.21.15.116:0/704352619 >> v1:172.21.15.156:6789/0 conn(0x7f92a0144430 legacy=0x7f92a00703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.209+0000 7f929e7fc640 1 -- 172.21.15.116:0/704352619 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a0070eb0 con 0x7f92a015f7b0 2023-11-28T17:10:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.209+0000 7f929e7fc640 1 -- 172.21.15.116:0/704352619 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f9288001670 con 0x7f92a015f7b0 2023-11-28T17:10:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.209+0000 7f929e7fc640 1 -- 172.21.15.116:0/704352619 >> v1:172.21.15.116:6789/0 conn(0x7f92a015f7b0 legacy=0x7f92a0071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.209+0000 7f929e7fc640 1 --2- 172.21.15.116:0/704352619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92800039c0 0x7f9280003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.209+0000 7f929e7fc640 1 --2- 172.21.15.116:0/704352619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f92800047c0 0x7f9280004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929e7fc640 1 --2- 172.21.15.116:0/704352619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92800051d0 0x7f928000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929e7fc640 1 -- 172.21.15.116:0/704352619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a0144810 con 0x7f92800039c0 2023-11-28T17:10:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929e7fc640 1 -- 172.21.15.116:0/704352619 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928000c100 con 0x7f92800051d0 2023-11-28T17:10:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929e7fc640 1 -- 172.21.15.116:0/704352619 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f928000c2c0 con 0x7f92800047c0 2023-11-28T17:10:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929f7fe640 1 --2- 172.21.15.116:0/704352619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92800039c0 0x7f9280003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929effd640 1 --2- 172.21.15.116:0/704352619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f92800047c0 0x7f9280004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929ffff640 1 --2- 172.21.15.116:0/704352619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92800051d0 0x7f928000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929f7fe640 1 -- 172.21.15.116:0/704352619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f92800047c0 msgr2=0x7f9280004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929f7fe640 1 --2- 172.21.15.116:0/704352619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f92800047c0 0x7f9280004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929f7fe640 1 -- 172.21.15.116:0/704352619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92800051d0 msgr2=0x7f928000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929f7fe640 1 --2- 172.21.15.116:0/704352619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92800051d0 0x7f928000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929f7fe640 1 -- 172.21.15.116:0/704352619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9280003520 con 0x7f92800039c0 2023-11-28T17:10:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929effd640 1 --2- 172.21.15.116:0/704352619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f92800047c0 0x7f9280004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929ffff640 1 --2- 172.21.15.116:0/704352619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92800051d0 0x7f928000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.210+0000 7f929ffff640 1 -- 172.21.15.116:0/704352619 reap_dead start 2023-11-28T17:10:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.211+0000 7f929f7fe640 1 --2- 172.21.15.116:0/704352619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92800039c0 0x7f9280003dc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9290003d80 tx=0x7f92800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f46777cb9a0ed9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:42.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.212+0000 7f929e7fc640 1 -- 172.21.15.116:0/704352619 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9290032600 con 0x7f92800039c0 2023-11-28T17:10:42.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.212+0000 7f929e7fc640 1 -- 172.21.15.116:0/704352619 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f929002e050 con 0x7f92800039c0 2023-11-28T17:10:42.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.212+0000 7f929e7fc640 1 -- 172.21.15.116:0/704352619 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92900329e0 con 0x7f92800039c0 2023-11-28T17:10:42.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.212+0000 7f92a5b41640 1 -- 172.21.15.116:0/704352619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92800039c0 msgr2=0x7f9280003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:42.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.212+0000 7f92a5b41640 1 --2- 172.21.15.116:0/704352619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92800039c0 0x7f9280003dc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9290003d80 tx=0x7f92800057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.212+0000 7f92a5b41640 1 -- 172.21.15.116:0/704352619 shutdown_connections 2023-11-28T17:10:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.212+0000 7f92a5b41640 1 --2- 172.21.15.116:0/704352619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92800039c0 0x7f9280003dc0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.212+0000 7f92a5b41640 1 -- 172.21.15.116:0/704352619 >> 172.21.15.116:0/704352619 conn(0x7f92a006a4c0 msgr2=0x7f92a006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.212+0000 7f92a5b41640 1 -- 172.21.15.116:0/704352619 shutdown_connections 2023-11-28T17:10:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.213+0000 7f92a5b41640 1 -- 172.21.15.116:0/704352619 wait complete. 2023-11-28T17:10:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.214+0000 7f92a5b41640 1 Processor -- start 2023-11-28T17:10:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.214+0000 7f92a5b41640 1 -- start start 2023-11-28T17:10:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.215+0000 7f92a5b41640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92800039c0 0x7f92a015a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.215+0000 7f92a5b41640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a015c130 0x7f92a015c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.215+0000 7f92a5b41640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f92a01f6220 0x7f92a01f66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.215+0000 7f92a5b41640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a015ce80 con 0x7f92800039c0 2023-11-28T17:10:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.215+0000 7f92a5b41640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92a012f5c0 con 0x7f92a015c130 2023-11-28T17:10:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.215+0000 7f92a5b41640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92a012f8a0 con 0x7f92a01f6220 2023-11-28T17:10:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.216+0000 7f929f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92800039c0 0x7f92a015a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.216+0000 7f929f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92800039c0 0x7f92a015a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42836/0 (socket says 172.21.15.116:42836) 2023-11-28T17:10:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.216+0000 7f929f7fe640 1 -- 172.21.15.116:0/182060561 learned_addr learned my addr 172.21.15.116:0/182060561 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.216+0000 7f929ffff640 1 --2- 172.21.15.116:0/182060561 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f92a01f6220 0x7f92a01f66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.216+0000 7f929f7fe640 1 -- 172.21.15.116:0/182060561 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f92a01f6220 msgr2=0x7f92a01f66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.216+0000 7f929f7fe640 1 --2- 172.21.15.116:0/182060561 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f92a01f6220 0x7f92a01f66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.216+0000 7f929f7fe640 1 -- 172.21.15.116:0/182060561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a015c130 msgr2=0x7f92a015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.216+0000 7f929f7fe640 1 --2- 172.21.15.116:0/182060561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a015c130 0x7f92a015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.216+0000 7f929f7fe640 1 -- 172.21.15.116:0/182060561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a012fb80 con 0x7f92800039c0 2023-11-28T17:10:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.216+0000 7f929f7fe640 1 --2- 172.21.15.116:0/182060561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92800039c0 0x7f92a015a850 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9290000c00 tx=0x7f92900036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.217+0000 7f9297fff640 1 -- 172.21.15.116:0/182060561 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929003c410 con 0x7f92800039c0 2023-11-28T17:10:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.217+0000 7f9297fff640 1 -- 172.21.15.116:0/182060561 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f929003c570 con 0x7f92800039c0 2023-11-28T17:10:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.217+0000 7f92a5b41640 1 -- 172.21.15.116:0/182060561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a012fe60 con 0x7f92800039c0 2023-11-28T17:10:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.217+0000 7f9297fff640 1 -- 172.21.15.116:0/182060561 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929003c8e0 con 0x7f92800039c0 2023-11-28T17:10:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.217+0000 7f92a5b41640 1 -- 172.21.15.116:0/182060561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a015ad90 con 0x7f92800039c0 2023-11-28T17:10:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.218+0000 7f92a5b41640 1 -- 172.21.15.116:0/182060561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a00693d0 con 0x7f92800039c0 2023-11-28T17:10:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.219+0000 7f9297fff640 1 -- 172.21.15.116:0/182060561 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f9290039050 con 0x7f92800039c0 2023-11-28T17:10:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.220+0000 7f9297fff640 1 --2- 172.21.15.116:0/182060561 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f928406b2d0 0x7f928406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.220+0000 7f9297fff640 1 -- 172.21.15.116:0/182060561 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92900ab100 con 0x7f92800039c0 2023-11-28T17:10:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.220+0000 7f929effd640 1 --2- 172.21.15.116:0/182060561 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f928406b2d0 0x7f928406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.221+0000 7f929effd640 1 --2- 172.21.15.116:0/182060561 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f928406b2d0 0x7f928406d790 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f92a00ccf50 tx=0x7f9288009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.224+0000 7f9297fff640 1 -- 172.21.15.116:0/182060561 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92900760a0 con 0x7f92800039c0 2023-11-28T17:10:42.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.428+0000 7f92a5b41640 1 -- 172.21.15.116:0/182060561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f92a015ce80 con 0x7f92800039c0 2023-11-28T17:10:42.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.429+0000 7f9297fff640 1 -- 172.21.15.116:0/182060561 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f92900796c0 con 0x7f92800039c0 2023-11-28T17:10:42.431 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-28T17:10:42.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.431+0000 7f92a5b41640 1 -- 172.21.15.116:0/182060561 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f928406b2d0 msgr2=0x7f928406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:42.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.431+0000 7f92a5b41640 1 --2- 172.21.15.116:0/182060561 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f928406b2d0 0x7f928406d790 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f92a00ccf50 tx=0x7f9288009040 comp rx=0 tx=0).stop 2023-11-28T17:10:42.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.431+0000 7f92a5b41640 1 -- 172.21.15.116:0/182060561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92800039c0 msgr2=0x7f92a015a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:42.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.431+0000 7f92a5b41640 1 --2- 172.21.15.116:0/182060561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92800039c0 0x7f92a015a850 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9290000c00 tx=0x7f92900036f0 comp rx=0 tx=0).stop 2023-11-28T17:10:42.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.431+0000 7f92a5b41640 1 -- 172.21.15.116:0/182060561 shutdown_connections 2023-11-28T17:10:42.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.431+0000 7f92a5b41640 1 --2- 172.21.15.116:0/182060561 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f928406b2d0 0x7f928406d790 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:42.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.431+0000 7f92a5b41640 1 --2- 172.21.15.116:0/182060561 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f92a01f6220 0x7f92a01f66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:42.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.431+0000 7f92a5b41640 1 --2- 172.21.15.116:0/182060561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a015c130 0x7f92a015c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:42.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.431+0000 7f92a5b41640 1 --2- 172.21.15.116:0/182060561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92800039c0 0x7f92a015a850 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:42.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.431+0000 7f92a5b41640 1 -- 172.21.15.116:0/182060561 >> 172.21.15.116:0/182060561 conn(0x7f92a006a4c0 msgr2=0x7f92a00cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:42.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.432+0000 7f92a5b41640 1 -- 172.21.15.116:0/182060561 shutdown_connections 2023-11-28T17:10:42.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:42.432+0000 7f92a5b41640 1 -- 172.21.15.116:0/182060561 wait complete. 2023-11-28T17:10:42.455 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2023-11-28T17:10:42.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:42.455 DEBUG:teuthology.orchestra.run.smithi156:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-11-28T17:10:42.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[94394]: fuse finished with error 0 and tester_r 0 2023-11-28T17:10:42.562 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:10:48.664 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:10:48.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:48.664 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-28T17:10:48.726 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:10:48.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.947+0000 7f5ce18d0640 1 Processor -- start 2023-11-28T17:10:48.949 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.947+0000 7f5ce18d0640 1 -- start start 2023-11-28T17:10:48.950 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.949+0000 7f5ce18d0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5cdc15aba0 0x7f5cdc164e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.949+0000 7f5ce18d0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cdc0aea20 con 0x7f5cdc13cee0 2023-11-28T17:10:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.949+0000 7f5ce18d0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cdc13d2c0 con 0x7f5cdc06c960 2023-11-28T17:10:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.949+0000 7f5ce18d0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cdc15b3a0 con 0x7f5cdc15aba0 2023-11-28T17:10:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.949+0000 7f5cdaffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5cdc13cee0 0x7f5cdc06c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34426/0 (socket says 172.21.15.116:34426) 2023-11-28T17:10:48.951 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.949+0000 7f5cdaffd640 1 -- 172.21.15.116:0/2959807327 learned_addr learned my addr 172.21.15.116:0/2959807327 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:48.953 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.950+0000 7f5cd9ffb640 1 -- 172.21.15.116:0/2959807327 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996859651 0 0) 0x7f5cdc0aea20 con 0x7f5cdc13cee0 2023-11-28T17:10:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.950+0000 7f5cd9ffb640 1 -- 172.21.15.116:0/2959807327 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cc8005730 con 0x7f5cdc13cee0 2023-11-28T17:10:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.950+0000 7f5cdb7fe640 1 --2- 172.21.15.116:0/2959807327 >> v2:172.21.15.156:3300/0 conn(0x7f5cdc15aba0 0x7f5cdc164e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.950+0000 7f5cd9ffb640 1 -- 172.21.15.116:0/2959807327 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3947664743 0 0) 0x7f5cc8005730 con 0x7f5cdc13cee0 2023-11-28T17:10:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.950+0000 7f5cd9ffb640 1 -- 172.21.15.116:0/2959807327 >> v2:172.21.15.156:3300/0 conn(0x7f5cdc15aba0 msgr2=0x7f5cdc164e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.950+0000 7f5cd9ffb640 1 --2- 172.21.15.116:0/2959807327 >> v2:172.21.15.156:3300/0 conn(0x7f5cdc15aba0 0x7f5cdc164e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:48.954 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.950+0000 7f5cd9ffb640 1 -- 172.21.15.116:0/2959807327 >> v1:172.21.15.156:6789/0 conn(0x7f5cdc06c960 legacy=0x7f5cdc15a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.950+0000 7f5cd9ffb640 1 -- 172.21.15.116:0/2959807327 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc15b0c0 con 0x7f5cdc13cee0 2023-11-28T17:10:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.950+0000 7f5cd9ffb640 1 -- 172.21.15.116:0/2959807327 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f5ccc0019f0 con 0x7f5cdc13cee0 2023-11-28T17:10:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.950+0000 7f5cd9ffb640 1 -- 172.21.15.116:0/2959807327 >> v1:172.21.15.116:6789/0 conn(0x7f5cdc13cee0 legacy=0x7f5cdc06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.950+0000 7f5cd9ffb640 1 --2- 172.21.15.116:0/2959807327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5cc8005b80 0x7f5cc8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.950+0000 7f5cd9ffb640 1 --2- 172.21.15.116:0/2959807327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cc80069d0 0x7f5cc8006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:48.955 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cd9ffb640 1 --2- 172.21.15.116:0/2959807327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cc80074b0 0x7f5cc800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cd9ffb640 1 -- 172.21.15.116:0/2959807327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc15b3a0 con 0x7f5cc80069d0 2023-11-28T17:10:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cd9ffb640 1 -- 172.21.15.116:0/2959807327 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc800e380 con 0x7f5cc80074b0 2023-11-28T17:10:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cd9ffb640 1 -- 172.21.15.116:0/2959807327 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cc800e540 con 0x7f5cc8005b80 2023-11-28T17:10:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cda7fc640 1 --2- 172.21.15.116:0/2959807327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cc80069d0 0x7f5cc8006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cdb7fe640 1 --2- 172.21.15.116:0/2959807327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cc80074b0 0x7f5cc800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cda7fc640 1 -- 172.21.15.116:0/2959807327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5cc8005b80 msgr2=0x7f5cc8005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cdaffd640 1 --2- 172.21.15.116:0/2959807327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5cc8005b80 0x7f5cc8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cda7fc640 1 --2- 172.21.15.116:0/2959807327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5cc8005b80 0x7f5cc8005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cda7fc640 1 -- 172.21.15.116:0/2959807327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cc80074b0 msgr2=0x7f5cc800db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cda7fc640 1 --2- 172.21.15.116:0/2959807327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cc80074b0 0x7f5cc800db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:48.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cda7fc640 1 -- 172.21.15.116:0/2959807327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc8005730 con 0x7f5cc80069d0 2023-11-28T17:10:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cdb7fe640 1 --2- 172.21.15.116:0/2959807327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cc80074b0 0x7f5cc800db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cdb7fe640 1 -- 172.21.15.116:0/2959807327 reap_dead start 2023-11-28T17:10:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.951+0000 7f5cdaffd640 1 --2- 172.21.15.116:0/2959807327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5cc8005b80 0x7f5cc8005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.952+0000 7f5cda7fc640 1 --2- 172.21.15.116:0/2959807327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cc80069d0 0x7f5cc8006e50 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4001580 tx=0x7f5cc8007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6ea8c6565e249d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.953+0000 7f5cd9ffb640 1 -- 172.21.15.116:0/2959807327 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc40052d0 con 0x7f5cc80069d0 2023-11-28T17:10:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.953+0000 7f5cd9ffb640 1 -- 172.21.15.116:0/2959807327 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc402e050 con 0x7f5cc80069d0 2023-11-28T17:10:48.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.953+0000 7f5cd9ffb640 1 -- 172.21.15.116:0/2959807327 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4003830 con 0x7f5cc80069d0 2023-11-28T17:10:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.953+0000 7f5ce18d0640 1 -- 172.21.15.116:0/2959807327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cc80069d0 msgr2=0x7f5cc8006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.953+0000 7f5ce18d0640 1 --2- 172.21.15.116:0/2959807327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cc80069d0 0x7f5cc8006e50 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4001580 tx=0x7f5cc8007a00 comp rx=0 tx=0).stop 2023-11-28T17:10:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.953+0000 7f5ce18d0640 1 -- 172.21.15.116:0/2959807327 shutdown_connections 2023-11-28T17:10:48.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.953+0000 7f5ce18d0640 1 --2- 172.21.15.116:0/2959807327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cc80069d0 0x7f5cc8006e50 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.953+0000 7f5ce18d0640 1 -- 172.21.15.116:0/2959807327 >> 172.21.15.116:0/2959807327 conn(0x7f5cdc06a4c0 msgr2=0x7f5cdc06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.953+0000 7f5ce18d0640 1 -- 172.21.15.116:0/2959807327 shutdown_connections 2023-11-28T17:10:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.954+0000 7f5ce18d0640 1 -- 172.21.15.116:0/2959807327 wait complete. 2023-11-28T17:10:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.955+0000 7f5ce18d0640 1 Processor -- start 2023-11-28T17:10:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.955+0000 7f5ce18d0640 1 -- start start 2023-11-28T17:10:48.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.956+0000 7f5ce18d0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cc80069d0 0x7f5cdc15cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.956+0000 7f5ce18d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cdc1f29e0 0x7f5cdc159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.956+0000 7f5ce18d0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5cdc1595b0 0x7f5cdc14f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.956+0000 7f5ce18d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc159a30 con 0x7f5cdc1f29e0 2023-11-28T17:10:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.956+0000 7f5ce18d0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc15b4e0 con 0x7f5cc80069d0 2023-11-28T17:10:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.956+0000 7f5ce18d0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc15b730 con 0x7f5cdc1595b0 2023-11-28T17:10:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.957+0000 7f5cda7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cdc1f29e0 0x7f5cdc159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:48.963 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.957+0000 7f5cda7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cdc1f29e0 0x7f5cdc159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53378/0 (socket says 172.21.15.116:53378) 2023-11-28T17:10:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.957+0000 7f5cda7fc640 1 -- 172.21.15.116:0/1377388551 learned_addr learned my addr 172.21.15.116:0/1377388551 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.957+0000 7f5cdaffd640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cc80069d0 0x7f5cdc15cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.957+0000 7f5cdb7fe640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5cdc1595b0 0x7f5cdc14f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.957+0000 7f5cda7fc640 1 -- 172.21.15.116:0/1377388551 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5cdc1595b0 msgr2=0x7f5cdc14f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.957+0000 7f5cda7fc640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5cdc1595b0 0x7f5cdc14f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.957+0000 7f5cda7fc640 1 -- 172.21.15.116:0/1377388551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cc80069d0 msgr2=0x7f5cdc15cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:48.964 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.957+0000 7f5cda7fc640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cc80069d0 0x7f5cdc15cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.957+0000 7f5cda7fc640 1 -- 172.21.15.116:0/1377388551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc15ba10 con 0x7f5cdc1f29e0 2023-11-28T17:10:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.957+0000 7f5cdaffd640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cc80069d0 0x7f5cdc15cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:10:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.958+0000 7f5cda7fc640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cdc1f29e0 0x7f5cdc159070 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5cc402ba40 tx=0x7f5cc4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.958+0000 7f5cd3fff640 1 -- 172.21.15.116:0/1377388551 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc40076c0 con 0x7f5cdc1f29e0 2023-11-28T17:10:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.958+0000 7f5cd3fff640 1 -- 172.21.15.116:0/1377388551 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc4007a30 con 0x7f5cdc1f29e0 2023-11-28T17:10:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.958+0000 7f5ce18d0640 1 -- 172.21.15.116:0/1377388551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cdc15bcf0 con 0x7f5cdc1f29e0 2023-11-28T17:10:48.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.958+0000 7f5cd3fff640 1 -- 172.21.15.116:0/1377388551 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4006be0 con 0x7f5cdc1f29e0 2023-11-28T17:10:48.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.958+0000 7f5ce18d0640 1 -- 172.21.15.116:0/1377388551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cdc15c2b0 con 0x7f5cdc1f29e0 2023-11-28T17:10:48.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.959+0000 7f5ce18d0640 1 -- 172.21.15.116:0/1377388551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cdc15d7f0 con 0x7f5cdc1f29e0 2023-11-28T17:10:48.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.960+0000 7f5cd3fff640 1 -- 172.21.15.116:0/1377388551 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f5cc4006d40 con 0x7f5cdc1f29e0 2023-11-28T17:10:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.961+0000 7f5cd3fff640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5cbc06b2d0 0x7f5cbc06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.962+0000 7f5cd3fff640 1 -- 172.21.15.116:0/1377388551 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5cc40ab8b0 con 0x7f5cdc1f29e0 2023-11-28T17:10:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.962+0000 7f5cdaffd640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5cbc06b2d0 0x7f5cbc06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:48.967 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.962+0000 7f5cdaffd640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5cbc06b2d0 0x7f5cbc06d790 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc050060 tx=0x7f5ccc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:48.968 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:48.966+0000 7f5cd3fff640 1 -- 172.21.15.116:0/1377388551 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5cc4076690 con 0x7f5cdc1f29e0 2023-11-28T17:10:49.196 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.195+0000 7f5ce18d0640 1 -- 172.21.15.116:0/1377388551 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cdc13d2c0 con 0x7f5cdc1f29e0 2023-11-28T17:10:49.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.196+0000 7f5cd3fff640 1 -- 172.21.15.116:0/1377388551 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f5cdc13d2c0 con 0x7f5cdc1f29e0 2023-11-28T17:10:49.198 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:49.198 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/491158514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":491158514},{"type":"v1","addr":"172.21.15.156:6834","nonce":491158514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3927645596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3927645596},{"type":"v1","addr":"172.21.15.116:6835","nonce":3927645596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4738,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/141796020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":141796020},{"type":"v1","addr":"172.21.15.156:6835","nonce":141796020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:10:19.725913+0000","modified":"2023-11-28T17:10:22.808343+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6837/3177126349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3177126349},{"type":"v1","addr":"172.21.15.116:6837","nonce":3177126349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T17:10:49.198 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 15 2023-11-28T17:10:49.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.199+0000 7f5ce18d0640 1 -- 172.21.15.116:0/1377388551 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5cbc06b2d0 msgr2=0x7f5cbc06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:49.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.199+0000 7f5ce18d0640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5cbc06b2d0 0x7f5cbc06d790 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc050060 tx=0x7f5ccc009040 comp rx=0 tx=0).stop 2023-11-28T17:10:49.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.199+0000 7f5ce18d0640 1 -- 172.21.15.116:0/1377388551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cdc1f29e0 msgr2=0x7f5cdc159070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.199+0000 7f5ce18d0640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cdc1f29e0 0x7f5cdc159070 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5cc402ba40 tx=0x7f5cc4001120 comp rx=0 tx=0).stop 2023-11-28T17:10:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.199+0000 7f5ce18d0640 1 -- 172.21.15.116:0/1377388551 shutdown_connections 2023-11-28T17:10:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.199+0000 7f5ce18d0640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5cbc06b2d0 0x7f5cbc06d790 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.199+0000 7f5ce18d0640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5cdc1595b0 0x7f5cdc14f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.199+0000 7f5ce18d0640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cdc1f29e0 0x7f5cdc159070 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.199+0000 7f5ce18d0640 1 --2- 172.21.15.116:0/1377388551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5cc80069d0 0x7f5cdc15cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.199+0000 7f5ce18d0640 1 -- 172.21.15.116:0/1377388551 >> 172.21.15.116:0/1377388551 conn(0x7f5cdc06a4c0 msgr2=0x7f5cdc0ad3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:49.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.199+0000 7f5ce18d0640 1 -- 172.21.15.116:0/1377388551 shutdown_connections 2023-11-28T17:10:49.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.199+0000 7f5ce18d0640 1 -- 172.21.15.116:0/1377388551 wait complete. 2023-11-28T17:10:49.220 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2023-11-28T17:10:49.445 INFO:teuthology.orchestra.run.smithi116.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":82,"msgr_send_messages":67,"msgr_recv_bytes":121284,"msgr_send_bytes":37253,"msgr_created_connections":8,"msgr_active_connections":1,"msgr_running_total_time":0.021900149,"msgr_running_send_time":0.005517022,"msgr_running_recv_time":0.015728504,"msgr_running_fast_dispatch_time":0.001285442,"msgr_send_messages_queue_lat":{"avgcount":67,"sum":0.066495576,"avgtime":0.000992471},"msgr_handle_ack_lat":{"avgcount":23,"sum":0.000004874,"avgtime":0.000000211},"msgr_recv_encrypted_bytes":115040,"msgr_send_encrypted_bytes":14560},"AsyncMessenger::Worker-1":{"msgr_recv_messages":34,"msgr_send_messages":33,"msgr_recv_bytes":10239,"msgr_send_bytes":113807,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":0.011505469,"msgr_running_send_time":0.002538017,"msgr_running_recv_time":0.007476507,"msgr_running_fast_dispatch_time":0.001150231,"msgr_send_messages_queue_lat":{"avgcount":32,"sum":0.171276220,"avgtime":0.005352381},"msgr_handle_ack_lat":{"avgcount":19,"sum":0.000005149,"avgtime":0.000000271},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":99024},"AsyncMessenger::Worker-2":{"msgr_recv_messages":26,"msgr_send_messages":27,"msgr_recv_bytes":6649,"msgr_send_bytes":14220,"msgr_created_connections":5,"msgr_active_connections":3,"msgr_running_total_time":0.005573228,"msgr_running_send_time":0.000754653,"msgr_running_recv_time":0.003601071,"msgr_running_fast_dispatch_time":0.000842297,"msgr_send_messages_queue_lat":{"avgcount":26,"sum":0.085082837,"avgtime":0.003272416},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":45,"sum":0.003607796,"avgtime":0.000080173}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2,"sum":0.000020594,"avgtime":0.000010297}},"mds":{"request":3,"reply":3,"reply_latency":{"avgcount":3,"sum":0.001013830,"avgtime":0.000337943},"slow_reply":0,"forward":0,"dir_fetch_complete":0,"dir_fetch_keys":0,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":10,"inodes_top":0,"inodes_bottom":10,"inodes_pin_tail":0,"inodes_pinned":10,"inodes_expired":0,"inodes_with_caps":0,"caps":0,"subtrees":2,"traverse":5,"traverse_hit":5,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":44,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":0,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":0,"handle_client_caps_dirty":0,"handle_client_cap_release":0,"process_request_cap_release":1},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":7,"evex":0,"evtrm":0,"ev":7,"evexg":0,"evexd":0,"segadd":2,"segex":0,"segtrm":0,"seg":2,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4197222,"rdpos":0,"jlat":{"avgcount":6,"sum":0.021537577,"avgtime":0.003589596},"replayed":0},"mds_mem":{"ino":13,"ino+":13,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":10,"dn+":10,"dn-":0,"cap":0,"cap+":2,"cap-":2,"rss":38912,"heap":207196},"mds_server":{"dispatch_client_request":3,"dispatch_server_request":0,"handle_client_request":3,"handle_client_session":8,"handle_peer_request":0,"req_create_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getattr_latency":{"avgcount":2,"sum":0.000649879,"avgtime":0.000324939},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":1,"sum":0.000363951,"avgtime":0.000363951},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":0,"session_add":2,"session_remove":2,"sessions_open":0,"sessions_stale":0,"total_load":2,"average_load":1,"avg_session_uptime":8,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":213331,"buffer_anon_items":58,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5696,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":51520,"mds_co_items":211,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":45,"op_send_bytes":14128,"op_resend":31,"op_reply":45,"op_latency":{"avgcount":45,"sum":0.953984161,"avgtime":0.021199648},"op_inflight":0,"oplen_avg":{"avgcount":45,"sum":69},"op":45,"op_r":0,"op_w":45,"op_rmw":0,"op_pg":0,"osdop_stat":0,"osdop_create":11,"osdop_read":0,"osdop_write":6,"osdop_writefull":8,"osdop_writesame":0,"osdop_append":0,"osdop_zero":0,"osdop_truncate":0,"osdop_delete":6,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":0,"osdop_setxattr":22,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":36,"map_full":0,"map_inc":3,"osd_sessions":6,"osd_session_open":6,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":0,"omap_del":1},"oft":{"omap_total_objs":1,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":136,"get_sum":121379,"get_or_fail_fail":0,"get_or_fail_success":136,"take":0,"take_sum":0,"put":136,"put_sum":121379,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":45,"take_sum":14128,"put":45,"put_sum":14128,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":45,"take_sum":45,"put":45,"put_sum":45,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x561658c2c6e0":{"val":0,"max":3758096384,"get_started":0,"get":7,"get_sum":2918,"get_or_fail_fail":0,"get_or_fail_success":7,"take":0,"take_sum":0,"put":6,"put_sum":2918,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2023-11-28T17:10:49.447 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 8, "msgr_handle_ack_lat": { "avgcount": 23, "avgtime": 2.11e-07, "sum": 4.874e-06 }, "msgr_recv_bytes": 121284, "msgr_recv_encrypted_bytes": 115040, "msgr_recv_messages": 82, "msgr_running_fast_dispatch_time": 0.001285442, "msgr_running_recv_time": 0.015728504, "msgr_running_send_time": 0.005517022, "msgr_running_total_time": 0.021900149, "msgr_send_bytes": 37253, "msgr_send_encrypted_bytes": 14560, "msgr_send_messages": 67, "msgr_send_messages_queue_lat": { "avgcount": 67, "avgtime": 0.000992471, "sum": 0.066495576 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 19, "avgtime": 2.71e-07, "sum": 5.149e-06 }, "msgr_recv_bytes": 10239, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 34, "msgr_running_fast_dispatch_time": 0.001150231, "msgr_running_recv_time": 0.007476507, "msgr_running_send_time": 0.002538017, "msgr_running_total_time": 0.011505469, "msgr_send_bytes": 113807, "msgr_send_encrypted_bytes": 99024, "msgr_send_messages": 33, "msgr_send_messages_queue_lat": { "avgcount": 32, "avgtime": 0.005352381, "sum": 0.17127622 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 6649, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 26, "msgr_running_fast_dispatch_time": 0.000842297, "msgr_running_recv_time": 0.003601071, "msgr_running_send_time": 0.000754653, "msgr_running_total_time": 0.005573228, "msgr_send_bytes": 14220, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 27, "msgr_send_messages_queue_lat": { "avgcount": 26, "avgtime": 0.003272416, "sum": 0.085082837 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 45, "avgtime": 8.0173e-05, "sum": 0.003607796 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2, "avgtime": 1.0297e-05, "sum": 2.0594e-05 }, "queue_len": 0 }, "mds": { "caps": 0, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 0, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 0, "dir_fetch_keys": 0, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 0, "handle_client_caps_dirty": 0, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 10, "inodes_bottom": 10, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 10, "inodes_top": 0, "inodes_with_caps": 0, "load_cent": 44, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1, "q": 0, "reply": 3, "reply_latency": { "avgcount": 3, "avgtime": 0.000337943, "sum": 0.00101383 }, "request": 3, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7, "evadd": 7, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 6, "avgtime": 0.003589596, "sum": 0.021537577 }, "rdpos": 0, "replayed": 0, "seg": 2, "segadd": 2, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 4197222 }, "mds_mem": { "cap": 0, "cap+": 2, "cap-": 2, "dir": 12, "dir+": 12, "dir-": 0, "dn": 10, "dn+": 10, "dn-": 0, "heap": 207196, "ino": 13, "ino+": 13, "ino-": 0, "rss": 38912 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3, "dispatch_server_request": 0, "handle_client_request": 3, "handle_client_session": 8, "handle_peer_request": 0, "req_create_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.000324939, "sum": 0.000649879 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 1, "avgtime": 0.000363951, "sum": 0.000363951 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 1, "avg_session_uptime": 8, "mdthresh_evicted": 0, "session_add": 2, "session_count": 0, "session_remove": 2, "sessions_open": 0, "sessions_stale": 0, "total_load": 2 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 213331, "buffer_anon_items": 58, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 51520, "mds_co_items": 211, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5696, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 36, "map_full": 0, "map_inc": 3, "omap_del": 1, "omap_rd": 0, "omap_wr": 15, "op": 45, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 45, "avgtime": 0.021199648, "sum": 0.953984161 }, "op_pg": 0, "op_r": 0, "op_reply": 45, "op_resend": 31, "op_rmw": 0, "op_send": 45, "op_send_bytes": 14128, "op_w": 45, "oplen_avg": { "avgcount": 45, "sum": 69 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 6, "osd_sessions": 6, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 11, "osdop_delete": 6, "osdop_getxattr": 0, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 0, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 22, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 0, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 6, "osdop_writefull": 8, "osdop_writesame": 0, "osdop_zero": 0, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 1, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 136, "get_or_fail_fail": 0, "get_or_fail_success": 136, "get_started": 0, "get_sum": 121379, "max": 104857600, "put": 136, "put_sum": 121379, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 45, "put_sum": 14128, "take": 45, "take_sum": 14128, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 45, "put_sum": 45, "take": 45, "take_sum": 45, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x561658c2c6e0": { "get": 7, "get_or_fail_fail": 0, "get_or_fail_success": 7, "get_started": 0, "get_sum": 2918, "max": 3758096384, "put": 6, "put_sum": 2918, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2023-11-28T17:10:49.448 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T17:10:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.651+0000 7f162ab08640 1 Processor -- start 2023-11-28T17:10:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.652+0000 7f162ab08640 1 -- start start 2023-11-28T17:10:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.653+0000 7f162ab08640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f162415d2c0 0x7f16240cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.653+0000 7f162ab08640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16240b6640 con 0x7f16240d2f40 2023-11-28T17:10:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.653+0000 7f162ab08640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16240d3480 con 0x7f16240cdf70 2023-11-28T17:10:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.653+0000 7f162ab08640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16240ce420 con 0x7f162415d2c0 2023-11-28T17:10:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.653+0000 7f16288a4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f162415d2c0 0x7f16240cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.653+0000 7f16290a5640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f16240d2f40 0x7f16240b6ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34436/0 (socket says 172.21.15.116:34436) 2023-11-28T17:10:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.653+0000 7f16288a4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f162415d2c0 0x7f16240cda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53384/0 (socket says 172.21.15.116:53384) 2023-11-28T17:10:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.653+0000 7f16290a5640 1 -- 172.21.15.116:0/1709381486 learned_addr learned my addr 172.21.15.116:0/1709381486 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.654+0000 7f16288a4640 1 -- 172.21.15.116:0/1709381486 >> v1:172.21.15.156:6789/0 conn(0x7f16240cdf70 legacy=0x7f16240d2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:49.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.654+0000 7f16288a4640 1 -- 172.21.15.116:0/1709381486 >> v1:172.21.15.116:6789/0 conn(0x7f16240d2f40 legacy=0x7f16240b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:49.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.654+0000 7f16288a4640 1 -- 172.21.15.116:0/1709381486 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16240d3760 con 0x7f162415d2c0 2023-11-28T17:10:49.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.654+0000 7f16237fe640 1 -- 172.21.15.116:0/1709381486 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826365044 0 0) 0x7f16240b6640 con 0x7f16240d2f40 2023-11-28T17:10:49.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.655+0000 7f16288a4640 1 --2- 172.21.15.116:0/1709381486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f162415d2c0 0x7f16240cda30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f16180097a0 tx=0x7f161802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0df718777e67c4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:49.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.655+0000 7f16237fe640 1 -- 172.21.15.116:0/1709381486 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618031830 con 0x7f162415d2c0 2023-11-28T17:10:49.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.655+0000 7f16237fe640 1 -- 172.21.15.116:0/1709381486 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1618031990 con 0x7f162415d2c0 2023-11-28T17:10:49.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.655+0000 7f16237fe640 1 -- 172.21.15.116:0/1709381486 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618031d20 con 0x7f162415d2c0 2023-11-28T17:10:49.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.655+0000 7f162ab08640 1 -- 172.21.15.116:0/1709381486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f162415d2c0 msgr2=0x7f16240cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:49.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.655+0000 7f162ab08640 1 --2- 172.21.15.116:0/1709381486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f162415d2c0 0x7f16240cda30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f16180097a0 tx=0x7f161802b8d0 comp rx=0 tx=0).stop 2023-11-28T17:10:49.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.656+0000 7f162ab08640 1 -- 172.21.15.116:0/1709381486 shutdown_connections 2023-11-28T17:10:49.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.656+0000 7f162ab08640 1 --2- 172.21.15.116:0/1709381486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f162415d2c0 0x7f16240cda30 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:49.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.656+0000 7f162ab08640 1 -- 172.21.15.116:0/1709381486 >> 172.21.15.116:0/1709381486 conn(0x7f1624072010 msgr2=0x7f1624072420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:49.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.656+0000 7f162ab08640 1 -- 172.21.15.116:0/1709381486 shutdown_connections 2023-11-28T17:10:49.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.656+0000 7f162ab08640 1 -- 172.21.15.116:0/1709381486 wait complete. 2023-11-28T17:10:49.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.657+0000 7f162ab08640 1 Processor -- start 2023-11-28T17:10:49.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.658+0000 7f162ab08640 1 -- start start 2023-11-28T17:10:49.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.658+0000 7f162ab08640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f16240cdf70 0x7f1624159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:49.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.658+0000 7f162ab08640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16240d2f40 0x7f162415faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:49.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.658+0000 7f162ab08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f162415d2c0 0x7f162415ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:49.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.659+0000 7f162ab08640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1624159850 con 0x7f162415d2c0 2023-11-28T17:10:49.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.659+0000 7f162ab08640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16240ccd90 con 0x7f16240d2f40 2023-11-28T17:10:49.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.659+0000 7f162ab08640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16241f3690 con 0x7f16240cdf70 2023-11-28T17:10:49.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.659+0000 7f16290a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f162415d2c0 0x7f162415ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:49.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.659+0000 7f16290a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f162415d2c0 0x7f162415ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53390/0 (socket says 172.21.15.116:53390) 2023-11-28T17:10:49.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.659+0000 7f16290a5640 1 -- 172.21.15.116:0/1400280128 learned_addr learned my addr 172.21.15.116:0/1400280128 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:49.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.659+0000 7f1623fff640 1 --2- 172.21.15.116:0/1400280128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16240d2f40 0x7f162415faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:49.668 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.659+0000 7f16290a5640 1 -- 172.21.15.116:0/1400280128 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f16240cdf70 msgr2=0x7f1624159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:49.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.659+0000 7f16290a5640 1 --2- 172.21.15.116:0/1400280128 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f16240cdf70 0x7f1624159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:49.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.659+0000 7f16290a5640 1 -- 172.21.15.116:0/1400280128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16240d2f40 msgr2=0x7f162415faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:49.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.659+0000 7f16290a5640 1 --2- 172.21.15.116:0/1400280128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16240d2f40 0x7f162415faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:49.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.659+0000 7f16290a5640 1 -- 172.21.15.116:0/1400280128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1618009400 con 0x7f162415d2c0 2023-11-28T17:10:49.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.659+0000 7f1623fff640 1 --2- 172.21.15.116:0/1400280128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16240d2f40 0x7f162415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:49.669 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.659+0000 7f16290a5640 1 --2- 172.21.15.116:0/1400280128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f162415d2c0 0x7f162415ffe0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f161c00c780 tx=0x7f161c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:49.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.660+0000 7f16217fa640 1 -- 172.21.15.116:0/1400280128 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c00ce90 con 0x7f162415d2c0 2023-11-28T17:10:49.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.660+0000 7f16217fa640 1 -- 172.21.15.116:0/1400280128 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161c00a640 con 0x7f162415d2c0 2023-11-28T17:10:49.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.660+0000 7f16217fa640 1 -- 172.21.15.116:0/1400280128 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c00a9d0 con 0x7f162415d2c0 2023-11-28T17:10:49.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.660+0000 7f162ab08640 1 -- 172.21.15.116:0/1400280128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16241f3970 con 0x7f162415d2c0 2023-11-28T17:10:49.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.660+0000 7f162ab08640 1 -- 172.21.15.116:0/1400280128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16240c7080 con 0x7f162415d2c0 2023-11-28T17:10:49.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.661+0000 7f162ab08640 1 -- 172.21.15.116:0/1400280128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16240693d0 con 0x7f162415d2c0 2023-11-28T17:10:49.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.662+0000 7f16217fa640 1 -- 172.21.15.116:0/1400280128 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f161c00ab30 con 0x7f162415d2c0 2023-11-28T17:10:49.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.663+0000 7f16217fa640 1 --2- 172.21.15.116:0/1400280128 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f160806b2a0 0x7f160806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:49.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.663+0000 7f16217fa640 1 -- 172.21.15.116:0/1400280128 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f161c089110 con 0x7f162415d2c0 2023-11-28T17:10:49.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.663+0000 7f16288a4640 1 --2- 172.21.15.116:0/1400280128 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f160806b2a0 0x7f160806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:49.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.664+0000 7f16288a4640 1 --2- 172.21.15.116:0/1400280128 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f160806b2a0 0x7f160806d760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1618036000 tx=0x7f1618030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:49.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.667+0000 7f16217fa640 1 -- 172.21.15.116:0/1400280128 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f161c053f70 con 0x7f162415d2c0 2023-11-28T17:10:49.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.899+0000 7f162ab08640 1 -- 172.21.15.116:0/1400280128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f162406a4c0 con 0x7f162415d2c0 2023-11-28T17:10:49.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.900+0000 7f16217fa640 1 -- 172.21.15.116:0/1400280128 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7f161c018070 con 0x7f162415d2c0 2023-11-28T17:10:49.901 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T17:10:49.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.903+0000 7f162ab08640 1 -- 172.21.15.116:0/1400280128 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f160806b2a0 msgr2=0x7f160806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:49.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.903+0000 7f162ab08640 1 --2- 172.21.15.116:0/1400280128 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f160806b2a0 0x7f160806d760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1618036000 tx=0x7f1618030000 comp rx=0 tx=0).stop 2023-11-28T17:10:49.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.903+0000 7f162ab08640 1 -- 172.21.15.116:0/1400280128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f162415d2c0 msgr2=0x7f162415ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:49.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.903+0000 7f162ab08640 1 --2- 172.21.15.116:0/1400280128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f162415d2c0 0x7f162415ffe0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f161c00c780 tx=0x7f161c00cc40 comp rx=0 tx=0).stop 2023-11-28T17:10:49.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.903+0000 7f162ab08640 1 -- 172.21.15.116:0/1400280128 shutdown_connections 2023-11-28T17:10:49.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.903+0000 7f162ab08640 1 --2- 172.21.15.116:0/1400280128 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f160806b2a0 0x7f160806d760 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:49.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.903+0000 7f162ab08640 1 --2- 172.21.15.116:0/1400280128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f162415d2c0 0x7f162415ffe0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:49.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.903+0000 7f162ab08640 1 --2- 172.21.15.116:0/1400280128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16240d2f40 0x7f162415faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:49.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.903+0000 7f162ab08640 1 --2- 172.21.15.116:0/1400280128 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f16240cdf70 0x7f1624159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:49.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.903+0000 7f162ab08640 1 -- 172.21.15.116:0/1400280128 >> 172.21.15.116:0/1400280128 conn(0x7f1624072010 msgr2=0x7f16240b3990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:49.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.903+0000 7f162ab08640 1 -- 172.21.15.116:0/1400280128 shutdown_connections 2023-11-28T17:10:49.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:49.903+0000 7f162ab08640 1 -- 172.21.15.116:0/1400280128 wait complete. 2023-11-28T17:10:49.923 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T17:10:49.923 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T17:10:49.923 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2023-11-28T17:10:49.923 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T17:10:49.923 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:49.923 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T17:10:49.923 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T17:10:49.923 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T17:10:49.923 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T17:10:49.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:49.924 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2023-11-28T17:10:49.975 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T17:10:49.975 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T17:10:49.975 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T17:10:49.975 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:49.975 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2023-11-28T17:10:49.975 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:49.975 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:10:49.976 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:49.976 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2023-11-28T17:10:49.976 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:10:49.976 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:49.976 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2023-11-28T17:10:49.977 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:10:49.977 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:49.977 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T17:10:49.977 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:49.977 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T17:10:49.977 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42288sec preferred_lft 42288sec 2023-11-28T17:10:49.978 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2023-11-28T17:10:49.978 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:49.978 INFO:teuthology.orchestra.run.smithi116.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T17:10:49.978 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 66:db:2d:c1:d7:e7 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:10:49.978 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T17:10:49.979 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:49.979 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::c06c:35ff:fe15:dc8e/64 scope link 2023-11-28T17:10:49.979 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:49.979 INFO:teuthology.orchestra.run.smithi116.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T17:10:49.979 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 66:db:2d:c1:d7:e7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:10:49.980 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::64db:2dff:fec1:d7e7/64 scope link 2023-11-28T17:10:49.980 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:49.980 INFO:teuthology.orchestra.run.smithi116.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T17:10:49.980 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether c2:79:c2:c8:b7:18 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:10:49.980 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::c079:c2ff:fec8:b718/64 scope link 2023-11-28T17:10:49.980 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:10:49.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:49.980 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T17:10:50.070 INFO:teuthology.orchestra.run.smithi116.stdout:1 2023-11-28T17:10:50.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:50.075 DEBUG:teuthology.orchestra.run.smithi116:> route 2023-11-28T17:10:50.132 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2023-11-28T17:10:50.132 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T17:10:50.133 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T17:10:50.133 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T17:10:50.133 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T17:10:50.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:50.134 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-28T17:10:50.134 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-28T17:10:50.134 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T17:10:50.134 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T17:10:50.134 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T17:10:50.134 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-28T17:10:50.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:50.314 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2023-11-28T17:10:50.364 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T17:10:50.364 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T17:10:50.365 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T17:10:50.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:50.365 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2023-11-28T17:10:50.458 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-11-28T17:10:50.458 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:10:50.458 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:50.514 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T17:10:50.514 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:10:50.514 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:50.567 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-28T17:10:50.663 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:10:50.720 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2023-11-28T17:10:50.720 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2023-11-28T17:10:50.721 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2023-11-28T17:10:50.721 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2023-11-28T17:10:50.721 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2023-11-28T17:10:50.721 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2023-11-28T17:10:50.721 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2023-11-28T17:10:50.721 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2023-11-28T17:10:50.722 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2023-11-28T17:10:50.722 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2023-11-28T17:10:50.722 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2023-11-28T17:10:50.722 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:10:50.722 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2023-11-28T17:10:50.723 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2023-11-28T17:10:50.723 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2023-11-28T17:10:50.723 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2023-11-28T17:10:50.723 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2023-11-28T17:10:50.723 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2023-11-28T17:10:50.723 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:10:50.723 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:10:50.723 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:10:50.723 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:10:50.724 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2023-11-28T17:10:50.724 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2023-11-28T17:10:50.724 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T17:10:50.724 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T17:10:50.724 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T17:10:50.724 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T17:10:50.724 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2023-11-28T17:10:50.724 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:10:50.724 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:10:50.724 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:10:50.724 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:10:50.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:50.725 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-28T17:10:50.775 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-28T17:10:50.775 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-11-28T17:10:50.781 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-28T17:10:50.862 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:10:50.908 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2023-11-28T17:10:50.908 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2023-11-28T17:10:50.908 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2023-11-28T17:10:50.908 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2023-11-28T17:10:50.908 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2023-11-28T17:10:50.908 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2023-11-28T17:10:50.908 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2023-11-28T17:10:50.908 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2023-11-28T17:10:50.908 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2023-11-28T17:10:50.909 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2023-11-28T17:10:50.909 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2023-11-28T17:10:50.909 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:10:50.909 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2023-11-28T17:10:50.909 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2023-11-28T17:10:50.909 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2023-11-28T17:10:50.909 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2023-11-28T17:10:50.910 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2023-11-28T17:10:50.910 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2023-11-28T17:10:50.910 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:10:50.910 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:10:50.910 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:10:50.910 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:10:50.910 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2023-11-28T17:10:50.910 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2023-11-28T17:10:50.910 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T17:10:50.911 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T17:10:50.911 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T17:10:50.911 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T17:10:50.911 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2023-11-28T17:10:50.911 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:10:50.911 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:10:50.911 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:10:50.911 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:10:50.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:50.913 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-28T17:10:50.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:2023-11-28T17:10:50.968+0000 7fd0eac64480 -1 init, newargv = 0x5560f6589750 newargc=15 2023-11-28T17:10:50.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[104085]: starting ceph client 2023-11-28T17:10:50.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[104085]: starting fuse 2023-11-28T17:10:50.999 INFO:teuthology.orchestra.run.smithi116.stdout:39 2023-11-28T17:10:50.999 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2023-11-28T17:10:50.999 DEBUG:teuthology.orchestra.run.smithi116:> sudo stdin-killer -- python3 -c ' 2023-11-28T17:10:50.999 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2023-11-28T17:10:51.000 DEBUG:teuthology.orchestra.run.smithi116:> import re 2023-11-28T17:10:51.000 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-28T17:10:51.000 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2023-11-28T17:10:51.000 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:10:51.000 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2023-11-28T17:10:51.000 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-11-28T17:10:51.000 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2023-11-28T17:10:51.000 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-11-28T17:10:51.000 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:10:51.000 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2023-11-28T17:10:51.000 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2023-11-28T17:10:51.000 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2023-11-28T17:10:51.000 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2023-11-28T17:10:51.001 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:10:51.001 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2023-11-28T17:10:51.001 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-28T17:10:51.001 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-28T17:10:51.001 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-28T17:10:51.001 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2023-11-28T17:10:51.001 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2023-11-28T17:10:51.001 DEBUG:teuthology.orchestra.run.smithi116:> return f 2023-11-28T17:10:51.001 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-28T17:10:51.001 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:10:51.001 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.0")) 2023-11-28T17:10:51.001 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-28T17:10:51.209 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.0.104085.asok 2023-11-28T17:10:51.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-11-28T17:10:51.227 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.104085.asok 2023-11-28T17:10:51.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:51.229 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.104085.asok status 2023-11-28T17:10:51.470 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-28T17:10:51.470 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2023-11-28T17:10:51.470 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "47c1c547372ed5bff7984647dc200bac0e019caf", 2023-11-28T17:10:51.470 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev)", 2023-11-28T17:10:51.470 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "0", 2023-11-28T17:10:51.471 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2023-11-28T17:10:51.471 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-11-28T17:10:51.471 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "104085", 2023-11-28T17:10:51.471 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2023-11-28T17:10:51.471 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:51.471 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2023-11-28T17:10:51.471 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2023-11-28T17:10:51.471 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 5577, 2023-11-28T17:10:51.471 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2023-11-28T17:10:51.471 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2023-11-28T17:10:51.471 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2023-11-28T17:10:51.471 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5577 2023-11-28T17:10:51.471 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:51.472 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-28T17:10:51.472 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-28T17:10:51.472 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-28T17:10:51.472 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1091306782 2023-11-28T17:10:51.472 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:10:51.472 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:51.472 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-28T17:10:51.472 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-28T17:10:51.472 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-28T17:10:51.472 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1091306782 2023-11-28T17:10:51.472 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:10:51.472 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.5577 192.168.0.1:0/1091306782", 2023-11-28T17:10:51.472 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.1:0/1091306782", 2023-11-28T17:10:51.473 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2023-11-28T17:10:51.473 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 15, 2023-11-28T17:10:51.473 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 36, 2023-11-28T17:10:51.473 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2023-11-28T17:10:51.473 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2023-11-28T17:10:51.473 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2023-11-28T17:10:51.473 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-28T17:10:51.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:51.485 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2023-11-28T17:10:51.486 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:51.543 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2023-11-28T17:10:51.543 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:51.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:51.543 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:51.640 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2023-11-28T17:10:51.870 INFO:teuthology.orchestra.run.smithi116.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":85,"msgr_send_messages":70,"msgr_recv_bytes":122112,"msgr_send_bytes":38095,"msgr_created_connections":9,"msgr_active_connections":2,"msgr_running_total_time":0.022907441,"msgr_running_send_time":0.005683842,"msgr_running_recv_time":0.016449289,"msgr_running_fast_dispatch_time":0.001285442,"msgr_send_messages_queue_lat":{"avgcount":70,"sum":0.066585909,"avgtime":0.000951227},"msgr_handle_ack_lat":{"avgcount":29,"sum":0.000005899,"avgtime":0.000000203},"msgr_recv_encrypted_bytes":115040,"msgr_send_encrypted_bytes":14560},"AsyncMessenger::Worker-1":{"msgr_recv_messages":34,"msgr_send_messages":34,"msgr_recv_bytes":10239,"msgr_send_bytes":115583,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":0.011658827,"msgr_running_send_time":0.002667542,"msgr_running_recv_time":0.007476507,"msgr_running_fast_dispatch_time":0.001150231,"msgr_send_messages_queue_lat":{"avgcount":33,"sum":0.171378345,"avgtime":0.005193283},"msgr_handle_ack_lat":{"avgcount":19,"sum":0.000005149,"avgtime":0.000000271},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":100800},"AsyncMessenger::Worker-2":{"msgr_recv_messages":27,"msgr_send_messages":28,"msgr_recv_bytes":6895,"msgr_send_bytes":14972,"msgr_created_connections":5,"msgr_active_connections":3,"msgr_running_total_time":0.005654236,"msgr_running_send_time":0.000781763,"msgr_running_recv_time":0.003623646,"msgr_running_fast_dispatch_time":0.000865621,"msgr_send_messages_queue_lat":{"avgcount":27,"sum":0.085110545,"avgtime":0.003152242},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":47,"sum":0.003656549,"avgtime":0.000077798}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2,"sum":0.000020594,"avgtime":0.000010297}},"mds":{"request":4,"reply":4,"reply_latency":{"avgcount":4,"sum":0.001170998,"avgtime":0.000292749},"slow_reply":0,"forward":0,"dir_fetch_complete":0,"dir_fetch_keys":0,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":10,"inodes_top":0,"inodes_bottom":10,"inodes_pin_tail":0,"inodes_pinned":10,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":7,"traverse_hit":7,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":44,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":0,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":0,"handle_client_caps_dirty":0,"handle_client_cap_release":0,"process_request_cap_release":1},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":8,"evex":0,"evtrm":0,"ev":8,"evexg":0,"evexd":0,"segadd":2,"segex":0,"segtrm":0,"seg":2,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4197663,"rdpos":0,"jlat":{"avgcount":7,"sum":0.026253464,"avgtime":0.003750494},"replayed":0},"mds_mem":{"ino":13,"ino+":13,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":10,"dn+":10,"dn-":0,"cap":1,"cap+":3,"cap-":2,"rss":39296,"heap":207196},"mds_server":{"dispatch_client_request":4,"dispatch_server_request":0,"handle_client_request":4,"handle_client_session":10,"handle_peer_request":0,"req_create_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getattr_latency":{"avgcount":3,"sum":0.000807047,"avgtime":0.000269015},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":1,"sum":0.000363951,"avgtime":0.000363951},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":3,"session_remove":2,"sessions_open":1,"sessions_stale":0,"total_load":3,"average_load":3,"avg_session_uptime":8,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":213331,"buffer_anon_items":58,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5696,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":51520,"mds_co_items":211,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":46,"op_send_bytes":14569,"op_resend":31,"op_reply":46,"op_latency":{"avgcount":46,"sum":0.958984078,"avgtime":0.020847479},"op_inflight":0,"oplen_avg":{"avgcount":46,"sum":70},"op":46,"op_r":0,"op_w":46,"op_rmw":0,"op_pg":0,"osdop_stat":0,"osdop_create":11,"osdop_read":0,"osdop_write":7,"osdop_writefull":8,"osdop_writesame":0,"osdop_append":0,"osdop_zero":0,"osdop_truncate":0,"osdop_delete":6,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":0,"osdop_setxattr":22,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":36,"map_full":0,"map_inc":3,"osd_sessions":6,"osd_session_open":6,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":0,"omap_del":1},"oft":{"omap_total_objs":1,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":140,"get_sum":122093,"get_or_fail_fail":0,"get_or_fail_success":140,"take":0,"take_sum":0,"put":140,"put_sum":122093,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":46,"take_sum":14569,"put":46,"put_sum":14569,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":46,"take_sum":46,"put":46,"put_sum":46,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x561658c2c6e0":{"val":0,"max":3758096384,"get_started":0,"get":8,"get_sum":3359,"get_or_fail_fail":0,"get_or_fail_success":8,"take":0,"take_sum":0,"put":7,"put_sum":3359,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2023-11-28T17:10:51.872 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 2, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 29, "avgtime": 2.03e-07, "sum": 5.899e-06 }, "msgr_recv_bytes": 122112, "msgr_recv_encrypted_bytes": 115040, "msgr_recv_messages": 85, "msgr_running_fast_dispatch_time": 0.001285442, "msgr_running_recv_time": 0.016449289, "msgr_running_send_time": 0.005683842, "msgr_running_total_time": 0.022907441, "msgr_send_bytes": 38095, "msgr_send_encrypted_bytes": 14560, "msgr_send_messages": 70, "msgr_send_messages_queue_lat": { "avgcount": 70, "avgtime": 0.000951227, "sum": 0.066585909 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 19, "avgtime": 2.71e-07, "sum": 5.149e-06 }, "msgr_recv_bytes": 10239, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 34, "msgr_running_fast_dispatch_time": 0.001150231, "msgr_running_recv_time": 0.007476507, "msgr_running_send_time": 0.002667542, "msgr_running_total_time": 0.011658827, "msgr_send_bytes": 115583, "msgr_send_encrypted_bytes": 100800, "msgr_send_messages": 34, "msgr_send_messages_queue_lat": { "avgcount": 33, "avgtime": 0.005193283, "sum": 0.171378345 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 6895, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 27, "msgr_running_fast_dispatch_time": 0.000865621, "msgr_running_recv_time": 0.003623646, "msgr_running_send_time": 0.000781763, "msgr_running_total_time": 0.005654236, "msgr_send_bytes": 14972, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 28, "msgr_send_messages_queue_lat": { "avgcount": 27, "avgtime": 0.003152242, "sum": 0.085110545 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 47, "avgtime": 7.7798e-05, "sum": 0.003656549 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2, "avgtime": 1.0297e-05, "sum": 2.0594e-05 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 0, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 0, "dir_fetch_keys": 0, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 0, "handle_client_caps_dirty": 0, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 10, "inodes_bottom": 10, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 10, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 44, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1, "q": 0, "reply": 4, "reply_latency": { "avgcount": 4, "avgtime": 0.000292749, "sum": 0.001170998 }, "request": 4, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 7, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 7, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 8, "evadd": 8, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 7, "avgtime": 0.003750494, "sum": 0.026253464 }, "rdpos": 0, "replayed": 0, "seg": 2, "segadd": 2, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 4197663 }, "mds_mem": { "cap": 1, "cap+": 3, "cap-": 2, "dir": 12, "dir+": 12, "dir-": 0, "dn": 10, "dn+": 10, "dn-": 0, "heap": 207196, "ino": 13, "ino+": 13, "ino-": 0, "rss": 39296 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4, "dispatch_server_request": 0, "handle_client_request": 4, "handle_client_session": 10, "handle_peer_request": 0, "req_create_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getattr_latency": { "avgcount": 3, "avgtime": 0.000269015, "sum": 0.000807047 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 1, "avgtime": 0.000363951, "sum": 0.000363951 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 3, "avg_session_uptime": 8, "mdthresh_evicted": 0, "session_add": 3, "session_count": 1, "session_remove": 2, "sessions_open": 1, "sessions_stale": 0, "total_load": 3 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 213331, "buffer_anon_items": 58, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 51520, "mds_co_items": 211, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5696, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 36, "map_full": 0, "map_inc": 3, "omap_del": 1, "omap_rd": 0, "omap_wr": 15, "op": 46, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 46, "avgtime": 0.020847479, "sum": 0.958984078 }, "op_pg": 0, "op_r": 0, "op_reply": 46, "op_resend": 31, "op_rmw": 0, "op_send": 46, "op_send_bytes": 14569, "op_w": 46, "oplen_avg": { "avgcount": 46, "sum": 70 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 6, "osd_sessions": 6, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 11, "osdop_delete": 6, "osdop_getxattr": 0, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 0, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 22, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 0, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 7, "osdop_writefull": 8, "osdop_writesame": 0, "osdop_zero": 0, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 1, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 140, "get_or_fail_fail": 0, "get_or_fail_success": 140, "get_started": 0, "get_sum": 122093, "max": 104857600, "put": 140, "put_sum": 122093, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 46, "put_sum": 14569, "take": 46, "take_sum": 14569, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 46, "put_sum": 46, "take": 46, "take_sum": 46, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x561658c2c6e0": { "get": 8, "get_or_fail_fail": 0, "get_or_fail_success": 8, "get_started": 0, "get_sum": 3359, "max": 3758096384, "put": 7, "put_sum": 3359, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2023-11-28T17:10:51.873 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:10:51.873 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:10:51.928 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:10:52.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.143+0000 7f11ebfff640 1 Processor -- start 2023-11-28T17:10:52.145 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.143+0000 7f11ebfff640 1 -- start start 2023-11-28T17:10:52.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.145+0000 7f11ebfff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f11ec15d260 0x7f11ec0d16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:52.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.145+0000 7f11ebfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f11ec15d640 0x7f11ec0afb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:52.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.145+0000 7f11ebfff640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0d2000 con 0x7f11ec161c00 2023-11-28T17:10:52.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.145+0000 7f11ebfff640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11ec0ce120 con 0x7f11ec15d640 2023-11-28T17:10:52.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.145+0000 7f11ebfff640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11ec0ce400 con 0x7f11ec15d260 2023-11-28T17:10:52.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.145+0000 7f11e3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f11ec15d640 0x7f11ec0afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:52.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.146+0000 7f11e3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f11ec15d640 0x7f11ec0afb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53400/0 (socket says 172.21.15.116:53400) 2023-11-28T17:10:52.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.146+0000 7f11e3fff640 1 -- 172.21.15.116:0/500370303 learned_addr learned my addr 172.21.15.116:0/500370303 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:52.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.146+0000 7f11e3fff640 1 -- 172.21.15.116:0/500370303 >> v2:172.21.15.156:3300/0 conn(0x7f11ec15d260 msgr2=0x7f11ec0d16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:52.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.146+0000 7f11eaffd640 1 --2- 172.21.15.116:0/500370303 >> v2:172.21.15.156:3300/0 conn(0x7f11ec15d260 0x7f11ec0d16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:52.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.146+0000 7f11e3fff640 1 --2- 172.21.15.116:0/500370303 >> v2:172.21.15.156:3300/0 conn(0x7f11ec15d260 0x7f11ec0d16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.146+0000 7f11e3fff640 1 -- 172.21.15.116:0/500370303 >> v1:172.21.15.156:6789/0 conn(0x7f11ec161c00 legacy=0x7f11ec160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:52.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.146+0000 7f11e3fff640 1 -- 172.21.15.116:0/500370303 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ec0ce6e0 con 0x7f11ec15d640 2023-11-28T17:10:52.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.146+0000 7f11eaffd640 1 --2- 172.21.15.116:0/500370303 >> v2:172.21.15.156:3300/0 conn(0x7f11ec15d260 0x7f11ec0d16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:52.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.147+0000 7f11e3fff640 1 --2- 172.21.15.116:0/500370303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec15d640 0x7f11ec0afb20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f11ec08b230 tx=0x7f11d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7fad48b494840db server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:52.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.147+0000 7f11ea7fc640 1 -- 172.21.15.116:0/500370303 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d4031870 con 0x7f11ec15d640 2023-11-28T17:10:52.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.147+0000 7f11ea7fc640 1 -- 172.21.15.116:0/500370303 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d40319d0 con 0x7f11ec15d640 2023-11-28T17:10:52.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.147+0000 7f11ea7fc640 1 -- 172.21.15.116:0/500370303 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d4031d60 con 0x7f11ec15d640 2023-11-28T17:10:52.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.148+0000 7f11ebfff640 1 -- 172.21.15.116:0/500370303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec15d640 msgr2=0x7f11ec0afb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:52.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.148+0000 7f11ebfff640 1 --2- 172.21.15.116:0/500370303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec15d640 0x7f11ec0afb20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f11ec08b230 tx=0x7f11d402b910 comp rx=0 tx=0).stop 2023-11-28T17:10:52.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.148+0000 7f11ebfff640 1 -- 172.21.15.116:0/500370303 shutdown_connections 2023-11-28T17:10:52.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.148+0000 7f11ebfff640 1 --2- 172.21.15.116:0/500370303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec15d640 0x7f11ec0afb20 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.148+0000 7f11ebfff640 1 --2- 172.21.15.116:0/500370303 >> v2:172.21.15.156:3300/0 conn(0x7f11ec15d260 0x7f11ec0d16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.148+0000 7f11ebfff640 1 -- 172.21.15.116:0/500370303 >> 172.21.15.116:0/500370303 conn(0x7f11ec0d2f50 msgr2=0x7f11ec0d3360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:52.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.148+0000 7f11ebfff640 1 -- 172.21.15.116:0/500370303 shutdown_connections 2023-11-28T17:10:52.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.148+0000 7f11ebfff640 1 -- 172.21.15.116:0/500370303 wait complete. 2023-11-28T17:10:52.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.150+0000 7f11ebfff640 1 Processor -- start 2023-11-28T17:10:52.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.150+0000 7f11ebfff640 1 -- start start 2023-11-28T17:10:52.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.151+0000 7f11ebfff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11ec15d260 0x7f11ec14b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:52.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.151+0000 7f11ebfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11ec15d640 0x7f11ec15ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:52.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.151+0000 7f11ebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec161c00 0x7f11ec15f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:52.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.151+0000 7f11ebfff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ec0b2480 con 0x7f11ec161c00 2023-11-28T17:10:52.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.151+0000 7f11ebfff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ec0c85f0 con 0x7f11ec15d640 2023-11-28T17:10:52.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.151+0000 7f11ebfff640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11ec0c79e0 con 0x7f11ec15d260 2023-11-28T17:10:52.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.151+0000 7f11eb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec161c00 0x7f11ec15f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:52.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.151+0000 7f11eb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec161c00 0x7f11ec15f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53414/0 (socket says 172.21.15.116:53414) 2023-11-28T17:10:52.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.151+0000 7f11eb7fe640 1 -- 172.21.15.116:0/985884362 learned_addr learned my addr 172.21.15.116:0/985884362 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.152+0000 7f11eaffd640 1 --2- 172.21.15.116:0/985884362 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11ec15d260 0x7f11ec14b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.152+0000 7f11e3fff640 1 --2- 172.21.15.116:0/985884362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11ec15d640 0x7f11ec15ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.152+0000 7f11eb7fe640 1 -- 172.21.15.116:0/985884362 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11ec15d260 msgr2=0x7f11ec14b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.152+0000 7f11eb7fe640 1 --2- 172.21.15.116:0/985884362 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11ec15d260 0x7f11ec14b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.152+0000 7f11eb7fe640 1 -- 172.21.15.116:0/985884362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11ec15d640 msgr2=0x7f11ec15ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.152+0000 7f11eb7fe640 1 --2- 172.21.15.116:0/985884362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11ec15d640 0x7f11ec15ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.152+0000 7f11eb7fe640 1 -- 172.21.15.116:0/985884362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d40094d0 con 0x7f11ec161c00 2023-11-28T17:10:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.152+0000 7f11eb7fe640 1 --2- 172.21.15.116:0/985884362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec161c00 0x7f11ec15f4c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f11e400b970 tx=0x7f11e400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.152+0000 7f11e37fe640 1 -- 172.21.15.116:0/985884362 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e400cb40 con 0x7f11ec161c00 2023-11-28T17:10:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.152+0000 7f11e37fe640 1 -- 172.21.15.116:0/985884362 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e400cca0 con 0x7f11ec161c00 2023-11-28T17:10:52.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.152+0000 7f11e37fe640 1 -- 172.21.15.116:0/985884362 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e4014680 con 0x7f11ec161c00 2023-11-28T17:10:52.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.152+0000 7f11ebfff640 1 -- 172.21.15.116:0/985884362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ec0c7cc0 con 0x7f11ec161c00 2023-11-28T17:10:52.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.153+0000 7f11ebfff640 1 -- 172.21.15.116:0/985884362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11ec0c8280 con 0x7f11ec161c00 2023-11-28T17:10:52.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.154+0000 7f11ebfff640 1 -- 172.21.15.116:0/985884362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11ec0c8760 con 0x7f11ec161c00 2023-11-28T17:10:52.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.154+0000 7f11e37fe640 1 -- 172.21.15.116:0/985884362 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f11e401d3b0 con 0x7f11ec161c00 2023-11-28T17:10:52.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.155+0000 7f11e37fe640 1 --2- 172.21.15.116:0/985884362 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f11cc06b2d0 0x7f11cc06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:52.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.155+0000 7f11e37fe640 1 -- 172.21.15.116:0/985884362 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f11e4089ab0 con 0x7f11ec161c00 2023-11-28T17:10:52.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.156+0000 7f11eaffd640 1 --2- 172.21.15.116:0/985884362 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f11cc06b2d0 0x7f11cc06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:52.157 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.156+0000 7f11eaffd640 1 --2- 172.21.15.116:0/985884362 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f11cc06b2d0 0x7f11cc06d790 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00a990 tx=0x7f11dc008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.160+0000 7f11e37fe640 1 -- 172.21.15.116:0/985884362 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f11e4054800 con 0x7f11ec161c00 2023-11-28T17:10:52.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.355+0000 7f11ebfff640 1 -- 172.21.15.116:0/985884362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11ec0a0ea0 con 0x7f11ec161c00 2023-11-28T17:10:52.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.358+0000 7f11e37fe640 1 -- 172.21.15.116:0/985884362 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16770 (secure 0 0 0) 0x7f11e4057e20 con 0x7f11ec161c00 2023-11-28T17:10:52.359 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:52.360 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":36,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:10:44.069312+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:10:17.338444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-28T17:10:18.766338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.1d","mappings":[{"from":4,"to":5}]},{"pgid":"5.37","mappings":[{"from":4,"to":5}]},{"pgid":"5.3e","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:10:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.360+0000 7f11ebfff640 1 -- 172.21.15.116:0/985884362 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f11cc06b2d0 msgr2=0x7f11cc06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.360+0000 7f11ebfff640 1 --2- 172.21.15.116:0/985884362 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f11cc06b2d0 0x7f11cc06d790 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00a990 tx=0x7f11dc008040 comp rx=0 tx=0).stop 2023-11-28T17:10:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.360+0000 7f11ebfff640 1 -- 172.21.15.116:0/985884362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec161c00 msgr2=0x7f11ec15f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.360+0000 7f11ebfff640 1 --2- 172.21.15.116:0/985884362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec161c00 0x7f11ec15f4c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f11e400b970 tx=0x7f11e400be30 comp rx=0 tx=0).stop 2023-11-28T17:10:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.360+0000 7f11ebfff640 1 -- 172.21.15.116:0/985884362 shutdown_connections 2023-11-28T17:10:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.360+0000 7f11ebfff640 1 --2- 172.21.15.116:0/985884362 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f11cc06b2d0 0x7f11cc06d790 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.362 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.361+0000 7f11ebfff640 1 --2- 172.21.15.116:0/985884362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec161c00 0x7f11ec15f4c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.361+0000 7f11ebfff640 1 --2- 172.21.15.116:0/985884362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11ec15d640 0x7f11ec15ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.361+0000 7f11ebfff640 1 --2- 172.21.15.116:0/985884362 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11ec15d260 0x7f11ec14b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.361+0000 7f11ebfff640 1 -- 172.21.15.116:0/985884362 >> 172.21.15.116:0/985884362 conn(0x7f11ec0d2f50 msgr2=0x7f11ec0cf390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.361+0000 7f11ebfff640 1 -- 172.21.15.116:0/985884362 shutdown_connections 2023-11-28T17:10:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.361+0000 7f11ebfff640 1 -- 172.21.15.116:0/985884362 wait complete. 2023-11-28T17:10:52.381 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:10:17.338444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:10:18.766338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:10:52.381 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:10:52.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.594+0000 7f14961a9640 1 Processor -- start 2023-11-28T17:10:52.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.594+0000 7f14961a9640 1 -- start start 2023-11-28T17:10:52.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.596+0000 7f14961a9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f149013cee0 0x7f149013d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:52.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.596+0000 7f14961a9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f149013f640 0x7f149013fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:52.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.596+0000 7f14961a9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f149013d2c0 con 0x7f1490071d20 2023-11-28T17:10:52.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.596+0000 7f14961a9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14901400e0 con 0x7f149013f640 2023-11-28T17:10:52.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.596+0000 7f14961a9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14901403c0 con 0x7f149013cee0 2023-11-28T17:10:52.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.596+0000 7f148ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1490071d20 0x7f1490072130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34442/0 (socket says 172.21.15.116:34442) 2023-11-28T17:10:52.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.596+0000 7f148ffff640 1 -- 172.21.15.116:0/2347019076 learned_addr learned my addr 172.21.15.116:0/2347019076 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:52.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.596+0000 7f148effd640 1 --2- 172.21.15.116:0/2347019076 >> v2:172.21.15.116:3300/0 conn(0x7f149013f640 0x7f149013fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:52.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.596+0000 7f148f7fe640 1 --2- 172.21.15.116:0/2347019076 >> v2:172.21.15.156:3300/0 conn(0x7f149013cee0 0x7f149013d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:52.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.597+0000 7f148effd640 1 -- 172.21.15.116:0/2347019076 >> v2:172.21.15.156:3300/0 conn(0x7f149013cee0 msgr2=0x7f149013d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:52.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.597+0000 7f148effd640 1 --2- 172.21.15.116:0/2347019076 >> v2:172.21.15.156:3300/0 conn(0x7f149013cee0 0x7f149013d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.597+0000 7f148effd640 1 -- 172.21.15.116:0/2347019076 >> v1:172.21.15.116:6789/0 conn(0x7f1490071d20 legacy=0x7f1490072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:52.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.597+0000 7f148effd640 1 -- 172.21.15.116:0/2347019076 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14900cdb50 con 0x7f149013f640 2023-11-28T17:10:52.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.598+0000 7f148effd640 1 --2- 172.21.15.116:0/2347019076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f149013f640 0x7f149013fab0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f149006b470 tx=0x7f148402b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625679dead5dc2ee server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.598+0000 7f148e7fc640 1 -- 172.21.15.116:0/2347019076 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1484031520 con 0x7f149013f640 2023-11-28T17:10:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.598+0000 7f148e7fc640 1 -- 172.21.15.116:0/2347019076 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1484031680 con 0x7f149013f640 2023-11-28T17:10:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.598+0000 7f148e7fc640 1 -- 172.21.15.116:0/2347019076 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1484031a10 con 0x7f149013f640 2023-11-28T17:10:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.599+0000 7f14961a9640 1 -- 172.21.15.116:0/2347019076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f149013f640 msgr2=0x7f149013fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.599+0000 7f14961a9640 1 --2- 172.21.15.116:0/2347019076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f149013f640 0x7f149013fab0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f149006b470 tx=0x7f148402b7c0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.599+0000 7f14961a9640 1 -- 172.21.15.116:0/2347019076 shutdown_connections 2023-11-28T17:10:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.599+0000 7f14961a9640 1 --2- 172.21.15.116:0/2347019076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f149013f640 0x7f149013fab0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.599+0000 7f14961a9640 1 --2- 172.21.15.116:0/2347019076 >> v2:172.21.15.156:3300/0 conn(0x7f149013cee0 0x7f149013d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.599+0000 7f14961a9640 1 -- 172.21.15.116:0/2347019076 >> 172.21.15.116:0/2347019076 conn(0x7f14900b0cf0 msgr2=0x7f14900b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.599+0000 7f14961a9640 1 -- 172.21.15.116:0/2347019076 shutdown_connections 2023-11-28T17:10:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.599+0000 7f14961a9640 1 -- 172.21.15.116:0/2347019076 wait complete. 2023-11-28T17:10:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.601+0000 7f14961a9640 1 Processor -- start 2023-11-28T17:10:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.601+0000 7f14961a9640 1 -- start start 2023-11-28T17:10:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f14961a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1490071d20 0x7f1490140730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f14961a9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f149013cee0 0x7f1490158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f14961a9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f149013f640 0x7f14901594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f14961a9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149013a9e0 con 0x7f1490071d20 2023-11-28T17:10:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f14961a9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149013b800 con 0x7f149013cee0 2023-11-28T17:10:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f14961a9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149013abf0 con 0x7f149013f640 2023-11-28T17:10:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f148f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1490071d20 0x7f1490140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f148f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1490071d20 0x7f1490140730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53440/0 (socket says 172.21.15.116:53440) 2023-11-28T17:10:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f148f7fe640 1 -- 172.21.15.116:0/3307972357 learned_addr learned my addr 172.21.15.116:0/3307972357 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f148effd640 1 --2- 172.21.15.116:0/3307972357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f149013cee0 0x7f1490158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f148f7fe640 1 -- 172.21.15.116:0/3307972357 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f149013f640 msgr2=0x7f14901594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f148f7fe640 1 --2- 172.21.15.116:0/3307972357 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f149013f640 0x7f14901594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f148f7fe640 1 -- 172.21.15.116:0/3307972357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f149013cee0 msgr2=0x7f1490158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f148f7fe640 1 --2- 172.21.15.116:0/3307972357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f149013cee0 0x7f1490158f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.602+0000 7f148f7fe640 1 -- 172.21.15.116:0/3307972357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1484009400 con 0x7f1490071d20 2023-11-28T17:10:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.603+0000 7f148f7fe640 1 --2- 172.21.15.116:0/3307972357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1490071d20 0x7f1490140730 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f148000d800 tx=0x7f148000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.603+0000 7f1477fff640 1 -- 172.21.15.116:0/3307972357 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148000b6c0 con 0x7f1490071d20 2023-11-28T17:10:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.603+0000 7f1477fff640 1 -- 172.21.15.116:0/3307972357 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148000b820 con 0x7f1490071d20 2023-11-28T17:10:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.603+0000 7f1477fff640 1 -- 172.21.15.116:0/3307972357 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148000bbb0 con 0x7f1490071d20 2023-11-28T17:10:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.603+0000 7f14961a9640 1 -- 172.21.15.116:0/3307972357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149013aed0 con 0x7f1490071d20 2023-11-28T17:10:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.603+0000 7f14961a9640 1 -- 172.21.15.116:0/3307972357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149013b490 con 0x7f1490071d20 2023-11-28T17:10:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.604+0000 7f14961a9640 1 -- 172.21.15.116:0/3307972357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14901549b0 con 0x7f1490071d20 2023-11-28T17:10:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.605+0000 7f1477fff640 1 -- 172.21.15.116:0/3307972357 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f148000bd10 con 0x7f1490071d20 2023-11-28T17:10:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.606+0000 7f1477fff640 1 --2- 172.21.15.116:0/3307972357 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f147c06b2a0 0x7f147c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.606+0000 7f1477fff640 1 -- 172.21.15.116:0/3307972357 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f148008a290 con 0x7f1490071d20 2023-11-28T17:10:52.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.606+0000 7f148effd640 1 --2- 172.21.15.116:0/3307972357 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f147c06b2a0 0x7f147c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:52.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.607+0000 7f148effd640 1 --2- 172.21.15.116:0/3307972357 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f147c06b2a0 0x7f147c06d760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1484035000 tx=0x7f1484030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:52.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.610+0000 7f1477fff640 1 -- 172.21.15.116:0/3307972357 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1480054fe0 con 0x7f1490071d20 2023-11-28T17:10:52.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.800+0000 7f14961a9640 1 -- 172.21.15.116:0/3307972357 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1490069360 con 0x7f1490071d20 2023-11-28T17:10:52.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.801+0000 7f1477fff640 1 -- 172.21.15.116:0/3307972357 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1480058600 con 0x7f1490071d20 2023-11-28T17:10:52.802 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:10:52.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.804+0000 7f14961a9640 1 -- 172.21.15.116:0/3307972357 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f147c06b2a0 msgr2=0x7f147c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:52.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.804+0000 7f14961a9640 1 --2- 172.21.15.116:0/3307972357 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f147c06b2a0 0x7f147c06d760 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1484035000 tx=0x7f1484030000 comp rx=0 tx=0).stop 2023-11-28T17:10:52.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.804+0000 7f14961a9640 1 -- 172.21.15.116:0/3307972357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1490071d20 msgr2=0x7f1490140730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:52.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.804+0000 7f14961a9640 1 --2- 172.21.15.116:0/3307972357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1490071d20 0x7f1490140730 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f148000d800 tx=0x7f148000dcc0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.804+0000 7f14961a9640 1 -- 172.21.15.116:0/3307972357 shutdown_connections 2023-11-28T17:10:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.804+0000 7f14961a9640 1 --2- 172.21.15.116:0/3307972357 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f147c06b2a0 0x7f147c06d760 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.804+0000 7f14961a9640 1 --2- 172.21.15.116:0/3307972357 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f149013f640 0x7f14901594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.804+0000 7f14961a9640 1 --2- 172.21.15.116:0/3307972357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f149013cee0 0x7f1490158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.804+0000 7f14961a9640 1 --2- 172.21.15.116:0/3307972357 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1490071d20 0x7f1490140730 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.804+0000 7f14961a9640 1 -- 172.21.15.116:0/3307972357 >> 172.21.15.116:0/3307972357 conn(0x7f14900b0cf0 msgr2=0x7f14900ce250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.804+0000 7f14961a9640 1 -- 172.21.15.116:0/3307972357 shutdown_connections 2023-11-28T17:10:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:52.804+0000 7f14961a9640 1 -- 172.21.15.116:0/3307972357 wait complete. 2023-11-28T17:10:52.820 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:10:52.822 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:10:52.826 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:10:53.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.020+0000 7f966aee7640 1 Processor -- start 2023-11-28T17:10:53.022 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.020+0000 7f966aee7640 1 -- start start 2023-11-28T17:10:53.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.022+0000 7f966aee7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f966415d420 0x7f96640b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.022+0000 7f966aee7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f96640b5920 0x7f96640b6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.022+0000 7f966aee7640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966415d800 con 0x7f96640d2710 2023-11-28T17:10:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.022+0000 7f966aee7640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96640d2fa0 con 0x7f96640b5920 2023-11-28T17:10:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.022+0000 7f966aee7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96640d2280 con 0x7f966415d420 2023-11-28T17:10:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.022+0000 7f9663fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f96640d2710 0x7f96640b5210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34450/0 (socket says 172.21.15.116:34450) 2023-11-28T17:10:53.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.022+0000 7f9663fff640 1 -- 172.21.15.116:0/3004117343 learned_addr learned my addr 172.21.15.116:0/3004117343 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:53.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.022+0000 7f9669484640 1 --2- 172.21.15.116:0/3004117343 >> v2:172.21.15.116:3300/0 conn(0x7f96640b5920 0x7f96640b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:53.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.022+0000 7f9668c83640 1 --2- 172.21.15.116:0/3004117343 >> v2:172.21.15.156:3300/0 conn(0x7f966415d420 0x7f96640b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:53.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.022+0000 7f96637fe640 1 -- 172.21.15.116:0/3004117343 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736869989 0 0) 0x7f966415d800 con 0x7f96640d2710 2023-11-28T17:10:53.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.023+0000 7f96637fe640 1 -- 172.21.15.116:0/3004117343 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9654003520 con 0x7f96640d2710 2023-11-28T17:10:53.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.023+0000 7f9669484640 1 -- 172.21.15.116:0/3004117343 >> v2:172.21.15.156:3300/0 conn(0x7f966415d420 msgr2=0x7f96640b8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:53.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.023+0000 7f9669484640 1 --2- 172.21.15.116:0/3004117343 >> v2:172.21.15.156:3300/0 conn(0x7f966415d420 0x7f96640b8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.023+0000 7f9669484640 1 -- 172.21.15.116:0/3004117343 >> v1:172.21.15.116:6789/0 conn(0x7f96640d2710 legacy=0x7f96640b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.023+0000 7f9669484640 1 -- 172.21.15.116:0/3004117343 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96640d1670 con 0x7f96640b5920 2023-11-28T17:10:53.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.024+0000 7f9669484640 1 --2- 172.21.15.116:0/3004117343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96640b5920 0x7f96640b6360 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f96640cfcd0 tx=0x7f965c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02e3748b3a96a13 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:53.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.024+0000 7f96637fe640 1 -- 172.21.15.116:0/3004117343 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c02cc80 con 0x7f96640b5920 2023-11-28T17:10:53.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.024+0000 7f96637fe640 1 -- 172.21.15.116:0/3004117343 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965c02cde0 con 0x7f96640b5920 2023-11-28T17:10:53.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.024+0000 7f96637fe640 1 -- 172.21.15.116:0/3004117343 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c034660 con 0x7f96640b5920 2023-11-28T17:10:53.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.025+0000 7f966aee7640 1 -- 172.21.15.116:0/3004117343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96640b5920 msgr2=0x7f96640b6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.025+0000 7f966aee7640 1 --2- 172.21.15.116:0/3004117343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96640b5920 0x7f96640b6360 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f96640cfcd0 tx=0x7f965c02bed0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.025+0000 7f966aee7640 1 -- 172.21.15.116:0/3004117343 shutdown_connections 2023-11-28T17:10:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.025+0000 7f966aee7640 1 --2- 172.21.15.116:0/3004117343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96640b5920 0x7f96640b6360 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.025+0000 7f966aee7640 1 --2- 172.21.15.116:0/3004117343 >> v2:172.21.15.156:3300/0 conn(0x7f966415d420 0x7f96640b8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.025+0000 7f966aee7640 1 -- 172.21.15.116:0/3004117343 >> 172.21.15.116:0/3004117343 conn(0x7f9664072010 msgr2=0x7f9664072420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.025+0000 7f966aee7640 1 -- 172.21.15.116:0/3004117343 shutdown_connections 2023-11-28T17:10:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.025+0000 7f966aee7640 1 -- 172.21.15.116:0/3004117343 wait complete. 2023-11-28T17:10:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.027+0000 7f966aee7640 1 Processor -- start 2023-11-28T17:10:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.027+0000 7f966aee7640 1 -- start start 2023-11-28T17:10:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.028+0000 7f966aee7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96640b5920 0x7f96640ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.028+0000 7f966aee7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f96640d2710 0x7f96641f25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.028+0000 7f966aee7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f966415d420 0x7f9664146f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.028+0000 7f966aee7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96640c8cc0 con 0x7f96640b5920 2023-11-28T17:10:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.028+0000 7f966aee7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96640c80d0 con 0x7f966415d420 2023-11-28T17:10:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.028+0000 7f966aee7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96640ca840 con 0x7f96640d2710 2023-11-28T17:10:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.028+0000 7f9668c83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96640b5920 0x7f96640ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.028+0000 7f9668c83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96640b5920 0x7f96640ca300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53454/0 (socket says 172.21.15.116:53454) 2023-11-28T17:10:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.028+0000 7f9668c83640 1 -- 172.21.15.116:0/2067436924 learned_addr learned my addr 172.21.15.116:0/2067436924 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.029+0000 7f9668c83640 1 -- 172.21.15.116:0/2067436924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f96640d2710 msgr2=0x7f96641f25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.029+0000 7f9668c83640 1 --2- 172.21.15.116:0/2067436924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f96640d2710 0x7f96641f25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.029+0000 7f9668c83640 1 -- 172.21.15.116:0/2067436924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f966415d420 msgr2=0x7f9664146f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.029+0000 7f9669484640 1 --2- 172.21.15.116:0/2067436924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f966415d420 0x7f9664146f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.029+0000 7f9668c83640 1 --2- 172.21.15.116:0/2067436924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f966415d420 0x7f9664146f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.035 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.029+0000 7f9668c83640 1 -- 172.21.15.116:0/2067436924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965c009a70 con 0x7f96640b5920 2023-11-28T17:10:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.029+0000 7f9669484640 1 --2- 172.21.15.116:0/2067436924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f966415d420 0x7f9664146f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.029+0000 7f9668c83640 1 --2- 172.21.15.116:0/2067436924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96640b5920 0x7f96640ca300 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f965800d800 tx=0x7f965800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.029+0000 7f96617fa640 1 -- 172.21.15.116:0/2067436924 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965800b6c0 con 0x7f96640b5920 2023-11-28T17:10:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.029+0000 7f96617fa640 1 -- 172.21.15.116:0/2067436924 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965800b820 con 0x7f96640b5920 2023-11-28T17:10:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.029+0000 7f96617fa640 1 -- 172.21.15.116:0/2067436924 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965800bbb0 con 0x7f96640b5920 2023-11-28T17:10:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.029+0000 7f966aee7640 1 -- 172.21.15.116:0/2067436924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96640c8320 con 0x7f96640b5920 2023-11-28T17:10:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.029+0000 7f966aee7640 1 -- 172.21.15.116:0/2067436924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96640c88e0 con 0x7f96640b5920 2023-11-28T17:10:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.031+0000 7f966aee7640 1 -- 172.21.15.116:0/2067436924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96640c8f30 con 0x7f96640b5920 2023-11-28T17:10:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.032+0000 7f96617fa640 1 -- 172.21.15.116:0/2067436924 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f965800bd10 con 0x7f96640b5920 2023-11-28T17:10:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.032+0000 7f96617fa640 1 --2- 172.21.15.116:0/2067436924 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f964806b2a0 0x7f964806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.033+0000 7f96617fa640 1 -- 172.21.15.116:0/2067436924 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f965808a1e0 con 0x7f96640b5920 2023-11-28T17:10:53.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.033+0000 7f9663fff640 1 --2- 172.21.15.116:0/2067436924 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f964806b2a0 0x7f964806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:53.037 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.033+0000 7f9663fff640 1 --2- 172.21.15.116:0/2067436924 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f964806b2a0 0x7f964806d760 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f96640c8040 tx=0x7f9650007930 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:53.038 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.037+0000 7f96617fa640 1 -- 172.21.15.116:0/2067436924 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9658054fc0 con 0x7f96640b5920 2023-11-28T17:10:53.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.225+0000 7f966aee7640 1 -- 172.21.15.116:0/2067436924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9664069360 con 0x7f96640b5920 2023-11-28T17:10:53.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.226+0000 7f96617fa640 1 -- 172.21.15.116:0/2067436924 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7f96580585e0 con 0x7f96640b5920 2023-11-28T17:10:53.228 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:10:53.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.229+0000 7f966aee7640 1 -- 172.21.15.116:0/2067436924 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f964806b2a0 msgr2=0x7f964806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.229+0000 7f966aee7640 1 --2- 172.21.15.116:0/2067436924 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f964806b2a0 0x7f964806d760 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f96640c8040 tx=0x7f9650007930 comp rx=0 tx=0).stop 2023-11-28T17:10:53.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.229+0000 7f966aee7640 1 -- 172.21.15.116:0/2067436924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96640b5920 msgr2=0x7f96640ca300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.229+0000 7f966aee7640 1 --2- 172.21.15.116:0/2067436924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96640b5920 0x7f96640ca300 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f965800d800 tx=0x7f965800dcc0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.229+0000 7f966aee7640 1 -- 172.21.15.116:0/2067436924 shutdown_connections 2023-11-28T17:10:53.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.229+0000 7f966aee7640 1 --2- 172.21.15.116:0/2067436924 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f964806b2a0 0x7f964806d760 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.229+0000 7f966aee7640 1 --2- 172.21.15.116:0/2067436924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f966415d420 0x7f9664146f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.229+0000 7f966aee7640 1 --2- 172.21.15.116:0/2067436924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f96640d2710 0x7f96641f25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.229+0000 7f966aee7640 1 --2- 172.21.15.116:0/2067436924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96640b5920 0x7f96640ca300 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.229+0000 7f966aee7640 1 -- 172.21.15.116:0/2067436924 >> 172.21.15.116:0/2067436924 conn(0x7f9664072010 msgr2=0x7f966415c140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:53.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.229+0000 7f966aee7640 1 -- 172.21.15.116:0/2067436924 shutdown_connections 2023-11-28T17:10:53.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.230+0000 7f966aee7640 1 -- 172.21.15.116:0/2067436924 wait complete. 2023-11-28T17:10:53.248 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:10:53.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.465+0000 7f88a1661640 1 Processor -- start 2023-11-28T17:10:53.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.465+0000 7f88a1661640 1 -- start start 2023-11-28T17:10:53.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.466+0000 7f88a1661640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f889c13f640 0x7f889c13fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.466+0000 7f88a1661640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889c13dd10 con 0x7f889c071d20 2023-11-28T17:10:53.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.467+0000 7f88a1661640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889c13d2c0 con 0x7f889c13cee0 2023-11-28T17:10:53.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.467+0000 7f88a1661640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889c1400e0 con 0x7f889c13f640 2023-11-28T17:10:53.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.467+0000 7f889b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f889c071d20 0x7f889c072130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34452/0 (socket says 172.21.15.116:34452) 2023-11-28T17:10:53.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.467+0000 7f889b7fe640 1 -- 172.21.15.116:0/3997919247 learned_addr learned my addr 172.21.15.116:0/3997919247 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:53.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.467+0000 7f889a7fc640 1 -- 172.21.15.116:0/3997919247 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611981915 0 0) 0x7f889c13dd10 con 0x7f889c071d20 2023-11-28T17:10:53.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.467+0000 7f889a7fc640 1 -- 172.21.15.116:0/3997919247 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8888003520 con 0x7f889c071d20 2023-11-28T17:10:53.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.467+0000 7f88927fc640 1 --2- 172.21.15.116:0/3997919247 >> v2:172.21.15.156:3300/0 conn(0x7f889c13f640 0x7f889c13fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:53.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889a7fc640 1 -- 172.21.15.116:0/3997919247 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3436311138 0 0) 0x7f8888003520 con 0x7f889c071d20 2023-11-28T17:10:53.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889a7fc640 1 -- 172.21.15.116:0/3997919247 >> v2:172.21.15.156:3300/0 conn(0x7f889c13f640 msgr2=0x7f889c13fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:53.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889a7fc640 1 --2- 172.21.15.116:0/3997919247 >> v2:172.21.15.156:3300/0 conn(0x7f889c13f640 0x7f889c13fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889a7fc640 1 -- 172.21.15.116:0/3997919247 >> v1:172.21.15.156:6789/0 conn(0x7f889c13cee0 legacy=0x7f889c13d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889a7fc640 1 -- 172.21.15.116:0/3997919247 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c1403c0 con 0x7f889c071d20 2023-11-28T17:10:53.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889a7fc640 1 -- 172.21.15.116:0/3997919247 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f88940019f0 con 0x7f889c071d20 2023-11-28T17:10:53.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889a7fc640 1 -- 172.21.15.116:0/3997919247 >> v1:172.21.15.116:6789/0 conn(0x7f889c071d20 legacy=0x7f889c072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889a7fc640 1 --2- 172.21.15.116:0/3997919247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88880039c0 0x7f8888003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889a7fc640 1 --2- 172.21.15.116:0/3997919247 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88880047c0 0x7f8888004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889a7fc640 1 --2- 172.21.15.116:0/3997919247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8888005380 0x7f888800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889a7fc640 1 -- 172.21.15.116:0/3997919247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c1400e0 con 0x7f88880039c0 2023-11-28T17:10:53.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889a7fc640 1 -- 172.21.15.116:0/3997919247 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888800c260 con 0x7f8888005380 2023-11-28T17:10:53.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889a7fc640 1 -- 172.21.15.116:0/3997919247 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888800e630 con 0x7f88880047c0 2023-11-28T17:10:53.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889affd640 1 --2- 172.21.15.116:0/3997919247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88880039c0 0x7f8888003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:53.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889affd640 1 -- 172.21.15.116:0/3997919247 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88880047c0 msgr2=0x7f8888004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:53.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889affd640 1 --2- 172.21.15.116:0/3997919247 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88880047c0 0x7f8888004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889affd640 1 -- 172.21.15.116:0/3997919247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8888005380 msgr2=0x7f888800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:53.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.468+0000 7f889affd640 1 --2- 172.21.15.116:0/3997919247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8888005380 0x7f888800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.469+0000 7f889affd640 1 -- 172.21.15.116:0/3997919247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8888003520 con 0x7f88880039c0 2023-11-28T17:10:53.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.469+0000 7f889b7fe640 1 -- 172.21.15.116:0/3997919247 reap_dead start 2023-11-28T17:10:53.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.470+0000 7f889affd640 1 --2- 172.21.15.116:0/3997919247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88880039c0 0x7f8888003de0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f888c001750 tx=0x7f88880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2c83e4225e7844 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:53.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.470+0000 7f889a7fc640 1 -- 172.21.15.116:0/3997919247 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c0052d0 con 0x7f88880039c0 2023-11-28T17:10:53.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.470+0000 7f889a7fc640 1 -- 172.21.15.116:0/3997919247 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888c02e050 con 0x7f88880039c0 2023-11-28T17:10:53.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.470+0000 7f889a7fc640 1 -- 172.21.15.116:0/3997919247 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c003830 con 0x7f88880039c0 2023-11-28T17:10:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.470+0000 7f88a1661640 1 -- 172.21.15.116:0/3997919247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88880039c0 msgr2=0x7f8888003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.470+0000 7f88a1661640 1 --2- 172.21.15.116:0/3997919247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88880039c0 0x7f8888003de0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f888c001750 tx=0x7f88880057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.470+0000 7f88a1661640 1 -- 172.21.15.116:0/3997919247 shutdown_connections 2023-11-28T17:10:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.470+0000 7f88a1661640 1 --2- 172.21.15.116:0/3997919247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88880039c0 0x7f8888003de0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.470+0000 7f88a1661640 1 -- 172.21.15.116:0/3997919247 >> 172.21.15.116:0/3997919247 conn(0x7f889c0b0cf0 msgr2=0x7f889c0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.471+0000 7f88a1661640 1 -- 172.21.15.116:0/3997919247 shutdown_connections 2023-11-28T17:10:53.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.471+0000 7f88a1661640 1 -- 172.21.15.116:0/3997919247 wait complete. 2023-11-28T17:10:53.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.472+0000 7f88a1661640 1 Processor -- start 2023-11-28T17:10:53.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.473+0000 7f88a1661640 1 -- start start 2023-11-28T17:10:53.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.473+0000 7f88a1661640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88880039c0 0x7f889c1427e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.473+0000 7f88a1661640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f889c142d20 0x7f889c1f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.474+0000 7f88a1661640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f889c1f2820 0x7f889c0acb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.474+0000 7f88a1661640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c140730 con 0x7f889c142d20 2023-11-28T17:10:53.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.474+0000 7f88a1661640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c1f2cb0 con 0x7f889c1f2820 2023-11-28T17:10:53.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.474+0000 7f88a1661640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889c0ad840 con 0x7f88880039c0 2023-11-28T17:10:53.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.474+0000 7f88927fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f889c142d20 0x7f889c1f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:53.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.474+0000 7f88927fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f889c142d20 0x7f889c1f22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53472/0 (socket says 172.21.15.116:53472) 2023-11-28T17:10:53.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.474+0000 7f88927fc640 1 -- 172.21.15.116:0/2636944469 learned_addr learned my addr 172.21.15.116:0/2636944469 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:53.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.474+0000 7f88927fc640 1 -- 172.21.15.116:0/2636944469 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88880039c0 msgr2=0x7f889c1427e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.474+0000 7f88927fc640 1 --2- 172.21.15.116:0/2636944469 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88880039c0 0x7f889c1427e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.474+0000 7f88927fc640 1 -- 172.21.15.116:0/2636944469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f889c1f2820 msgr2=0x7f889c0acb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.474+0000 7f88927fc640 1 --2- 172.21.15.116:0/2636944469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f889c1f2820 0x7f889c0acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.474+0000 7f88927fc640 1 -- 172.21.15.116:0/2636944469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c0adb20 con 0x7f889c142d20 2023-11-28T17:10:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.474+0000 7f88927fc640 1 --2- 172.21.15.116:0/2636944469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f889c142d20 0x7f889c1f22e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f888400d830 tx=0x7f888400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.475+0000 7f8893fff640 1 -- 172.21.15.116:0/2636944469 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888400b6c0 con 0x7f889c142d20 2023-11-28T17:10:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.475+0000 7f8893fff640 1 -- 172.21.15.116:0/2636944469 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888400b820 con 0x7f889c142d20 2023-11-28T17:10:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.475+0000 7f88a1661640 1 -- 172.21.15.116:0/2636944469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889c0ade00 con 0x7f889c142d20 2023-11-28T17:10:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.475+0000 7f8893fff640 1 -- 172.21.15.116:0/2636944469 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888400bbd0 con 0x7f889c142d20 2023-11-28T17:10:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.475+0000 7f88a1661640 1 -- 172.21.15.116:0/2636944469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889c0ad050 con 0x7f889c142d20 2023-11-28T17:10:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.476+0000 7f88a1661640 1 -- 172.21.15.116:0/2636944469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889c069360 con 0x7f889c142d20 2023-11-28T17:10:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.477+0000 7f8893fff640 1 -- 172.21.15.116:0/2636944469 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f888401e460 con 0x7f889c142d20 2023-11-28T17:10:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.478+0000 7f8893fff640 1 --2- 172.21.15.116:0/2636944469 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f887c06b2d0 0x7f887c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.478+0000 7f8893fff640 1 -- 172.21.15.116:0/2636944469 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f888408a320 con 0x7f889c142d20 2023-11-28T17:10:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.478+0000 7f889affd640 1 --2- 172.21.15.116:0/2636944469 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f887c06b2d0 0x7f887c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:53.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.479+0000 7f889affd640 1 --2- 172.21.15.116:0/2636944469 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f887c06b2d0 0x7f887c06d790 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f888c033000 tx=0x7f888c02f000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:53.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.482+0000 7f8893fff640 1 -- 172.21.15.116:0/2636944469 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8884055100 con 0x7f889c142d20 2023-11-28T17:10:53.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.670+0000 7f88a1661640 1 -- 172.21.15.116:0/2636944469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f889c070380 con 0x7f889c142d20 2023-11-28T17:10:53.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.671+0000 7f8893fff640 1 -- 172.21.15.116:0/2636944469 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8884058720 con 0x7f889c142d20 2023-11-28T17:10:53.672 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:10:53.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.673+0000 7f88a1661640 1 -- 172.21.15.116:0/2636944469 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f887c06b2d0 msgr2=0x7f887c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.673+0000 7f88a1661640 1 --2- 172.21.15.116:0/2636944469 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f887c06b2d0 0x7f887c06d790 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f888c033000 tx=0x7f888c02f000 comp rx=0 tx=0).stop 2023-11-28T17:10:53.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.673+0000 7f88a1661640 1 -- 172.21.15.116:0/2636944469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f889c142d20 msgr2=0x7f889c1f22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.673+0000 7f88a1661640 1 --2- 172.21.15.116:0/2636944469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f889c142d20 0x7f889c1f22e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f888400d830 tx=0x7f888400dcf0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.673+0000 7f88a1661640 1 -- 172.21.15.116:0/2636944469 shutdown_connections 2023-11-28T17:10:53.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.673+0000 7f88a1661640 1 --2- 172.21.15.116:0/2636944469 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f887c06b2d0 0x7f887c06d790 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.673+0000 7f88a1661640 1 --2- 172.21.15.116:0/2636944469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f889c1f2820 0x7f889c0acb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.673+0000 7f88a1661640 1 --2- 172.21.15.116:0/2636944469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f889c142d20 0x7f889c1f22e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.673+0000 7f88a1661640 1 --2- 172.21.15.116:0/2636944469 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88880039c0 0x7f889c1427e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.673+0000 7f88a1661640 1 -- 172.21.15.116:0/2636944469 >> 172.21.15.116:0/2636944469 conn(0x7f889c0b0cf0 msgr2=0x7f889c13dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:53.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.673+0000 7f88a1661640 1 -- 172.21.15.116:0/2636944469 shutdown_connections 2023-11-28T17:10:53.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.674+0000 7f88a1661640 1 -- 172.21.15.116:0/2636944469 wait complete. 2023-11-28T17:10:53.695 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:10:53.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.908+0000 7f72df59e640 1 Processor -- start 2023-11-28T17:10:53.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.908+0000 7f72df59e640 1 -- start start 2023-11-28T17:10:53.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.910+0000 7f72df59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72e015d4d0 0x7f72e00ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.910+0000 7f72df59e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f72e00d0e00 0x7f72e00d1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.910+0000 7f72df59e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e00d1c30 con 0x7f72e00d17a0 2023-11-28T17:10:53.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.910+0000 7f72df59e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e00cf710 con 0x7f72e00d0e00 2023-11-28T17:10:53.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.910+0000 7f72df59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e00cee70 con 0x7f72e015d4d0 2023-11-28T17:10:53.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.911+0000 7f72ddd9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f72e00d0e00 0x7f72e00d1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:53.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.911+0000 7f72de59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72e015d4d0 0x7f72e00ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:53.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.911+0000 7f72ddd9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f72e00d0e00 0x7f72e00d1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:33530/0 (socket says 172.21.15.116:33530) 2023-11-28T17:10:53.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.911+0000 7f72ddd9b640 1 -- 172.21.15.116:0/183496618 learned_addr learned my addr 172.21.15.116:0/183496618 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:53.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.911+0000 7f72ddd9b640 1 -- 172.21.15.116:0/183496618 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72e015d4d0 msgr2=0x7f72e00ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:53.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.911+0000 7f72ddd9b640 1 --2- 172.21.15.116:0/183496618 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72e015d4d0 0x7f72e00ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.911+0000 7f72ddd9b640 1 -- 172.21.15.116:0/183496618 >> v1:172.21.15.156:6789/0 conn(0x7f72e00d17a0 legacy=0x7f72e00cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.911+0000 7f72ddd9b640 1 -- 172.21.15.116:0/183496618 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e00cf150 con 0x7f72e00d0e00 2023-11-28T17:10:53.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.913+0000 7f72ddd9b640 1 --2- 172.21.15.116:0/183496618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72e00d0e00 0x7f72e00d1260 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f72c8009e80 tx=0x7f72c802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=789132ff27134046 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:53.914 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.913+0000 7f72dd59a640 1 -- 172.21.15.116:0/183496618 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c802caf0 con 0x7f72e00d0e00 2023-11-28T17:10:53.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.913+0000 7f72dd59a640 1 -- 172.21.15.116:0/183496618 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c802cc50 con 0x7f72e00d0e00 2023-11-28T17:10:53.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.913+0000 7f72dd59a640 1 -- 172.21.15.116:0/183496618 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c80343f0 con 0x7f72e00d0e00 2023-11-28T17:10:53.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.914+0000 7f72df59e640 1 -- 172.21.15.116:0/183496618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72e00d0e00 msgr2=0x7f72e00d1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.914+0000 7f72df59e640 1 --2- 172.21.15.116:0/183496618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72e00d0e00 0x7f72e00d1260 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f72c8009e80 tx=0x7f72c802bf50 comp rx=0 tx=0).stop 2023-11-28T17:10:53.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.914+0000 7f72df59e640 1 -- 172.21.15.116:0/183496618 shutdown_connections 2023-11-28T17:10:53.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.914+0000 7f72df59e640 1 --2- 172.21.15.116:0/183496618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72e00d0e00 0x7f72e00d1260 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.914+0000 7f72df59e640 1 --2- 172.21.15.116:0/183496618 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72e015d4d0 0x7f72e00ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.914+0000 7f72df59e640 1 -- 172.21.15.116:0/183496618 >> 172.21.15.116:0/183496618 conn(0x7f72e00d2f90 msgr2=0x7f72e00d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:53.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.914+0000 7f72df59e640 1 -- 172.21.15.116:0/183496618 shutdown_connections 2023-11-28T17:10:53.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.914+0000 7f72df59e640 1 -- 172.21.15.116:0/183496618 wait complete. 2023-11-28T17:10:53.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.916+0000 7f72df59e640 1 Processor -- start 2023-11-28T17:10:53.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.916+0000 7f72df59e640 1 -- start start 2023-11-28T17:10:53.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.917+0000 7f72df59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e00d0e00 0x7f72e00cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.917+0000 7f72df59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72e00d17a0 0x7f72e01f2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.917+0000 7f72df59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72e015d4d0 0x7f72e00b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.917+0000 7f72df59e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e00ccf50 con 0x7f72e00d0e00 2023-11-28T17:10:53.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.917+0000 7f72df59e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e00cd3e0 con 0x7f72e00d17a0 2023-11-28T17:10:53.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.917+0000 7f72df59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e00cc340 con 0x7f72e015d4d0 2023-11-28T17:10:53.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.917+0000 7f72de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e00d0e00 0x7f72e00cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:53.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.917+0000 7f72de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e00d0e00 0x7f72e00cba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53486/0 (socket says 172.21.15.116:53486) 2023-11-28T17:10:53.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.917+0000 7f72de59c640 1 -- 172.21.15.116:0/1260818629 learned_addr learned my addr 172.21.15.116:0/1260818629 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:53.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.918+0000 7f72ddd9b640 1 --2- 172.21.15.116:0/1260818629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72e00d17a0 0x7f72e01f2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:53.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.918+0000 7f72de59c640 1 -- 172.21.15.116:0/1260818629 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72e015d4d0 msgr2=0x7f72e00b7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.918+0000 7f72de59c640 1 --2- 172.21.15.116:0/1260818629 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72e015d4d0 0x7f72e00b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.918+0000 7f72de59c640 1 -- 172.21.15.116:0/1260818629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72e00d17a0 msgr2=0x7f72e01f2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:53.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.918+0000 7f72de59c640 1 --2- 172.21.15.116:0/1260818629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72e00d17a0 0x7f72e01f2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:53.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.918+0000 7f72de59c640 1 -- 172.21.15.116:0/1260818629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c8009b30 con 0x7f72e00d0e00 2023-11-28T17:10:53.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.918+0000 7f72de59c640 1 --2- 172.21.15.116:0/1260818629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e00d0e00 0x7f72e00cba30 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f72d800e870 tx=0x7f72d800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:53.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.918+0000 7f72d6ffd640 1 -- 172.21.15.116:0/1260818629 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d80094f0 con 0x7f72e00d0e00 2023-11-28T17:10:53.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.918+0000 7f72d6ffd640 1 -- 172.21.15.116:0/1260818629 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d800c940 con 0x7f72e00d0e00 2023-11-28T17:10:53.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.918+0000 7f72df59e640 1 -- 172.21.15.116:0/1260818629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e00cc620 con 0x7f72e00d0e00 2023-11-28T17:10:53.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.918+0000 7f72d6ffd640 1 -- 172.21.15.116:0/1260818629 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d800ccf0 con 0x7f72e00d0e00 2023-11-28T17:10:53.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.919+0000 7f72df59e640 1 -- 172.21.15.116:0/1260818629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e00ccbe0 con 0x7f72e00d0e00 2023-11-28T17:10:53.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.920+0000 7f72df59e640 1 -- 172.21.15.116:0/1260818629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e0067c40 con 0x7f72e00d0e00 2023-11-28T17:10:53.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.921+0000 7f72d6ffd640 1 -- 172.21.15.116:0/1260818629 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f72d8013470 con 0x7f72e00d0e00 2023-11-28T17:10:53.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.921+0000 7f72d6ffd640 1 --2- 172.21.15.116:0/1260818629 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f72c006b2a0 0x7f72c006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:53.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.921+0000 7f72d6ffd640 1 -- 172.21.15.116:0/1260818629 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f72d8089e10 con 0x7f72e00d0e00 2023-11-28T17:10:53.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.922+0000 7f72ddd9b640 1 --2- 172.21.15.116:0/1260818629 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f72c006b2a0 0x7f72c006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:53.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.922+0000 7f72ddd9b640 1 --2- 172.21.15.116:0/1260818629 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f72c006b2a0 0x7f72c006d760 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f72c8035000 tx=0x7f72c8030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:53.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:53.926+0000 7f72d6ffd640 1 -- 172.21.15.116:0/1260818629 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72d808d030 con 0x7f72e00d0e00 2023-11-28T17:10:54.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.134+0000 7f72df59e640 1 -- 172.21.15.116:0/1260818629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f72e00cd960 con 0x7f72e00d0e00 2023-11-28T17:10:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.135+0000 7f72d6ffd640 1 -- 172.21.15.116:0/1260818629 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) v1 ==== 80+0+0 (secure 0 0 0) 0x7f72d8021080 con 0x7f72e00d0e00 2023-11-28T17:10:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-28T17:10:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.137+0000 7f72df59e640 1 -- 172.21.15.116:0/1260818629 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f72c006b2a0 msgr2=0x7f72c006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.137+0000 7f72df59e640 1 --2- 172.21.15.116:0/1260818629 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f72c006b2a0 0x7f72c006d760 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f72c8035000 tx=0x7f72c8030000 comp rx=0 tx=0).stop 2023-11-28T17:10:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.137+0000 7f72df59e640 1 -- 172.21.15.116:0/1260818629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e00d0e00 msgr2=0x7f72e00cba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.137+0000 7f72df59e640 1 --2- 172.21.15.116:0/1260818629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e00d0e00 0x7f72e00cba30 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f72d800e870 tx=0x7f72d800ed30 comp rx=0 tx=0).stop 2023-11-28T17:10:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.138+0000 7f72df59e640 1 -- 172.21.15.116:0/1260818629 shutdown_connections 2023-11-28T17:10:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.138+0000 7f72df59e640 1 --2- 172.21.15.116:0/1260818629 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f72c006b2a0 0x7f72c006d760 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.138+0000 7f72df59e640 1 --2- 172.21.15.116:0/1260818629 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f72e015d4d0 0x7f72e00b7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.138+0000 7f72df59e640 1 --2- 172.21.15.116:0/1260818629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72e00d17a0 0x7f72e01f2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.138+0000 7f72df59e640 1 --2- 172.21.15.116:0/1260818629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72e00d0e00 0x7f72e00cba30 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.138+0000 7f72df59e640 1 -- 172.21.15.116:0/1260818629 >> 172.21.15.116:0/1260818629 conn(0x7f72e00d2f90 msgr2=0x7f72e00c7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.138+0000 7f72df59e640 1 -- 172.21.15.116:0/1260818629 shutdown_connections 2023-11-28T17:10:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.138+0000 7f72df59e640 1 -- 172.21.15.116:0/1260818629 wait complete. 2023-11-28T17:10:54.158 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:10:54.159 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:10:54.215 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:10:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.429+0000 7ff0f8009640 1 Processor -- start 2023-11-28T17:10:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.429+0000 7ff0f8009640 1 -- start start 2023-11-28T17:10:54.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.431+0000 7ff0f8009640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff0f015f7b0 0x7ff0f0071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.431+0000 7ff0f8009640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff0f00721b0 0x7ff0f006c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.431+0000 7ff0f8009640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f015fc10 con 0x7ff0f0144430 2023-11-28T17:10:54.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.431+0000 7ff0f8009640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0f0144810 con 0x7ff0f00721b0 2023-11-28T17:10:54.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.431+0000 7ff0f8009640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0f0070bd0 con 0x7ff0f015f7b0 2023-11-28T17:10:54.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.431+0000 7ff0f5da5640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff0f0144430 0x7ff0f00703f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34462/0 (socket says 172.21.15.116:34462) 2023-11-28T17:10:54.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.431+0000 7ff0f5da5640 1 -- 172.21.15.116:0/3818848379 learned_addr learned my addr 172.21.15.116:0/3818848379 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:54.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.431+0000 7ff0f65a6640 1 --2- 172.21.15.116:0/3818848379 >> v2:172.21.15.116:3300/0 conn(0x7ff0f00721b0 0x7ff0f006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:54.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.431+0000 7ff0f4da3640 1 -- 172.21.15.116:0/3818848379 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408431097 0 0) 0x7ff0f015fc10 con 0x7ff0f0144430 2023-11-28T17:10:54.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.431+0000 7ff0f4da3640 1 -- 172.21.15.116:0/3818848379 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0dc003520 con 0x7ff0f0144430 2023-11-28T17:10:54.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.431+0000 7ff0f55a4640 1 --2- 172.21.15.116:0/3818848379 >> v2:172.21.15.156:3300/0 conn(0x7ff0f015f7b0 0x7ff0f0071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:54.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f4da3640 1 -- 172.21.15.116:0/3818848379 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3747320799 0 0) 0x7ff0dc003520 con 0x7ff0f0144430 2023-11-28T17:10:54.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f4da3640 1 -- 172.21.15.116:0/3818848379 >> v2:172.21.15.156:3300/0 conn(0x7ff0f015f7b0 msgr2=0x7ff0f0071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:54.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f4da3640 1 --2- 172.21.15.116:0/3818848379 >> v2:172.21.15.156:3300/0 conn(0x7ff0f015f7b0 0x7ff0f0071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f4da3640 1 -- 172.21.15.116:0/3818848379 >> v2:172.21.15.116:3300/0 conn(0x7ff0f00721b0 msgr2=0x7ff0f006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:54.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f4da3640 1 --2- 172.21.15.116:0/3818848379 >> v2:172.21.15.116:3300/0 conn(0x7ff0f00721b0 0x7ff0f006c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f4da3640 1 -- 172.21.15.116:0/3818848379 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f0070eb0 con 0x7ff0f0144430 2023-11-28T17:10:54.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f4da3640 1 -- 172.21.15.116:0/3818848379 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7ff0ec001670 con 0x7ff0f0144430 2023-11-28T17:10:54.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f65a6640 1 --2- 172.21.15.116:0/3818848379 >> v2:172.21.15.116:3300/0 conn(0x7ff0f00721b0 0x7ff0f006c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:10:54.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f4da3640 1 -- 172.21.15.116:0/3818848379 >> v1:172.21.15.116:6789/0 conn(0x7ff0f0144430 legacy=0x7ff0f00703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:54.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f4da3640 1 --2- 172.21.15.116:0/3818848379 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0dc003970 0x7ff0dc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f4da3640 1 --2- 172.21.15.116:0/3818848379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0dc0047c0 0x7ff0dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f4da3640 1 --2- 172.21.15.116:0/3818848379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0dc005180 0x7ff0dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f4da3640 1 -- 172.21.15.116:0/3818848379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f0144810 con 0x7ff0dc0047c0 2023-11-28T17:10:54.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f4da3640 1 -- 172.21.15.116:0/3818848379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f0070bd0 con 0x7ff0dc005180 2023-11-28T17:10:54.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f4da3640 1 -- 172.21.15.116:0/3818848379 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc00bfc0 con 0x7ff0dc003970 2023-11-28T17:10:54.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f55a4640 1 --2- 172.21.15.116:0/3818848379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0dc0047c0 0x7ff0dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f55a4640 1 -- 172.21.15.116:0/3818848379 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0dc003970 msgr2=0x7ff0dc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f55a4640 1 --2- 172.21.15.116:0/3818848379 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0dc003970 0x7ff0dc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f65a6640 1 --2- 172.21.15.116:0/3818848379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0dc005180 0x7ff0dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f55a4640 1 -- 172.21.15.116:0/3818848379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0dc005180 msgr2=0x7ff0dc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.432+0000 7ff0f55a4640 1 --2- 172.21.15.116:0/3818848379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0dc005180 0x7ff0dc00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.433+0000 7ff0f65a6640 1 -- 172.21.15.116:0/3818848379 reap_dead start 2023-11-28T17:10:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.433+0000 7ff0f55a4640 1 -- 172.21.15.116:0/3818848379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc003520 con 0x7ff0dc0047c0 2023-11-28T17:10:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.434+0000 7ff0f55a4640 1 --2- 172.21.15.116:0/3818848379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0dc0047c0 0x7ff0dc004c40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400ba30 tx=0x7ff0dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a8ded554d90cc8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.434+0000 7ff0f4da3640 1 -- 172.21.15.116:0/3818848379 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4033440 con 0x7ff0dc0047c0 2023-11-28T17:10:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.434+0000 7ff0f4da3640 1 -- 172.21.15.116:0/3818848379 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e40335a0 con 0x7ff0dc0047c0 2023-11-28T17:10:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.434+0000 7ff0f4da3640 1 -- 172.21.15.116:0/3818848379 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4033930 con 0x7ff0dc0047c0 2023-11-28T17:10:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.434+0000 7ff0f8009640 1 -- 172.21.15.116:0/3818848379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0dc0047c0 msgr2=0x7ff0dc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.434+0000 7ff0f8009640 1 --2- 172.21.15.116:0/3818848379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0dc0047c0 0x7ff0dc004c40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400ba30 tx=0x7ff0dc0057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.434+0000 7ff0f8009640 1 -- 172.21.15.116:0/3818848379 shutdown_connections 2023-11-28T17:10:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.435+0000 7ff0f8009640 1 --2- 172.21.15.116:0/3818848379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0dc0047c0 0x7ff0dc004c40 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.435+0000 7ff0f8009640 1 -- 172.21.15.116:0/3818848379 >> 172.21.15.116:0/3818848379 conn(0x7ff0f006a4c0 msgr2=0x7ff0f006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.435+0000 7ff0f8009640 1 -- 172.21.15.116:0/3818848379 shutdown_connections 2023-11-28T17:10:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.435+0000 7ff0f8009640 1 -- 172.21.15.116:0/3818848379 wait complete. 2023-11-28T17:10:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.436+0000 7ff0f8009640 1 Processor -- start 2023-11-28T17:10:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.436+0000 7ff0f8009640 1 -- start start 2023-11-28T17:10:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.437+0000 7ff0f8009640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0dc0047c0 0x7ff0f015a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.437+0000 7ff0f8009640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0f00a8460 0x7ff0f00a88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.437+0000 7ff0f8009640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0f00b2b00 0x7ff0f00b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.437+0000 7ff0f8009640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00b3ab0 con 0x7ff0f00a8460 2023-11-28T17:10:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.437+0000 7ff0f8009640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00a8e00 con 0x7ff0f00b2b00 2023-11-28T17:10:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.437+0000 7ff0f8009640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f015aeb0 con 0x7ff0dc0047c0 2023-11-28T17:10:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.438+0000 7ff0f55a4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0f00a8460 0x7ff0f00a88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.438+0000 7ff0f55a4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0f00a8460 0x7ff0f00a88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53508/0 (socket says 172.21.15.116:53508) 2023-11-28T17:10:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.438+0000 7ff0f55a4640 1 -- 172.21.15.116:0/1109426180 learned_addr learned my addr 172.21.15.116:0/1109426180 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.438+0000 7ff0f55a4640 1 -- 172.21.15.116:0/1109426180 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0dc0047c0 msgr2=0x7ff0f015a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.438+0000 7ff0f55a4640 1 --2- 172.21.15.116:0/1109426180 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0dc0047c0 0x7ff0f015a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.438+0000 7ff0f55a4640 1 -- 172.21.15.116:0/1109426180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0f00b2b00 msgr2=0x7ff0f00b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.438+0000 7ff0f65a6640 1 --2- 172.21.15.116:0/1109426180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0f00b2b00 0x7ff0f00b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.438+0000 7ff0f55a4640 1 --2- 172.21.15.116:0/1109426180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0f00b2b00 0x7ff0f00b2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.438+0000 7ff0f55a4640 1 -- 172.21.15.116:0/1109426180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f01306d0 con 0x7ff0f00a8460 2023-11-28T17:10:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.438+0000 7ff0f55a4640 1 --2- 172.21.15.116:0/1109426180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0f00a8460 0x7ff0f00a88c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff0e402be20 tx=0x7ff0e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.439+0000 7ff0e27fc640 1 -- 172.21.15.116:0/1109426180 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4004220 con 0x7ff0f00a8460 2023-11-28T17:10:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.439+0000 7ff0e27fc640 1 -- 172.21.15.116:0/1109426180 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e4040070 con 0x7ff0f00a8460 2023-11-28T17:10:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.439+0000 7ff0e27fc640 1 -- 172.21.15.116:0/1109426180 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4003610 con 0x7ff0f00a8460 2023-11-28T17:10:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.439+0000 7ff0f8009640 1 -- 172.21.15.116:0/1109426180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f00b37d0 con 0x7ff0f00a8460 2023-11-28T17:10:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.439+0000 7ff0f8009640 1 -- 172.21.15.116:0/1109426180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f0130c50 con 0x7ff0f00a8460 2023-11-28T17:10:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.440+0000 7ff0f8009640 1 -- 172.21.15.116:0/1109426180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f0130530 con 0x7ff0f00a8460 2023-11-28T17:10:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.441+0000 7ff0e27fc640 1 -- 172.21.15.116:0/1109426180 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff0e4031050 con 0x7ff0f00a8460 2023-11-28T17:10:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.442+0000 7ff0e27fc640 1 --2- 172.21.15.116:0/1109426180 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff0d406b2a0 0x7ff0d406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.442+0000 7ff0e27fc640 1 -- 172.21.15.116:0/1109426180 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff0e40ab8f0 con 0x7ff0f00a8460 2023-11-28T17:10:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.443+0000 7ff0f5da5640 1 --2- 172.21.15.116:0/1109426180 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff0d406b2a0 0x7ff0d406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.443+0000 7ff0f5da5640 1 --2- 172.21.15.116:0/1109426180 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff0d406b2a0 0x7ff0d406d760 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec001eb0 tx=0x7ff0ec001520 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.447+0000 7ff0e27fc640 1 -- 172.21.15.116:0/1109426180 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff0f0130530 con 0x7ff0f00a8460 2023-11-28T17:10:54.640 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.639+0000 7ff0f8009640 1 -- 172.21.15.116:0/1109426180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0f0130530 con 0x7ff0f00a8460 2023-11-28T17:10:54.643 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.642+0000 7ff0e27fc640 1 -- 172.21.15.116:0/1109426180 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16770 (secure 0 0 0) 0x7ff0f0130530 con 0x7ff0f00a8460 2023-11-28T17:10:54.643 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:54.644 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":36,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:10:44.069312+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:10:17.338444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-28T17:10:18.766338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.1d","mappings":[{"from":4,"to":5}]},{"pgid":"5.37","mappings":[{"from":4,"to":5}]},{"pgid":"5.3e","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:10:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.644+0000 7ff0f8009640 1 -- 172.21.15.116:0/1109426180 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff0d406b2a0 msgr2=0x7ff0d406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.644+0000 7ff0f8009640 1 --2- 172.21.15.116:0/1109426180 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff0d406b2a0 0x7ff0d406d760 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec001eb0 tx=0x7ff0ec001520 comp rx=0 tx=0).stop 2023-11-28T17:10:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.644+0000 7ff0f8009640 1 -- 172.21.15.116:0/1109426180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0f00a8460 msgr2=0x7ff0f00a88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.644+0000 7ff0f8009640 1 --2- 172.21.15.116:0/1109426180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0f00a8460 0x7ff0f00a88c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ff0e402be20 tx=0x7ff0e4004730 comp rx=0 tx=0).stop 2023-11-28T17:10:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.644+0000 7ff0f8009640 1 -- 172.21.15.116:0/1109426180 shutdown_connections 2023-11-28T17:10:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.644+0000 7ff0f8009640 1 --2- 172.21.15.116:0/1109426180 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff0d406b2a0 0x7ff0d406d760 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.644+0000 7ff0f8009640 1 --2- 172.21.15.116:0/1109426180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0f00b2b00 0x7ff0f00b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.644+0000 7ff0f8009640 1 --2- 172.21.15.116:0/1109426180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0f00a8460 0x7ff0f00a88c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.644+0000 7ff0f8009640 1 --2- 172.21.15.116:0/1109426180 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0dc0047c0 0x7ff0f015a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.644+0000 7ff0f8009640 1 -- 172.21.15.116:0/1109426180 >> 172.21.15.116:0/1109426180 conn(0x7ff0f006a4c0 msgr2=0x7ff0f00cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.645+0000 7ff0f8009640 1 -- 172.21.15.116:0/1109426180 shutdown_connections 2023-11-28T17:10:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.645+0000 7ff0f8009640 1 -- 172.21.15.116:0/1109426180 wait complete. 2023-11-28T17:10:54.665 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:10:17.338444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:10:18.766338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:10:54.666 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:10:54.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.872+0000 7fa37c8ee640 1 Processor -- start 2023-11-28T17:10:54.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.872+0000 7fa37c8ee640 1 -- start start 2023-11-28T17:10:54.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.874+0000 7fa37c8ee640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa378146990 0x7fa37806d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.874+0000 7fa37c8ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3780b0b10 0x7fa3780b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.874+0000 7fa37c8ee640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37806d170 con 0x7fa3780b1e80 2023-11-28T17:10:54.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.874+0000 7fa37c8ee640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3780b1470 con 0x7fa378146990 2023-11-28T17:10:54.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.874+0000 7fa37c8ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37806c880 con 0x7fa3780b0b10 2023-11-28T17:10:54.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.874+0000 7fa37659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa378146990 0x7fa37806d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:54.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.874+0000 7fa376d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa3780b1e80 0x7fa378070310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34466/0 (socket says 172.21.15.116:34466) 2023-11-28T17:10:54.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.874+0000 7fa376d9d640 1 -- 172.21.15.116:0/2548238526 learned_addr learned my addr 172.21.15.116:0/2548238526 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:54.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.874+0000 7fa375d9b640 1 --2- 172.21.15.116:0/2548238526 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3780b0b10 0x7fa3780b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:54.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.875+0000 7fa37559a640 1 -- 172.21.15.116:0/2548238526 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187199218 0 0) 0x7fa37806d170 con 0x7fa3780b1e80 2023-11-28T17:10:54.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.875+0000 7fa37559a640 1 -- 172.21.15.116:0/2548238526 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa360003520 con 0x7fa3780b1e80 2023-11-28T17:10:54.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.875+0000 7fa37659c640 1 -- 172.21.15.116:0/2548238526 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3780b0b10 msgr2=0x7fa3780b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:54.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.875+0000 7fa37659c640 1 --2- 172.21.15.116:0/2548238526 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3780b0b10 0x7fa3780b0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.875+0000 7fa37659c640 1 -- 172.21.15.116:0/2548238526 >> v1:172.21.15.116:6789/0 conn(0x7fa3780b1e80 legacy=0x7fa378070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:54.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.875+0000 7fa37659c640 1 -- 172.21.15.116:0/2548238526 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37806cb20 con 0x7fa378146990 2023-11-28T17:10:54.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.876+0000 7fa37659c640 1 --2- 172.21.15.116:0/2548238526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa378146990 0x7fa37806d600 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa3680097a0 tx=0x7fa36802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=725da2aff952d46e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:54.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.876+0000 7fa37559a640 1 -- 172.21.15.116:0/2548238526 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368031830 con 0x7fa378146990 2023-11-28T17:10:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.876+0000 7fa37559a640 1 -- 172.21.15.116:0/2548238526 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa368031990 con 0x7fa378146990 2023-11-28T17:10:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.876+0000 7fa37559a640 1 -- 172.21.15.116:0/2548238526 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368031d20 con 0x7fa378146990 2023-11-28T17:10:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.877+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2548238526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa378146990 msgr2=0x7fa37806d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.877+0000 7fa37c8ee640 1 --2- 172.21.15.116:0/2548238526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa378146990 0x7fa37806d600 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa3680097a0 tx=0x7fa36802b8d0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.877+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2548238526 shutdown_connections 2023-11-28T17:10:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.877+0000 7fa37c8ee640 1 --2- 172.21.15.116:0/2548238526 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3780b0b10 0x7fa3780b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.877+0000 7fa37c8ee640 1 --2- 172.21.15.116:0/2548238526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa378146990 0x7fa37806d600 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.877+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2548238526 >> 172.21.15.116:0/2548238526 conn(0x7fa37806a4c0 msgr2=0x7fa37806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.877+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2548238526 shutdown_connections 2023-11-28T17:10:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.877+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2548238526 wait complete. 2023-11-28T17:10:54.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.879+0000 7fa37c8ee640 1 Processor -- start 2023-11-28T17:10:54.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.879+0000 7fa37c8ee640 1 -- start start 2023-11-28T17:10:54.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.880+0000 7fa37c8ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780b0b10 0x7fa378130ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.880+0000 7fa37c8ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3780b1e80 0x7fa378132db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.880+0000 7fa37c8ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378146990 0x7fa378145670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.880+0000 7fa37c8ee640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3781425e0 con 0x7fa3780b0b10 2023-11-28T17:10:54.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.880+0000 7fa37c8ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa378133980 con 0x7fa378146990 2023-11-28T17:10:54.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.880+0000 7fa37c8ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa378142850 con 0x7fa3780b1e80 2023-11-28T17:10:54.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.880+0000 7fa37659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780b0b10 0x7fa378130ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:54.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.880+0000 7fa37659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780b0b10 0x7fa378130ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53524/0 (socket says 172.21.15.116:53524) 2023-11-28T17:10:54.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.880+0000 7fa37659c640 1 -- 172.21.15.116:0/2054746529 learned_addr learned my addr 172.21.15.116:0/2054746529 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:54.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.880+0000 7fa37659c640 1 -- 172.21.15.116:0/2054746529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3780b1e80 msgr2=0x7fa378132db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:54.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.880+0000 7fa375d9b640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3780b1e80 0x7fa378132db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:54.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.880+0000 7fa376d9d640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378146990 0x7fa378145670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:54.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.881+0000 7fa37659c640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3780b1e80 0x7fa378132db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.881+0000 7fa37659c640 1 -- 172.21.15.116:0/2054746529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378146990 msgr2=0x7fa378145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:54.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.881+0000 7fa37659c640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378146990 0x7fa378145670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:54.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.881+0000 7fa37659c640 1 -- 172.21.15.116:0/2054746529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa368009400 con 0x7fa3780b0b10 2023-11-28T17:10:54.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.881+0000 7fa376d9d640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378146990 0x7fa378145670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:54.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.881+0000 7fa375d9b640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3780b1e80 0x7fa378132db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:54.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.881+0000 7fa37659c640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780b0b10 0x7fa378130ab0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa368034040 tx=0x7fa3680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:54.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.881+0000 7fa366ffd640 1 -- 172.21.15.116:0/2054746529 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36803a8f0 con 0x7fa3780b0b10 2023-11-28T17:10:54.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.881+0000 7fa366ffd640 1 -- 172.21.15.116:0/2054746529 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36803aa70 con 0x7fa3780b0b10 2023-11-28T17:10:54.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.881+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2054746529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa378142b30 con 0x7fa3780b0b10 2023-11-28T17:10:54.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.881+0000 7fa366ffd640 1 -- 172.21.15.116:0/2054746529 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36803ae20 con 0x7fa3780b0b10 2023-11-28T17:10:54.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.881+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2054746529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3781430f0 con 0x7fa3780b0b10 2023-11-28T17:10:54.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.883+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2054746529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3781432e0 con 0x7fa3780b0b10 2023-11-28T17:10:54.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.883+0000 7fa366ffd640 1 -- 172.21.15.116:0/2054746529 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa36804d440 con 0x7fa3780b0b10 2023-11-28T17:10:54.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.884+0000 7fa366ffd640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa35806b2a0 0x7fa35806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:54.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.884+0000 7fa366ffd640 1 -- 172.21.15.116:0/2054746529 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa3680abd30 con 0x7fa3780b0b10 2023-11-28T17:10:54.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.885+0000 7fa375d9b640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa35806b2a0 0x7fa35806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:54.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.885+0000 7fa375d9b640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa35806b2a0 0x7fa35806d760 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa378132b10 tx=0x7fa36c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:54.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:54.889+0000 7fa366ffd640 1 -- 172.21.15.116:0/2054746529 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa368076b20 con 0x7fa3780b0b10 2023-11-28T17:10:55.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.080+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2054746529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa378069360 con 0x7fa3780b0b10 2023-11-28T17:10:55.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.080+0000 7fa366ffd640 1 -- 172.21.15.116:0/2054746529 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa3680350a0 con 0x7fa3780b0b10 2023-11-28T17:10:55.082 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:10:55.084 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.083+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2054746529 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa35806b2a0 msgr2=0x7fa35806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:55.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.083+0000 7fa37c8ee640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa35806b2a0 0x7fa35806d760 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa378132b10 tx=0x7fa36c008040 comp rx=0 tx=0).stop 2023-11-28T17:10:55.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.083+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2054746529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780b0b10 msgr2=0x7fa378130ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:55.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.083+0000 7fa37c8ee640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780b0b10 0x7fa378130ab0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa368034040 tx=0x7fa3680046c0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.083+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2054746529 shutdown_connections 2023-11-28T17:10:55.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.083+0000 7fa37c8ee640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa35806b2a0 0x7fa35806d760 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.083+0000 7fa37c8ee640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378146990 0x7fa378145670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.083+0000 7fa37c8ee640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3780b1e80 0x7fa378132db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.083+0000 7fa37c8ee640 1 --2- 172.21.15.116:0/2054746529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3780b0b10 0x7fa378130ab0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.083+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2054746529 >> 172.21.15.116:0/2054746529 conn(0x7fa37806a4c0 msgr2=0x7fa378070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.084+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2054746529 shutdown_connections 2023-11-28T17:10:55.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.084+0000 7fa37c8ee640 1 -- 172.21.15.116:0/2054746529 wait complete. 2023-11-28T17:10:55.104 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:10:55.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.309+0000 7f926340e640 1 Processor -- start 2023-11-28T17:10:55.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.310+0000 7f926340e640 1 -- start start 2023-11-28T17:10:55.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.311+0000 7f926340e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f925c072250 0x7f925c15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:55.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.311+0000 7f926340e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925c0cd320 con 0x7f925c15b0c0 2023-11-28T17:10:55.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.311+0000 7f926340e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925c0ced90 con 0x7f925c0ce940 2023-11-28T17:10:55.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.311+0000 7f926340e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f925c0cc710 con 0x7f925c072250 2023-11-28T17:10:55.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.311+0000 7f92609a9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f925c072250 0x7f925c15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:55.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.311+0000 7f92611aa640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f925c15b0c0 0x7f925c071b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34482/0 (socket says 172.21.15.116:34482) 2023-11-28T17:10:55.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.311+0000 7f92611aa640 1 -- 172.21.15.116:0/2785376079 learned_addr learned my addr 172.21.15.116:0/2785376079 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:55.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.312+0000 7f92609a9640 1 -- 172.21.15.116:0/2785376079 >> v1:172.21.15.156:6789/0 conn(0x7f925c0ce940 legacy=0x7f925c15b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:55.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.312+0000 7f92609a9640 1 -- 172.21.15.116:0/2785376079 >> v1:172.21.15.116:6789/0 conn(0x7f925c15b0c0 legacy=0x7f925c071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:55.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.312+0000 7f92609a9640 1 -- 172.21.15.116:0/2785376079 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c0cc9f0 con 0x7f925c072250 2023-11-28T17:10:55.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.312+0000 7f9253fff640 1 -- 172.21.15.116:0/2785376079 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703478342 0 0) 0x7f925c0cd320 con 0x7f925c15b0c0 2023-11-28T17:10:55.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.313+0000 7f92609a9640 1 --2- 172.21.15.116:0/2785376079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f925c072250 0x7f925c15ed10 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f924c0097a0 tx=0x7f924c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa9e3c676abc331 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:55.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.313+0000 7f9253fff640 1 -- 172.21.15.116:0/2785376079 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924c02e070 con 0x7f925c072250 2023-11-28T17:10:55.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.313+0000 7f9253fff640 1 -- 172.21.15.116:0/2785376079 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924c0315a0 con 0x7f925c072250 2023-11-28T17:10:55.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.313+0000 7f9253fff640 1 -- 172.21.15.116:0/2785376079 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924c031930 con 0x7f925c072250 2023-11-28T17:10:55.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.314+0000 7f926340e640 1 -- 172.21.15.116:0/2785376079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f925c072250 msgr2=0x7f925c15ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:55.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.314+0000 7f926340e640 1 --2- 172.21.15.116:0/2785376079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f925c072250 0x7f925c15ed10 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f924c0097a0 tx=0x7f924c02b840 comp rx=0 tx=0).stop 2023-11-28T17:10:55.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.314+0000 7f926340e640 1 -- 172.21.15.116:0/2785376079 shutdown_connections 2023-11-28T17:10:55.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.314+0000 7f926340e640 1 --2- 172.21.15.116:0/2785376079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f925c072250 0x7f925c15ed10 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.314+0000 7f926340e640 1 -- 172.21.15.116:0/2785376079 >> 172.21.15.116:0/2785376079 conn(0x7f925c0b0cf0 msgr2=0x7f925c0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:55.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.314+0000 7f926340e640 1 -- 172.21.15.116:0/2785376079 shutdown_connections 2023-11-28T17:10:55.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.314+0000 7f926340e640 1 -- 172.21.15.116:0/2785376079 wait complete. 2023-11-28T17:10:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.316+0000 7f926340e640 1 Processor -- start 2023-11-28T17:10:55.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.316+0000 7f926340e640 1 -- start start 2023-11-28T17:10:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.317+0000 7f926340e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f925c072250 0x7f925c1e6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:55.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.317+0000 7f926340e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f925c0ce940 0x7f925c1e76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:55.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.317+0000 7f926340e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f925c15b0c0 0x7f925c154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:55.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.317+0000 7f926340e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c1e7c00 con 0x7f925c15b0c0 2023-11-28T17:10:55.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.317+0000 7f926340e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c1e7e60 con 0x7f925c0ce940 2023-11-28T17:10:55.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.317+0000 7f926340e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925c1e6110 con 0x7f925c072250 2023-11-28T17:10:55.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.317+0000 7f92619ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f925c15b0c0 0x7f925c154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:55.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.317+0000 7f92619ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f925c15b0c0 0x7f925c154980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53546/0 (socket says 172.21.15.116:53546) 2023-11-28T17:10:55.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.317+0000 7f92619ab640 1 -- 172.21.15.116:0/1075099274 learned_addr learned my addr 172.21.15.116:0/1075099274 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:55.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.318+0000 7f92611aa640 1 --2- 172.21.15.116:0/1075099274 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f925c072250 0x7f925c1e6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:55.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.318+0000 7f92609a9640 1 --2- 172.21.15.116:0/1075099274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f925c0ce940 0x7f925c1e76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:55.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.318+0000 7f92619ab640 1 -- 172.21.15.116:0/1075099274 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f925c072250 msgr2=0x7f925c1e6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:55.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.318+0000 7f92619ab640 1 --2- 172.21.15.116:0/1075099274 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f925c072250 0x7f925c1e6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.318+0000 7f92619ab640 1 -- 172.21.15.116:0/1075099274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f925c0ce940 msgr2=0x7f925c1e76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:55.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.318+0000 7f92619ab640 1 --2- 172.21.15.116:0/1075099274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f925c0ce940 0x7f925c1e76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.318+0000 7f92619ab640 1 -- 172.21.15.116:0/1075099274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924c009400 con 0x7f925c15b0c0 2023-11-28T17:10:55.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.318+0000 7f92619ab640 1 --2- 172.21.15.116:0/1075099274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f925c15b0c0 0x7f925c154980 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f925400c780 tx=0x7f925400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:55.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.318+0000 7f9251ffb640 1 -- 172.21.15.116:0/1075099274 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9254004b60 con 0x7f925c15b0c0 2023-11-28T17:10:55.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.318+0000 7f9251ffb640 1 -- 172.21.15.116:0/1075099274 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925400ce90 con 0x7f925c15b0c0 2023-11-28T17:10:55.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.318+0000 7f9251ffb640 1 -- 172.21.15.116:0/1075099274 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925400a850 con 0x7f925c15b0c0 2023-11-28T17:10:55.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.318+0000 7f926340e640 1 -- 172.21.15.116:0/1075099274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925c15b620 con 0x7f925c15b0c0 2023-11-28T17:10:55.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.319+0000 7f926340e640 1 -- 172.21.15.116:0/1075099274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925c1e6690 con 0x7f925c15b0c0 2023-11-28T17:10:55.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.320+0000 7f926340e640 1 -- 172.21.15.116:0/1075099274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925c069360 con 0x7f925c15b0c0 2023-11-28T17:10:55.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.321+0000 7f9251ffb640 1 -- 172.21.15.116:0/1075099274 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f925400a9b0 con 0x7f925c15b0c0 2023-11-28T17:10:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.321+0000 7f9251ffb640 1 --2- 172.21.15.116:0/1075099274 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f924006b2a0 0x7f924006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.322+0000 7f9251ffb640 1 -- 172.21.15.116:0/1075099274 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9254089d80 con 0x7f925c15b0c0 2023-11-28T17:10:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.322+0000 7f92611aa640 1 --2- 172.21.15.116:0/1075099274 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f924006b2a0 0x7f924006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:55.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.322+0000 7f92611aa640 1 --2- 172.21.15.116:0/1075099274 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f924006b2a0 0x7f924006d760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9258000f60 tx=0x7f9258007b70 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:55.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.326+0000 7f9251ffb640 1 -- 172.21.15.116:0/1075099274 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9254054c00 con 0x7f925c15b0c0 2023-11-28T17:10:55.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.516+0000 7f926340e640 1 -- 172.21.15.116:0/1075099274 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f925c0b1990 con 0x7f925c15b0c0 2023-11-28T17:10:55.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.516+0000 7f9251ffb640 1 -- 172.21.15.116:0/1075099274 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9254058220 con 0x7f925c15b0c0 2023-11-28T17:10:55.518 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:10:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.519+0000 7f926340e640 1 -- 172.21.15.116:0/1075099274 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f924006b2a0 msgr2=0x7f924006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.519+0000 7f926340e640 1 --2- 172.21.15.116:0/1075099274 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f924006b2a0 0x7f924006d760 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9258000f60 tx=0x7f9258007b70 comp rx=0 tx=0).stop 2023-11-28T17:10:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.519+0000 7f926340e640 1 -- 172.21.15.116:0/1075099274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f925c15b0c0 msgr2=0x7f925c154980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.519+0000 7f926340e640 1 --2- 172.21.15.116:0/1075099274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f925c15b0c0 0x7f925c154980 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f925400c780 tx=0x7f925400cc40 comp rx=0 tx=0).stop 2023-11-28T17:10:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.519+0000 7f926340e640 1 -- 172.21.15.116:0/1075099274 shutdown_connections 2023-11-28T17:10:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.519+0000 7f926340e640 1 --2- 172.21.15.116:0/1075099274 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f924006b2a0 0x7f924006d760 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.519+0000 7f926340e640 1 --2- 172.21.15.116:0/1075099274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f925c15b0c0 0x7f925c154980 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.519+0000 7f926340e640 1 --2- 172.21.15.116:0/1075099274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f925c0ce940 0x7f925c1e76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.519+0000 7f926340e640 1 --2- 172.21.15.116:0/1075099274 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f925c072250 0x7f925c1e6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.519+0000 7f926340e640 1 -- 172.21.15.116:0/1075099274 >> 172.21.15.116:0/1075099274 conn(0x7f925c0b0cf0 msgr2=0x7f925c0ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.519+0000 7f926340e640 1 -- 172.21.15.116:0/1075099274 shutdown_connections 2023-11-28T17:10:55.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.519+0000 7f926340e640 1 -- 172.21.15.116:0/1075099274 wait complete. 2023-11-28T17:10:55.539 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:10:55.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.754+0000 7f5c63bc9640 1 Processor -- start 2023-11-28T17:10:55.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.754+0000 7f5c63bc9640 1 -- start start 2023-11-28T17:10:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.756+0000 7f5c63bc9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5c5c15b0c0 0x7f5c5c15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.756+0000 7f5c63bc9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c5c0ce940 0x7f5c5c0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.756+0000 7f5c63bc9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5c0cd320 con 0x7f5c5c071b40 2023-11-28T17:10:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.756+0000 7f5c63bc9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c5c0ced80 con 0x7f5c5c15b0c0 2023-11-28T17:10:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.756+0000 7f5c63bc9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c0cbee0 con 0x7f5c5c0ce940 2023-11-28T17:10:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.756+0000 7f5c62166640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5c5c071b40 0x7f5c5c15b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34488/0 (socket says 172.21.15.116:34488) 2023-11-28T17:10:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.756+0000 7f5c62166640 1 -- 172.21.15.116:0/1021818372 learned_addr learned my addr 172.21.15.116:0/1021818372 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.756+0000 7f5c61965640 1 --2- 172.21.15.116:0/1021818372 >> v2:172.21.15.116:3300/0 conn(0x7f5c5c15b0c0 0x7f5c5c15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.757+0000 7f5c61164640 1 --2- 172.21.15.116:0/1021818372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c5c0ce940 0x7f5c5c0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.757+0000 7f5c60963640 1 -- 172.21.15.116:0/1021818372 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766344049 0 0) 0x7f5c5c0cd320 con 0x7f5c5c071b40 2023-11-28T17:10:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.757+0000 7f5c60963640 1 -- 172.21.15.116:0/1021818372 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c48003520 con 0x7f5c5c071b40 2023-11-28T17:10:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.757+0000 7f5c61965640 1 -- 172.21.15.116:0/1021818372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c5c0ce940 msgr2=0x7f5c5c0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.757+0000 7f5c61965640 1 --2- 172.21.15.116:0/1021818372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c5c0ce940 0x7f5c5c0cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.757+0000 7f5c61965640 1 -- 172.21.15.116:0/1021818372 >> v1:172.21.15.116:6789/0 conn(0x7f5c5c071b40 legacy=0x7f5c5c15b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.757+0000 7f5c61965640 1 -- 172.21.15.116:0/1021818372 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c0cc710 con 0x7f5c5c15b0c0 2023-11-28T17:10:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.758+0000 7f5c61965640 1 --2- 172.21.15.116:0/1021818372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c5c15b0c0 0x7f5c5c15ed10 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f5c580097a0 tx=0x7f5c5802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959806a9095cccd7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.759+0000 7f5c60963640 1 -- 172.21.15.116:0/1021818372 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5802e070 con 0x7f5c5c15b0c0 2023-11-28T17:10:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.759+0000 7f5c60963640 1 -- 172.21.15.116:0/1021818372 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c580315a0 con 0x7f5c5c15b0c0 2023-11-28T17:10:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.759+0000 7f5c60963640 1 -- 172.21.15.116:0/1021818372 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c58031930 con 0x7f5c5c15b0c0 2023-11-28T17:10:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.759+0000 7f5c63bc9640 1 -- 172.21.15.116:0/1021818372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c5c15b0c0 msgr2=0x7f5c5c15ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.759+0000 7f5c63bc9640 1 --2- 172.21.15.116:0/1021818372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c5c15b0c0 0x7f5c5c15ed10 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f5c580097a0 tx=0x7f5c5802b840 comp rx=0 tx=0).stop 2023-11-28T17:10:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.759+0000 7f5c63bc9640 1 -- 172.21.15.116:0/1021818372 shutdown_connections 2023-11-28T17:10:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.759+0000 7f5c63bc9640 1 --2- 172.21.15.116:0/1021818372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c5c0ce940 0x7f5c5c0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.759+0000 7f5c63bc9640 1 --2- 172.21.15.116:0/1021818372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c5c15b0c0 0x7f5c5c15ed10 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.759+0000 7f5c63bc9640 1 -- 172.21.15.116:0/1021818372 >> 172.21.15.116:0/1021818372 conn(0x7f5c5c0b0cf0 msgr2=0x7f5c5c0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.759+0000 7f5c63bc9640 1 -- 172.21.15.116:0/1021818372 shutdown_connections 2023-11-28T17:10:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.759+0000 7f5c63bc9640 1 -- 172.21.15.116:0/1021818372 wait complete. 2023-11-28T17:10:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.761+0000 7f5c63bc9640 1 Processor -- start 2023-11-28T17:10:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.761+0000 7f5c63bc9640 1 -- start start 2023-11-28T17:10:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.762+0000 7f5c63bc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c5c071b40 0x7f5c5c1f0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.762+0000 7f5c63bc9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c5c0ce940 0x7f5c5c0aa440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.762+0000 7f5c63bc9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c5c15b0c0 0x7f5c5c0aa980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.762+0000 7f5c63bc9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c0ade00 con 0x7f5c5c071b40 2023-11-28T17:10:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.762+0000 7f5c63bc9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c15a420 con 0x7f5c5c15b0c0 2023-11-28T17:10:55.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.762+0000 7f5c63bc9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5c0ad180 con 0x7f5c5c0ce940 2023-11-28T17:10:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.763+0000 7f5c61965640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c5c071b40 0x7f5c5c1f0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.763+0000 7f5c61965640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c5c071b40 0x7f5c5c1f0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53558/0 (socket says 172.21.15.116:53558) 2023-11-28T17:10:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.763+0000 7f5c61965640 1 -- 172.21.15.116:0/4115497662 learned_addr learned my addr 172.21.15.116:0/4115497662 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.763+0000 7f5c61164640 1 --2- 172.21.15.116:0/4115497662 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c5c0ce940 0x7f5c5c0aa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.763+0000 7f5c62166640 1 --2- 172.21.15.116:0/4115497662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c5c15b0c0 0x7f5c5c0aa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.763+0000 7f5c61965640 1 -- 172.21.15.116:0/4115497662 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c5c0ce940 msgr2=0x7f5c5c0aa440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.763+0000 7f5c61965640 1 --2- 172.21.15.116:0/4115497662 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c5c0ce940 0x7f5c5c0aa440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.763+0000 7f5c61965640 1 -- 172.21.15.116:0/4115497662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c5c15b0c0 msgr2=0x7f5c5c0aa980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.763+0000 7f5c61965640 1 --2- 172.21.15.116:0/4115497662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c5c15b0c0 0x7f5c5c0aa980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.763+0000 7f5c61965640 1 -- 172.21.15.116:0/4115497662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c58009400 con 0x7f5c5c071b40 2023-11-28T17:10:55.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.763+0000 7f5c61965640 1 --2- 172.21.15.116:0/4115497662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c5c071b40 0x7f5c5c1f0a10 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f5c58034040 tx=0x7f5c580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:55.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.764+0000 7f5c527fc640 1 -- 172.21.15.116:0/4115497662 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c58032840 con 0x7f5c5c071b40 2023-11-28T17:10:55.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.764+0000 7f5c527fc640 1 -- 172.21.15.116:0/4115497662 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c58031f00 con 0x7f5c5c071b40 2023-11-28T17:10:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.764+0000 7f5c63bc9640 1 -- 172.21.15.116:0/4115497662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5c0ad460 con 0x7f5c5c071b40 2023-11-28T17:10:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.764+0000 7f5c527fc640 1 -- 172.21.15.116:0/4115497662 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c58032be0 con 0x7f5c5c071b40 2023-11-28T17:10:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.764+0000 7f5c63bc9640 1 -- 172.21.15.116:0/4115497662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5c0ada20 con 0x7f5c5c071b40 2023-11-28T17:10:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.765+0000 7f5c63bc9640 1 -- 172.21.15.116:0/4115497662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5c069360 con 0x7f5c5c071b40 2023-11-28T17:10:55.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.766+0000 7f5c527fc640 1 -- 172.21.15.116:0/4115497662 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f5c58032d40 con 0x7f5c5c071b40 2023-11-28T17:10:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.767+0000 7f5c527fc640 1 --2- 172.21.15.116:0/4115497662 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5c4006b2a0 0x7f5c4006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.768+0000 7f5c527fc640 1 -- 172.21.15.116:0/4115497662 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5c580abc30 con 0x7f5c5c071b40 2023-11-28T17:10:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.768+0000 7f5c61164640 1 --2- 172.21.15.116:0/4115497662 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5c4006b2a0 0x7f5c4006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.768+0000 7f5c61164640 1 --2- 172.21.15.116:0/4115497662 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5c4006b2a0 0x7f5c4006d760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c1f4460 tx=0x7f5c4c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.772+0000 7f5c527fc640 1 -- 172.21.15.116:0/4115497662 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c58076980 con 0x7f5c5c071b40 2023-11-28T17:10:55.965 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.964+0000 7f5c63bc9640 1 -- 172.21.15.116:0/4115497662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5c5c0b12e0 con 0x7f5c5c071b40 2023-11-28T17:10:55.966 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.965+0000 7f5c527fc640 1 -- 172.21.15.116:0/4115497662 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5c58079fa0 con 0x7f5c5c071b40 2023-11-28T17:10:55.966 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:10:55.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.967+0000 7f5c63bc9640 1 -- 172.21.15.116:0/4115497662 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5c4006b2a0 msgr2=0x7f5c4006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:55.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.967+0000 7f5c63bc9640 1 --2- 172.21.15.116:0/4115497662 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5c4006b2a0 0x7f5c4006d760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c1f4460 tx=0x7f5c4c008040 comp rx=0 tx=0).stop 2023-11-28T17:10:55.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.967+0000 7f5c63bc9640 1 -- 172.21.15.116:0/4115497662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c5c071b40 msgr2=0x7f5c5c1f0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:55.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.967+0000 7f5c63bc9640 1 --2- 172.21.15.116:0/4115497662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c5c071b40 0x7f5c5c1f0a10 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f5c58034040 tx=0x7f5c580046c0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.968+0000 7f5c63bc9640 1 -- 172.21.15.116:0/4115497662 shutdown_connections 2023-11-28T17:10:55.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.968+0000 7f5c63bc9640 1 --2- 172.21.15.116:0/4115497662 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5c4006b2a0 0x7f5c4006d760 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.969 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.968+0000 7f5c63bc9640 1 --2- 172.21.15.116:0/4115497662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c5c15b0c0 0x7f5c5c0aa980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.968+0000 7f5c63bc9640 1 --2- 172.21.15.116:0/4115497662 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5c5c0ce940 0x7f5c5c0aa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.968+0000 7f5c63bc9640 1 --2- 172.21.15.116:0/4115497662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c5c071b40 0x7f5c5c1f0a10 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:55.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.968+0000 7f5c63bc9640 1 -- 172.21.15.116:0/4115497662 >> 172.21.15.116:0/4115497662 conn(0x7f5c5c0b0cf0 msgr2=0x7f5c5c155680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:55.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.968+0000 7f5c63bc9640 1 -- 172.21.15.116:0/4115497662 shutdown_connections 2023-11-28T17:10:55.970 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:55.968+0000 7f5c63bc9640 1 -- 172.21.15.116:0/4115497662 wait complete. 2023-11-28T17:10:55.989 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:10:56.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.201+0000 7f34a8cdc640 1 Processor -- start 2023-11-28T17:10:56.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.201+0000 7f34a8cdc640 1 -- start start 2023-11-28T17:10:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.203+0000 7f34a8cdc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34a415ae20 0x7f34a406ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.203+0000 7f34a8cdc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f34a40cf2a0 0x7f34a40ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.203+0000 7f34a8cdc640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a4070560 con 0x7f34a40cdf90 2023-11-28T17:10:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.203+0000 7f34a8cdc640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34a4070840 con 0x7f34a40cf2a0 2023-11-28T17:10:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.203+0000 7f34a8cdc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a4070b20 con 0x7f34a415ae20 2023-11-28T17:10:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.203+0000 7f34a3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f34a40cf2a0 0x7f34a40ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.203+0000 7f349bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f34a40cdf90 0x7f34a40ceb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34490/0 (socket says 172.21.15.116:34490) 2023-11-28T17:10:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.203+0000 7f349bfff640 1 -- 172.21.15.116:0/2197222625 learned_addr learned my addr 172.21.15.116:0/2197222625 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.204+0000 7f34a37fe640 1 --2- 172.21.15.116:0/2197222625 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34a415ae20 0x7f34a406ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.204+0000 7f34a3fff640 1 -- 172.21.15.116:0/2197222625 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34a415ae20 msgr2=0x7f34a406ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.204+0000 7f34a3fff640 1 --2- 172.21.15.116:0/2197222625 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34a415ae20 0x7f34a406ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.204+0000 7f34a3fff640 1 -- 172.21.15.116:0/2197222625 >> v1:172.21.15.116:6789/0 conn(0x7f34a40cdf90 legacy=0x7f34a40ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.204+0000 7f34a3fff640 1 -- 172.21.15.116:0/2197222625 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a4070e00 con 0x7f34a40cf2a0 2023-11-28T17:10:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.204+0000 7f34a2ffd640 1 -- 172.21.15.116:0/2197222625 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323889485 0 0) 0x7f34a4070560 con 0x7f34a40cdf90 2023-11-28T17:10:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.205+0000 7f34a3fff640 1 --2- 172.21.15.116:0/2197222625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a40cf2a0 0x7f34a40ce3f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f349c009dc0 tx=0x7f349c02bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c728eca1853cfd server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.205+0000 7f34a2ffd640 1 -- 172.21.15.116:0/2197222625 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c02cba0 con 0x7f34a40cf2a0 2023-11-28T17:10:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.205+0000 7f34a2ffd640 1 -- 172.21.15.116:0/2197222625 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c02cd00 con 0x7f34a40cf2a0 2023-11-28T17:10:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.205+0000 7f34a2ffd640 1 -- 172.21.15.116:0/2197222625 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c034660 con 0x7f34a40cf2a0 2023-11-28T17:10:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.206+0000 7f34a8cdc640 1 -- 172.21.15.116:0/2197222625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a40cf2a0 msgr2=0x7f34a40ce3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.206+0000 7f34a8cdc640 1 --2- 172.21.15.116:0/2197222625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a40cf2a0 0x7f34a40ce3f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f349c009dc0 tx=0x7f349c02bdf0 comp rx=0 tx=0).stop 2023-11-28T17:10:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.206+0000 7f34a8cdc640 1 -- 172.21.15.116:0/2197222625 shutdown_connections 2023-11-28T17:10:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.206+0000 7f34a8cdc640 1 --2- 172.21.15.116:0/2197222625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a40cf2a0 0x7f34a40ce3f0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.206+0000 7f34a8cdc640 1 --2- 172.21.15.116:0/2197222625 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34a415ae20 0x7f34a406ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.206+0000 7f34a8cdc640 1 -- 172.21.15.116:0/2197222625 >> 172.21.15.116:0/2197222625 conn(0x7f34a40b0950 msgr2=0x7f34a40b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.206+0000 7f34a8cdc640 1 -- 172.21.15.116:0/2197222625 shutdown_connections 2023-11-28T17:10:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.206+0000 7f34a8cdc640 1 -- 172.21.15.116:0/2197222625 wait complete. 2023-11-28T17:10:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.208+0000 7f34a8cdc640 1 Processor -- start 2023-11-28T17:10:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.208+0000 7f34a8cdc640 1 -- start start 2023-11-28T17:10:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.209+0000 7f34a8cdc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34a40cdf90 0x7f34a40b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:56.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.209+0000 7f34a8cdc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34a40cf2a0 0x7f34a415c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:56.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.209+0000 7f34a8cdc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a415ae20 0x7f34a40c9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:56.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.209+0000 7f34a8cdc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a415d7c0 con 0x7f34a415ae20 2023-11-28T17:10:56.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.209+0000 7f34a8cdc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a415cbb0 con 0x7f34a40cf2a0 2023-11-28T17:10:56.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.209+0000 7f34a8cdc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a415ce90 con 0x7f34a40cdf90 2023-11-28T17:10:56.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.209+0000 7f34a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a415ae20 0x7f34a40c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:56.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.209+0000 7f34a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a415ae20 0x7f34a40c9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53584/0 (socket says 172.21.15.116:53584) 2023-11-28T17:10:56.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.209+0000 7f34a3fff640 1 -- 172.21.15.116:0/398751490 learned_addr learned my addr 172.21.15.116:0/398751490 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:56.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.209+0000 7f34a3fff640 1 -- 172.21.15.116:0/398751490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34a40cdf90 msgr2=0x7f34a40b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:56.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.209+0000 7f34a3fff640 1 --2- 172.21.15.116:0/398751490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34a40cdf90 0x7f34a40b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:56.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.209+0000 7f34a3fff640 1 -- 172.21.15.116:0/398751490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34a40cf2a0 msgr2=0x7f34a415c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:56.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.209+0000 7f34a3fff640 1 --2- 172.21.15.116:0/398751490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34a40cf2a0 0x7f34a415c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:56.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.209+0000 7f34a3fff640 1 -- 172.21.15.116:0/398751490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c009a70 con 0x7f34a415ae20 2023-11-28T17:10:56.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.210+0000 7f34a3fff640 1 --2- 172.21.15.116:0/398751490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a415ae20 0x7f34a40c9110 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f349c037040 tx=0x7f349c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:56.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.210+0000 7f34a0ff9640 1 -- 172.21.15.116:0/398751490 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c040650 con 0x7f34a415ae20 2023-11-28T17:10:56.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.210+0000 7f34a0ff9640 1 -- 172.21.15.116:0/398751490 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c0407b0 con 0x7f34a415ae20 2023-11-28T17:10:56.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.210+0000 7f34a8cdc640 1 -- 172.21.15.116:0/398751490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a415b380 con 0x7f34a415ae20 2023-11-28T17:10:56.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.210+0000 7f34a0ff9640 1 -- 172.21.15.116:0/398751490 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c040b60 con 0x7f34a415ae20 2023-11-28T17:10:56.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.210+0000 7f34a8cdc640 1 -- 172.21.15.116:0/398751490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a415d450 con 0x7f34a415ae20 2023-11-28T17:10:56.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.212+0000 7f34a8cdc640 1 -- 172.21.15.116:0/398751490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a406d190 con 0x7f34a415ae20 2023-11-28T17:10:56.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.212+0000 7f34a0ff9640 1 -- 172.21.15.116:0/398751490 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f349c040cc0 con 0x7f34a415ae20 2023-11-28T17:10:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.214+0000 7f34a0ff9640 1 --2- 172.21.15.116:0/398751490 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f348406b2a0 0x7f348406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.214+0000 7f34a0ff9640 1 -- 172.21.15.116:0/398751490 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f349c0aca20 con 0x7f34a415ae20 2023-11-28T17:10:56.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.214+0000 7f34a37fe640 1 --2- 172.21.15.116:0/398751490 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f348406b2a0 0x7f348406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:56.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.215+0000 7f34a37fe640 1 --2- 172.21.15.116:0/398751490 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f348406b2a0 0x7f348406d760 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3494006fd0 tx=0x7f3494008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:56.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.219+0000 7f34a0ff9640 1 -- 172.21.15.116:0/398751490 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f349c077810 con 0x7f34a415ae20 2023-11-28T17:10:56.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.419+0000 7f34a8cdc640 1 -- 172.21.15.116:0/398751490 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f34a409d0b0 con 0x7f34a415ae20 2023-11-28T17:10:56.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.420+0000 7f34a0ff9640 1 -- 172.21.15.116:0/398751490 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) v1 ==== 80+0+0 (secure 0 0 0) 0x7f349c07ae30 con 0x7f34a415ae20 2023-11-28T17:10:56.421 INFO:teuthology.orchestra.run.smithi116.stderr:listed 0 entries 2023-11-28T17:10:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.422+0000 7f34a8cdc640 1 -- 172.21.15.116:0/398751490 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f348406b2a0 msgr2=0x7f348406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.422+0000 7f34a8cdc640 1 --2- 172.21.15.116:0/398751490 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f348406b2a0 0x7f348406d760 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3494006fd0 tx=0x7f3494008040 comp rx=0 tx=0).stop 2023-11-28T17:10:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.422+0000 7f34a8cdc640 1 -- 172.21.15.116:0/398751490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a415ae20 msgr2=0x7f34a40c9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.422+0000 7f34a8cdc640 1 --2- 172.21.15.116:0/398751490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a415ae20 0x7f34a40c9110 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f349c037040 tx=0x7f349c004a40 comp rx=0 tx=0).stop 2023-11-28T17:10:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.422+0000 7f34a8cdc640 1 -- 172.21.15.116:0/398751490 shutdown_connections 2023-11-28T17:10:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.422+0000 7f34a8cdc640 1 --2- 172.21.15.116:0/398751490 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f348406b2a0 0x7f348406d760 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.422+0000 7f34a8cdc640 1 --2- 172.21.15.116:0/398751490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34a415ae20 0x7f34a40c9110 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.422+0000 7f34a8cdc640 1 --2- 172.21.15.116:0/398751490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34a40cf2a0 0x7f34a415c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.422+0000 7f34a8cdc640 1 --2- 172.21.15.116:0/398751490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34a40cdf90 0x7f34a40b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.422+0000 7f34a8cdc640 1 -- 172.21.15.116:0/398751490 >> 172.21.15.116:0/398751490 conn(0x7f34a40b0950 msgr2=0x7f34a40ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.423+0000 7f34a8cdc640 1 -- 172.21.15.116:0/398751490 shutdown_connections 2023-11-28T17:10:56.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:56.423+0000 7f34a8cdc640 1 -- 172.21.15.116:0/398751490 wait complete. 2023-11-28T17:10:56.441 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2023-11-28T17:10:56.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:56.442 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-11-28T17:10:56.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[104085]: fuse finished with error 0 and tester_r 0 2023-11-28T17:10:56.529 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:10:56.569 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:10:56.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:56.570 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T17:10:56.623 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2023-11-28T17:10:56.842 INFO:teuthology.orchestra.run.smithi116.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":94,"msgr_send_messages":74,"msgr_recv_bytes":124982,"msgr_send_bytes":39579,"msgr_created_connections":9,"msgr_active_connections":1,"msgr_running_total_time":0.024954521,"msgr_running_send_time":0.006581371,"msgr_running_recv_time":0.017012997,"msgr_running_fast_dispatch_time":0.001727907,"msgr_send_messages_queue_lat":{"avgcount":74,"sum":0.066902199,"avgtime":0.000904083},"msgr_handle_ack_lat":{"avgcount":37,"sum":0.000007392,"avgtime":0.000000199},"msgr_recv_encrypted_bytes":115584,"msgr_send_encrypted_bytes":15808},"AsyncMessenger::Worker-1":{"msgr_recv_messages":34,"msgr_send_messages":35,"msgr_recv_bytes":10239,"msgr_send_bytes":117359,"msgr_created_connections":6,"msgr_active_connections":3,"msgr_running_total_time":0.011813388,"msgr_running_send_time":0.002798380,"msgr_running_recv_time":0.007476507,"msgr_running_fast_dispatch_time":0.001150231,"msgr_send_messages_queue_lat":{"avgcount":34,"sum":0.171487153,"avgtime":0.005043739},"msgr_handle_ack_lat":{"avgcount":19,"sum":0.000005149,"avgtime":0.000000271},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":102576},"AsyncMessenger::Worker-2":{"msgr_recv_messages":29,"msgr_send_messages":30,"msgr_recv_bytes":7387,"msgr_send_bytes":15825,"msgr_created_connections":5,"msgr_active_connections":3,"msgr_running_total_time":0.005855941,"msgr_running_send_time":0.000854075,"msgr_running_recv_time":0.003690054,"msgr_running_fast_dispatch_time":0.000915965,"msgr_send_messages_queue_lat":{"avgcount":29,"sum":0.085185704,"avgtime":0.002937438},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":50,"sum":0.003728026,"avgtime":0.000074560}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2,"sum":0.000020594,"avgtime":0.000010297}},"mds":{"request":4,"reply":4,"reply_latency":{"avgcount":4,"sum":0.001170998,"avgtime":0.000292749},"slow_reply":0,"forward":0,"dir_fetch_complete":0,"dir_fetch_keys":0,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":10,"inodes_top":0,"inodes_bottom":10,"inodes_pin_tail":0,"inodes_pinned":10,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":7,"traverse_hit":7,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":6,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":0,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":0,"handle_client_caps_dirty":0,"handle_client_cap_release":0,"process_request_cap_release":1},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":9,"evex":0,"evtrm":0,"ev":9,"evexg":0,"evexd":0,"segadd":2,"segex":0,"segtrm":0,"seg":2,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4197804,"rdpos":0,"jlat":{"avgcount":8,"sum":0.031413246,"avgtime":0.003926655},"replayed":0},"mds_mem":{"ino":13,"ino+":13,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":10,"dn+":10,"dn-":0,"cap":1,"cap+":3,"cap-":2,"rss":39424,"heap":207196},"mds_server":{"dispatch_client_request":4,"dispatch_server_request":0,"handle_client_request":4,"handle_client_session":12,"handle_peer_request":0,"req_create_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getattr_latency":{"avgcount":3,"sum":0.000807047,"avgtime":0.000269015},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":1,"sum":0.000363951,"avgtime":0.000363951},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":0,"session_add":3,"session_remove":3,"sessions_open":0,"sessions_stale":0,"total_load":3,"average_load":3,"avg_session_uptime":4,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":213659,"buffer_anon_items":62,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5696,"osdmap_items":83,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":51880,"mds_co_items":212,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":48,"op_send_bytes":14800,"op_resend":31,"op_reply":48,"op_latency":{"avgcount":48,"sum":0.969983895,"avgtime":0.020207997},"op_inflight":0,"oplen_avg":{"avgcount":48,"sum":72},"op":48,"op_r":0,"op_w":48,"op_rmw":0,"op_pg":0,"osdop_stat":0,"osdop_create":11,"osdop_read":0,"osdop_write":8,"osdop_writefull":9,"osdop_writesame":0,"osdop_append":0,"osdop_zero":0,"osdop_truncate":0,"osdop_delete":6,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":0,"osdop_setxattr":22,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":36,"map_full":0,"map_inc":3,"osd_sessions":6,"osd_session_open":6,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":0,"omap_del":1},"oft":{"omap_total_objs":1,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":151,"get_sum":124361,"get_or_fail_fail":0,"get_or_fail_success":151,"take":0,"take_sum":0,"put":151,"put_sum":124361,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":48,"take_sum":14800,"put":48,"put_sum":14800,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":48,"take_sum":48,"put":48,"put_sum":48,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x561658c2c6e0":{"val":0,"max":3758096384,"get_started":0,"get":9,"get_sum":3500,"get_or_fail_fail":0,"get_or_fail_success":9,"take":0,"take_sum":0,"put":8,"put_sum":3500,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2023-11-28T17:10:56.844 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 1, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 37, "avgtime": 1.99e-07, "sum": 7.392e-06 }, "msgr_recv_bytes": 124982, "msgr_recv_encrypted_bytes": 115584, "msgr_recv_messages": 94, "msgr_running_fast_dispatch_time": 0.001727907, "msgr_running_recv_time": 0.017012997, "msgr_running_send_time": 0.006581371, "msgr_running_total_time": 0.024954521, "msgr_send_bytes": 39579, "msgr_send_encrypted_bytes": 15808, "msgr_send_messages": 74, "msgr_send_messages_queue_lat": { "avgcount": 74, "avgtime": 0.000904083, "sum": 0.066902199 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 3, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 19, "avgtime": 2.71e-07, "sum": 5.149e-06 }, "msgr_recv_bytes": 10239, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 34, "msgr_running_fast_dispatch_time": 0.001150231, "msgr_running_recv_time": 0.007476507, "msgr_running_send_time": 0.00279838, "msgr_running_total_time": 0.011813388, "msgr_send_bytes": 117359, "msgr_send_encrypted_bytes": 102576, "msgr_send_messages": 35, "msgr_send_messages_queue_lat": { "avgcount": 34, "avgtime": 0.005043739, "sum": 0.171487153 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 3, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 7387, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 29, "msgr_running_fast_dispatch_time": 0.000915965, "msgr_running_recv_time": 0.003690054, "msgr_running_send_time": 0.000854075, "msgr_running_total_time": 0.005855941, "msgr_send_bytes": 15825, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 30, "msgr_send_messages_queue_lat": { "avgcount": 29, "avgtime": 0.002937438, "sum": 0.085185704 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 50, "avgtime": 7.456e-05, "sum": 0.003728026 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2, "avgtime": 1.0297e-05, "sum": 2.0594e-05 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 0, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 0, "dir_fetch_keys": 0, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 0, "handle_client_caps_dirty": 0, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 10, "inodes_bottom": 10, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 10, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 6, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1, "q": 0, "reply": 4, "reply_latency": { "avgcount": 4, "avgtime": 0.000292749, "sum": 0.001170998 }, "request": 4, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 7, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 7, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 9, "evadd": 9, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 8, "avgtime": 0.003926655, "sum": 0.031413246 }, "rdpos": 0, "replayed": 0, "seg": 2, "segadd": 2, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 4197804 }, "mds_mem": { "cap": 1, "cap+": 3, "cap-": 2, "dir": 12, "dir+": 12, "dir-": 0, "dn": 10, "dn+": 10, "dn-": 0, "heap": 207196, "ino": 13, "ino+": 13, "ino-": 0, "rss": 39424 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4, "dispatch_server_request": 0, "handle_client_request": 4, "handle_client_session": 12, "handle_peer_request": 0, "req_create_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getattr_latency": { "avgcount": 3, "avgtime": 0.000269015, "sum": 0.000807047 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 1, "avgtime": 0.000363951, "sum": 0.000363951 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 3, "avg_session_uptime": 4, "mdthresh_evicted": 0, "session_add": 3, "session_count": 0, "session_remove": 3, "sessions_open": 0, "sessions_stale": 0, "total_load": 3 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 213659, "buffer_anon_items": 62, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 51880, "mds_co_items": 212, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5696, "osdmap_items": 83, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 36, "map_full": 0, "map_inc": 3, "omap_del": 1, "omap_rd": 0, "omap_wr": 15, "op": 48, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 48, "avgtime": 0.020207997, "sum": 0.969983895 }, "op_pg": 0, "op_r": 0, "op_reply": 48, "op_resend": 31, "op_rmw": 0, "op_send": 48, "op_send_bytes": 14800, "op_w": 48, "oplen_avg": { "avgcount": 48, "sum": 72 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 6, "osd_sessions": 6, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 11, "osdop_delete": 6, "osdop_getxattr": 0, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 0, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 22, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 0, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 8, "osdop_writefull": 9, "osdop_writesame": 0, "osdop_zero": 0, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 1, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 151, "get_or_fail_fail": 0, "get_or_fail_success": 151, "get_started": 0, "get_sum": 124361, "max": 104857600, "put": 151, "put_sum": 124361, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 48, "put_sum": 14800, "take": 48, "take_sum": 14800, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 48, "put_sum": 48, "take": 48, "take_sum": 48, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x561658c2c6e0": { "get": 9, "get_or_fail_fail": 0, "get_or_fail_success": 9, "get_started": 0, "get_sum": 3500, "max": 3758096384, "put": 8, "put_sum": 3500, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2023-11-28T17:10:56.844 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2023-11-28T17:10:56.845 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T17:10:56.845 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T17:10:56.955 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T17:10:57.079 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T17:10:57.079 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:10:57.079 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:10:57.134 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054299,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T17:10:57.135 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:10:57.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:57.135 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T17:10:57.182 INFO:teuthology.orchestra.run.smithi116.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-11-28T17:10:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:10:57.184 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:10:57.184 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:10:57.243 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T17:10:57.244 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:10:57.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:57.244 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-28T17:10:57.296 INFO:teuthology.orchestra.run.smithi156.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-11-28T17:10:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:10:57.298 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:10:57.298 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:10:57.359 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054299,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T17:10:57.359 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:10:57.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:57.359 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-11-28T17:10:57.411 INFO:teuthology.orchestra.run.smithi116.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-11-28T17:10:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:10:57.412 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:10:57.413 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:10:57.472 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T17:10:57.473 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:10:57.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:10:57.473 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-11-28T17:10:57.528 INFO:teuthology.orchestra.run.smithi156.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-11-28T17:10:57.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:10:57.529 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:10:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.752+0000 7fbb86090640 1 Processor -- start 2023-11-28T17:10:57.753 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.752+0000 7fbb86090640 1 -- start start 2023-11-28T17:10:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.753+0000 7fbb86090640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbb80144430 0x7fbb8015f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.753+0000 7fbb86090640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb800703f0 0x7fbb80071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.753+0000 7fbb86090640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb80070bd0 con 0x7fbb800721b0 2023-11-28T17:10:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.754+0000 7fbb86090640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb80144810 con 0x7fbb80144430 2023-11-28T17:10:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.754+0000 7fbb86090640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb80070eb0 con 0x7fbb800703f0 2023-11-28T17:10:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.754+0000 7fbb7ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbb800721b0 0x7fbb8006c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34498/0 (socket says 172.21.15.116:34498) 2023-11-28T17:10:57.755 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.754+0000 7fbb7ffff640 1 -- 172.21.15.116:0/3739800697 learned_addr learned my addr 172.21.15.116:0/3739800697 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:57.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.754+0000 7fbb7f7fe640 1 --2- 172.21.15.116:0/3739800697 >> v2:172.21.15.156:3300/0 conn(0x7fbb80144430 0x7fbb8015f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:57.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.754+0000 7fbb77fff640 1 --2- 172.21.15.116:0/3739800697 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb800703f0 0x7fbb80071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:57.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.754+0000 7fbb7effd640 1 -- 172.21.15.116:0/3739800697 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322577371 0 0) 0x7fbb80070bd0 con 0x7fbb800721b0 2023-11-28T17:10:57.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.754+0000 7fbb7effd640 1 -- 172.21.15.116:0/3739800697 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb6c003520 con 0x7fbb800721b0 2023-11-28T17:10:57.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb7effd640 1 -- 172.21.15.116:0/3739800697 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2281128126 0 0) 0x7fbb6c003520 con 0x7fbb800721b0 2023-11-28T17:10:57.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb7effd640 1 -- 172.21.15.116:0/3739800697 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb800703f0 msgr2=0x7fbb80071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:57.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb7effd640 1 --2- 172.21.15.116:0/3739800697 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb800703f0 0x7fbb80071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:57.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb7effd640 1 -- 172.21.15.116:0/3739800697 >> v2:172.21.15.156:3300/0 conn(0x7fbb80144430 msgr2=0x7fbb8015f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:57.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb7effd640 1 --2- 172.21.15.116:0/3739800697 >> v2:172.21.15.156:3300/0 conn(0x7fbb80144430 0x7fbb8015f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:57.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb7effd640 1 -- 172.21.15.116:0/3739800697 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb80071190 con 0x7fbb800721b0 2023-11-28T17:10:57.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb7effd640 1 -- 172.21.15.116:0/3739800697 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fbb780019f0 con 0x7fbb800721b0 2023-11-28T17:10:57.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb7effd640 1 -- 172.21.15.116:0/3739800697 >> v1:172.21.15.116:6789/0 conn(0x7fbb800721b0 legacy=0x7fbb8006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:57.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb7effd640 1 --2- 172.21.15.116:0/3739800697 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb800703f0 0x7fbb6c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:57.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb7effd640 1 --2- 172.21.15.116:0/3739800697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c003ef0 0x7fbb6c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:57.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb7effd640 1 --2- 172.21.15.116:0/3739800697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb6c004d00 0x7fbb6c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:57.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb7effd640 1 -- 172.21.15.116:0/3739800697 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb80144810 con 0x7fbb6c003ef0 2023-11-28T17:10:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb7effd640 1 -- 172.21.15.116:0/3739800697 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb80070eb0 con 0x7fbb6c004d00 2023-11-28T17:10:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb7effd640 1 -- 172.21.15.116:0/3739800697 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c00b980 con 0x7fbb800703f0 2023-11-28T17:10:57.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.755+0000 7fbb77fff640 1 --2- 172.21.15.116:0/3739800697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c003ef0 0x7fbb6c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.756+0000 7fbb77fff640 1 -- 172.21.15.116:0/3739800697 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb800703f0 msgr2=0x7fbb6c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.756+0000 7fbb77fff640 1 --2- 172.21.15.116:0/3739800697 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb800703f0 0x7fbb6c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.756+0000 7fbb77fff640 1 -- 172.21.15.116:0/3739800697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb6c004d00 msgr2=0x7fbb6c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:57.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.756+0000 7fbb77fff640 1 --2- 172.21.15.116:0/3739800697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb6c004d00 0x7fbb6c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:57.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.756+0000 7fbb77fff640 1 -- 172.21.15.116:0/3739800697 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c003520 con 0x7fbb6c003ef0 2023-11-28T17:10:57.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.757+0000 7fbb77fff640 1 --2- 172.21.15.116:0/3739800697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c003ef0 0x7fbb6c0047c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009ac0 tx=0x7fbb6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4c19734b073b94 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.757+0000 7fbb7effd640 1 -- 172.21.15.116:0/3739800697 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6802c6d0 con 0x7fbb6c003ef0 2023-11-28T17:10:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.757+0000 7fbb7effd640 1 -- 172.21.15.116:0/3739800697 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6802c830 con 0x7fbb6c003ef0 2023-11-28T17:10:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.757+0000 7fbb7effd640 1 -- 172.21.15.116:0/3739800697 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6802cbc0 con 0x7fbb6c003ef0 2023-11-28T17:10:57.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.758+0000 7fbb86090640 1 -- 172.21.15.116:0/3739800697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c003ef0 msgr2=0x7fbb6c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:57.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.758+0000 7fbb86090640 1 --2- 172.21.15.116:0/3739800697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c003ef0 0x7fbb6c0047c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009ac0 tx=0x7fbb6c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:57.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.758+0000 7fbb7ffff640 1 -- 172.21.15.116:0/3739800697 reap_dead start 2023-11-28T17:10:57.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.758+0000 7fbb86090640 1 -- 172.21.15.116:0/3739800697 shutdown_connections 2023-11-28T17:10:57.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.758+0000 7fbb86090640 1 -- 172.21.15.116:0/3739800697 >> 172.21.15.116:0/3739800697 conn(0x7fbb8006a4c0 msgr2=0x7fbb8006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:57.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.758+0000 7fbb86090640 1 -- 172.21.15.116:0/3739800697 shutdown_connections 2023-11-28T17:10:57.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.758+0000 7fbb86090640 1 -- 172.21.15.116:0/3739800697 wait complete. 2023-11-28T17:10:57.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.759+0000 7fbb86090640 1 Processor -- start 2023-11-28T17:10:57.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.760+0000 7fbb86090640 1 -- start start 2023-11-28T17:10:57.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.760+0000 7fbb86090640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c003ef0 0x7fbb801f17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:57.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb86090640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb8015a3b0 0x7fbb8015d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:57.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb86090640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb8015d5f0 0x7fbb8013b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:57.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb86090640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8015daa0 con 0x7fbb6c003ef0 2023-11-28T17:10:57.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb86090640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb80137f60 con 0x7fbb8015a3b0 2023-11-28T17:10:57.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb86090640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb801f1d20 con 0x7fbb8015d5f0 2023-11-28T17:10:57.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c003ef0 0x7fbb801f17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:57.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c003ef0 0x7fbb801f17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53596/0 (socket says 172.21.15.116:53596) 2023-11-28T17:10:57.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb7f7fe640 1 -- 172.21.15.116:0/3911182696 learned_addr learned my addr 172.21.15.116:0/3911182696 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:57.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb77fff640 1 --2- 172.21.15.116:0/3911182696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb8015a3b0 0x7fbb8015d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:57.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb7f7fe640 1 -- 172.21.15.116:0/3911182696 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb8015d5f0 msgr2=0x7fbb8013b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:57.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb7f7fe640 1 --2- 172.21.15.116:0/3911182696 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb8015d5f0 0x7fbb8013b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:57.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb7f7fe640 1 -- 172.21.15.116:0/3911182696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb8015a3b0 msgr2=0x7fbb8015d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:57.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb7f7fe640 1 --2- 172.21.15.116:0/3911182696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb8015a3b0 0x7fbb8015d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:57.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb7f7fe640 1 -- 172.21.15.116:0/3911182696 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8013d540 con 0x7fbb6c003ef0 2023-11-28T17:10:57.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.761+0000 7fbb77fff640 1 --2- 172.21.15.116:0/3911182696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb8015a3b0 0x7fbb8015d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:57.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.762+0000 7fbb7f7fe640 1 --2- 172.21.15.116:0/3911182696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c003ef0 0x7fbb801f17e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000e860 tx=0x7fbb7000ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:57.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.762+0000 7fbb7cff9640 1 -- 172.21.15.116:0/3911182696 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb70009600 con 0x7fbb6c003ef0 2023-11-28T17:10:57.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.762+0000 7fbb7cff9640 1 -- 172.21.15.116:0/3911182696 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7000c940 con 0x7fbb6c003ef0 2023-11-28T17:10:57.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.762+0000 7fbb7cff9640 1 -- 172.21.15.116:0/3911182696 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7000ccd0 con 0x7fbb6c003ef0 2023-11-28T17:10:57.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.762+0000 7fbb86090640 1 -- 172.21.15.116:0/3911182696 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb8013d820 con 0x7fbb6c003ef0 2023-11-28T17:10:57.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.762+0000 7fbb86090640 1 -- 172.21.15.116:0/3911182696 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb80138490 con 0x7fbb6c003ef0 2023-11-28T17:10:57.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.763+0000 7fbb86090640 1 -- 172.21.15.116:0/3911182696 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb800693d0 con 0x7fbb6c003ef0 2023-11-28T17:10:57.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.764+0000 7fbb7cff9640 1 -- 172.21.15.116:0/3911182696 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fbb70013470 con 0x7fbb6c003ef0 2023-11-28T17:10:57.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.765+0000 7fbb7cff9640 1 --2- 172.21.15.116:0/3911182696 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbb6006b2d0 0x7fbb6006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:57.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.765+0000 7fbb7cff9640 1 -- 172.21.15.116:0/3911182696 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbb70089f80 con 0x7fbb6c003ef0 2023-11-28T17:10:57.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.765+0000 7fbb77fff640 1 --2- 172.21.15.116:0/3911182696 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbb6006b2d0 0x7fbb6006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:57.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.766+0000 7fbb77fff640 1 --2- 172.21.15.116:0/3911182696 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbb6006b2d0 0x7fbb6006d790 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbb68036000 tx=0x7fbb68030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:57.785 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.769+0000 7fbb7cff9640 1 -- 172.21.15.116:0/3911182696 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb70058760 con 0x7fbb6c003ef0 2023-11-28T17:10:57.998 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.997+0000 7fbb86090640 1 -- 172.21.15.116:0/3911182696 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb800cbbc0 con 0x7fbb6c003ef0 2023-11-28T17:10:58.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:57.998+0000 7fbb7cff9640 1 -- 172.21.15.116:0/3911182696 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fbb700582f0 con 0x7fbb6c003ef0 2023-11-28T17:10:58.002 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:58.003 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/491158514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":491158514},{"type":"v1","addr":"172.21.15.156:6834","nonce":491158514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3927645596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3927645596},{"type":"v1","addr":"172.21.15.116:6835","nonce":3927645596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4738,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/141796020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":141796020},{"type":"v1","addr":"172.21.15.156:6835","nonce":141796020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:10:19.725913+0000","modified":"2023-11-28T17:10:22.808343+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6837/3177126349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3177126349},{"type":"v1","addr":"172.21.15.116:6837","nonce":3177126349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T17:10:58.004 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 15 2023-11-28T17:10:58.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.000+0000 7fbb86090640 1 -- 172.21.15.116:0/3911182696 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbb6006b2d0 msgr2=0x7fbb6006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.000+0000 7fbb86090640 1 --2- 172.21.15.116:0/3911182696 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbb6006b2d0 0x7fbb6006d790 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbb68036000 tx=0x7fbb68030000 comp rx=0 tx=0).stop 2023-11-28T17:10:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.000+0000 7fbb86090640 1 -- 172.21.15.116:0/3911182696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c003ef0 msgr2=0x7fbb801f17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.000+0000 7fbb86090640 1 --2- 172.21.15.116:0/3911182696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c003ef0 0x7fbb801f17e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000e860 tx=0x7fbb7000ed20 comp rx=0 tx=0).stop 2023-11-28T17:10:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.001+0000 7fbb86090640 1 -- 172.21.15.116:0/3911182696 shutdown_connections 2023-11-28T17:10:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.001+0000 7fbb86090640 1 --2- 172.21.15.116:0/3911182696 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbb6006b2d0 0x7fbb6006d790 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.001+0000 7fbb86090640 1 --2- 172.21.15.116:0/3911182696 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb8015d5f0 0x7fbb8013b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.001+0000 7fbb86090640 1 --2- 172.21.15.116:0/3911182696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb8015a3b0 0x7fbb8015d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.001+0000 7fbb86090640 1 --2- 172.21.15.116:0/3911182696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c003ef0 0x7fbb801f17e0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.001+0000 7fbb86090640 1 -- 172.21.15.116:0/3911182696 >> 172.21.15.116:0/3911182696 conn(0x7fbb8006a4c0 msgr2=0x7fbb80143880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:58.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.001+0000 7fbb86090640 1 -- 172.21.15.116:0/3911182696 shutdown_connections 2023-11-28T17:10:58.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.001+0000 7fbb86090640 1 -- 172.21.15.116:0/3911182696 wait complete. 2023-11-28T17:10:58.022 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:10:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.225+0000 7fa447f78640 1 Processor -- start 2023-11-28T17:10:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.225+0000 7fa447f78640 1 -- start start 2023-11-28T17:10:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.227+0000 7fa447f78640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa44013cee0 0x7fa44006c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.227+0000 7fa447f78640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa44015aba0 0x7fa44006c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.227+0000 7fa447f78640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4400aea20 con 0x7fa440140200 2023-11-28T17:10:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.227+0000 7fa447f78640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa44013d2c0 con 0x7fa44015aba0 2023-11-28T17:10:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.227+0000 7fa447f78640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa44015b3a0 con 0x7fa44013cee0 2023-11-28T17:10:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.227+0000 7fa446515640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa44015aba0 0x7fa44006c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.227+0000 7fa446515640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa44015aba0 0x7fa44006c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54452/0 (socket says 172.21.15.116:54452) 2023-11-28T17:10:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.227+0000 7fa446515640 1 -- 172.21.15.116:0/3075446181 learned_addr learned my addr 172.21.15.116:0/3075446181 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.228+0000 7fa445d14640 1 --2- 172.21.15.116:0/3075446181 >> v2:172.21.15.156:3300/0 conn(0x7fa44013cee0 0x7fa44006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.228+0000 7fa446515640 1 -- 172.21.15.116:0/3075446181 >> v2:172.21.15.156:3300/0 conn(0x7fa44013cee0 msgr2=0x7fa44006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.228+0000 7fa446515640 1 --2- 172.21.15.116:0/3075446181 >> v2:172.21.15.156:3300/0 conn(0x7fa44013cee0 0x7fa44006c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.228+0000 7fa446515640 1 -- 172.21.15.116:0/3075446181 >> v1:172.21.15.156:6789/0 conn(0x7fa440140200 legacy=0x7fa44015a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.228+0000 7fa446515640 1 -- 172.21.15.116:0/3075446181 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44015b0c0 con 0x7fa44015aba0 2023-11-28T17:10:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.229+0000 7fa446515640 1 --2- 172.21.15.116:0/3075446181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa44015aba0 0x7fa44006c790 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa4400b2280 tx=0x7fa43802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1356c98d803a00e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.230+0000 7fa444d12640 1 -- 172.21.15.116:0/3075446181 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43802c9f0 con 0x7fa44015aba0 2023-11-28T17:10:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.230+0000 7fa444d12640 1 -- 172.21.15.116:0/3075446181 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43802cb50 con 0x7fa44015aba0 2023-11-28T17:10:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.230+0000 7fa444d12640 1 -- 172.21.15.116:0/3075446181 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4380343f0 con 0x7fa44015aba0 2023-11-28T17:10:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.230+0000 7fa447f78640 1 -- 172.21.15.116:0/3075446181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa44015aba0 msgr2=0x7fa44006c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.230+0000 7fa447f78640 1 --2- 172.21.15.116:0/3075446181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa44015aba0 0x7fa44006c790 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa4400b2280 tx=0x7fa43802be50 comp rx=0 tx=0).stop 2023-11-28T17:10:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.230+0000 7fa447f78640 1 -- 172.21.15.116:0/3075446181 shutdown_connections 2023-11-28T17:10:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.230+0000 7fa447f78640 1 --2- 172.21.15.116:0/3075446181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa44015aba0 0x7fa44006c790 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.230+0000 7fa447f78640 1 --2- 172.21.15.116:0/3075446181 >> v2:172.21.15.156:3300/0 conn(0x7fa44013cee0 0x7fa44006c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.230+0000 7fa447f78640 1 -- 172.21.15.116:0/3075446181 >> 172.21.15.116:0/3075446181 conn(0x7fa44006a4c0 msgr2=0x7fa44006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.231+0000 7fa447f78640 1 -- 172.21.15.116:0/3075446181 shutdown_connections 2023-11-28T17:10:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.231+0000 7fa447f78640 1 -- 172.21.15.116:0/3075446181 wait complete. 2023-11-28T17:10:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.232+0000 7fa447f78640 1 Processor -- start 2023-11-28T17:10:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.233+0000 7fa447f78640 1 -- start start 2023-11-28T17:10:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.233+0000 7fa447f78640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa44013cee0 0x7fa4401363e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.233+0000 7fa447f78640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa440140200 0x7fa44013d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.233+0000 7fa447f78640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa44015aba0 0x7fa44013d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.233+0000 7fa447f78640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4401665e0 con 0x7fa44013cee0 2023-11-28T17:10:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.233+0000 7fa447f78640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44013e350 con 0x7fa44015aba0 2023-11-28T17:10:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.233+0000 7fa447f78640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa440165cb0 con 0x7fa440140200 2023-11-28T17:10:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.234+0000 7fa445d14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa44013cee0 0x7fa4401363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.234+0000 7fa445d14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa44013cee0 0x7fa4401363e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54454/0 (socket says 172.21.15.116:54454) 2023-11-28T17:10:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.234+0000 7fa445d14640 1 -- 172.21.15.116:0/2478002090 learned_addr learned my addr 172.21.15.116:0/2478002090 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.234+0000 7fa446515640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa44015aba0 0x7fa44013d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.234+0000 7fa445513640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa440140200 0x7fa44013d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.234+0000 7fa445d14640 1 -- 172.21.15.116:0/2478002090 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa440140200 msgr2=0x7fa44013d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.234+0000 7fa445d14640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa440140200 0x7fa44013d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.234+0000 7fa445d14640 1 -- 172.21.15.116:0/2478002090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa44015aba0 msgr2=0x7fa44013d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.234+0000 7fa445d14640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa44015aba0 0x7fa44013d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.234+0000 7fa445d14640 1 -- 172.21.15.116:0/2478002090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa438009b30 con 0x7fa44013cee0 2023-11-28T17:10:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.234+0000 7fa446515640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa44015aba0 0x7fa44013d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.235+0000 7fa445d14640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa44013cee0 0x7fa4401363e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa43c00d800 tx=0x7fa43c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.235+0000 7fa4367fc640 1 -- 172.21.15.116:0/2478002090 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c00b6c0 con 0x7fa44013cee0 2023-11-28T17:10:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.235+0000 7fa4367fc640 1 -- 172.21.15.116:0/2478002090 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43c00b820 con 0x7fa44013cee0 2023-11-28T17:10:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.235+0000 7fa4367fc640 1 -- 172.21.15.116:0/2478002090 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c00bbb0 con 0x7fa44013cee0 2023-11-28T17:10:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.235+0000 7fa447f78640 1 -- 172.21.15.116:0/2478002090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa440165f90 con 0x7fa44013cee0 2023-11-28T17:10:58.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.235+0000 7fa447f78640 1 -- 172.21.15.116:0/2478002090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa440136920 con 0x7fa44013cee0 2023-11-28T17:10:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.236+0000 7fa447f78640 1 -- 172.21.15.116:0/2478002090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4400703d0 con 0x7fa44013cee0 2023-11-28T17:10:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.237+0000 7fa4367fc640 1 -- 172.21.15.116:0/2478002090 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa43c00bd10 con 0x7fa44013cee0 2023-11-28T17:10:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.238+0000 7fa4367fc640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa42406b2a0 0x7fa42406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:58.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.238+0000 7fa4367fc640 1 -- 172.21.15.116:0/2478002090 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa43c08a200 con 0x7fa44013cee0 2023-11-28T17:10:58.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.238+0000 7fa445513640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa42406b2a0 0x7fa42406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:58.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.239+0000 7fa445513640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa42406b2a0 0x7fa42406d760 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa440157d80 tx=0x7fa430009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:58.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.242+0000 7fa4367fc640 1 -- 172.21.15.116:0/2478002090 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4400703d0 con 0x7fa44013cee0 2023-11-28T17:10:58.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.434+0000 7fa447f78640 1 -- 172.21.15.116:0/2478002090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4400703d0 con 0x7fa44013cee0 2023-11-28T17:10:58.438 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.437+0000 7fa4367fc640 1 -- 172.21.15.116:0/2478002090 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16770 (secure 0 0 0) 0x7fa4400703d0 con 0x7fa44013cee0 2023-11-28T17:10:58.439 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:10:58.439 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":36,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:10:44.069312+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:10:17.338444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-28T17:10:18.766338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.1d","mappings":[{"from":4,"to":5}]},{"pgid":"5.37","mappings":[{"from":4,"to":5}]},{"pgid":"5.3e","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:10:58.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.440+0000 7fa447f78640 1 -- 172.21.15.116:0/2478002090 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa42406b2a0 msgr2=0x7fa42406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:58.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.440+0000 7fa447f78640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa42406b2a0 0x7fa42406d760 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa440157d80 tx=0x7fa430009040 comp rx=0 tx=0).stop 2023-11-28T17:10:58.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.440+0000 7fa447f78640 1 -- 172.21.15.116:0/2478002090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa44013cee0 msgr2=0x7fa4401363e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:58.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.440+0000 7fa447f78640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa44013cee0 0x7fa4401363e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa43c00d800 tx=0x7fa43c00dcc0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.440+0000 7fa447f78640 1 -- 172.21.15.116:0/2478002090 shutdown_connections 2023-11-28T17:10:58.442 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.440+0000 7fa447f78640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa42406b2a0 0x7fa42406d760 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.440+0000 7fa447f78640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa44015aba0 0x7fa44013d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.440+0000 7fa447f78640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa440140200 0x7fa44013d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.440+0000 7fa447f78640 1 --2- 172.21.15.116:0/2478002090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa44013cee0 0x7fa4401363e0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.440+0000 7fa447f78640 1 -- 172.21.15.116:0/2478002090 >> 172.21.15.116:0/2478002090 conn(0x7fa44006a4c0 msgr2=0x7fa44015dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.440+0000 7fa447f78640 1 -- 172.21.15.116:0/2478002090 shutdown_connections 2023-11-28T17:10:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.440+0000 7fa447f78640 1 -- 172.21.15.116:0/2478002090 wait complete. 2023-11-28T17:10:58.460 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:10:17.338444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:10:18.766338+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:10:58.461 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:10:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.673+0000 7fdc45c48640 1 Processor -- start 2023-11-28T17:10:58.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.673+0000 7fdc45c48640 1 -- start start 2023-11-28T17:10:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.675+0000 7fdc45c48640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdc40071c30 0x7fdc400cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.675+0000 7fdc45c48640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc400724c0 con 0x7fdc40143390 2023-11-28T17:10:58.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.675+0000 7fdc45c48640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc400729f0 con 0x7fdc401438f0 2023-11-28T17:10:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.675+0000 7fdc45c48640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc40143770 con 0x7fdc40071c30 2023-11-28T17:10:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.675+0000 7fdc3f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdc40143390 0x7fdc400cf5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54172/0 (socket says 172.21.15.116:54172) 2023-11-28T17:10:58.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.675+0000 7fdc3f7fe640 1 -- 172.21.15.116:0/1065307522 learned_addr learned my addr 172.21.15.116:0/1065307522 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:58.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3ffff640 1 --2- 172.21.15.116:0/1065307522 >> v2:172.21.15.156:3300/0 conn(0x7fdc40071c30 0x7fdc400cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 -- 172.21.15.116:0/1065307522 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422932148 0 0) 0x7fdc400724c0 con 0x7fdc40143390 2023-11-28T17:10:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 -- 172.21.15.116:0/1065307522 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc28003520 con 0x7fdc40143390 2023-11-28T17:10:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 -- 172.21.15.116:0/1065307522 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2194371512 0 0) 0x7fdc28003520 con 0x7fdc40143390 2023-11-28T17:10:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 -- 172.21.15.116:0/1065307522 >> v2:172.21.15.156:3300/0 conn(0x7fdc40071c30 msgr2=0x7fdc400cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 --2- 172.21.15.116:0/1065307522 >> v2:172.21.15.156:3300/0 conn(0x7fdc40071c30 0x7fdc400cd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 -- 172.21.15.116:0/1065307522 >> v1:172.21.15.156:6789/0 conn(0x7fdc401438f0 legacy=0x7fdc40071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 -- 172.21.15.116:0/1065307522 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4006e540 con 0x7fdc40143390 2023-11-28T17:10:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 -- 172.21.15.116:0/1065307522 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fdc30001670 con 0x7fdc40143390 2023-11-28T17:10:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3ffff640 1 --2- 172.21.15.116:0/1065307522 >> v2:172.21.15.156:3300/0 conn(0x7fdc40071c30 0x7fdc400cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:10:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 -- 172.21.15.116:0/1065307522 >> v1:172.21.15.116:6789/0 conn(0x7fdc40143390 legacy=0x7fdc400cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 --2- 172.21.15.116:0/1065307522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc28003970 0x7fdc28003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:58.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 --2- 172.21.15.116:0/1065307522 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc280047c0 0x7fdc28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 --2- 172.21.15.116:0/1065307522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc28005180 0x7fdc2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 -- 172.21.15.116:0/1065307522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc40143770 con 0x7fdc28003970 2023-11-28T17:10:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 -- 172.21.15.116:0/1065307522 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2800c0d0 con 0x7fdc28005180 2023-11-28T17:10:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.676+0000 7fdc3e7fc640 1 -- 172.21.15.116:0/1065307522 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2800c290 con 0x7fdc280047c0 2023-11-28T17:10:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.677+0000 7fdc3f7fe640 1 --2- 172.21.15.116:0/1065307522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc28003970 0x7fdc28003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.677+0000 7fdc3f7fe640 1 -- 172.21.15.116:0/1065307522 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc280047c0 msgr2=0x7fdc28004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.677+0000 7fdc3f7fe640 1 --2- 172.21.15.116:0/1065307522 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc280047c0 0x7fdc28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.677+0000 7fdc3ffff640 1 --2- 172.21.15.116:0/1065307522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc28005180 0x7fdc2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.677+0000 7fdc3f7fe640 1 -- 172.21.15.116:0/1065307522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc28005180 msgr2=0x7fdc2800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.677+0000 7fdc3f7fe640 1 --2- 172.21.15.116:0/1065307522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc28005180 0x7fdc2800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.677+0000 7fdc3ffff640 1 -- 172.21.15.116:0/1065307522 reap_dead start 2023-11-28T17:10:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.677+0000 7fdc3f7fe640 1 -- 172.21.15.116:0/1065307522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc28003520 con 0x7fdc28003970 2023-11-28T17:10:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.678+0000 7fdc3f7fe640 1 --2- 172.21.15.116:0/1065307522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc28003970 0x7fdc28003d90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdc3000c190 tx=0x7fdc280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f07d215f8c5646f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.678+0000 7fdc3e7fc640 1 -- 172.21.15.116:0/1065307522 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc30005ad0 con 0x7fdc28003970 2023-11-28T17:10:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.678+0000 7fdc3e7fc640 1 -- 172.21.15.116:0/1065307522 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc30005c30 con 0x7fdc28003970 2023-11-28T17:10:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.678+0000 7fdc3e7fc640 1 -- 172.21.15.116:0/1065307522 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3002cae0 con 0x7fdc28003970 2023-11-28T17:10:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.679+0000 7fdc45c48640 1 -- 172.21.15.116:0/1065307522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc28003970 msgr2=0x7fdc28003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.679+0000 7fdc45c48640 1 --2- 172.21.15.116:0/1065307522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc28003970 0x7fdc28003d90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdc3000c190 tx=0x7fdc280057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.679+0000 7fdc45c48640 1 -- 172.21.15.116:0/1065307522 shutdown_connections 2023-11-28T17:10:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.679+0000 7fdc45c48640 1 --2- 172.21.15.116:0/1065307522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc28003970 0x7fdc28003d90 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.679+0000 7fdc45c48640 1 -- 172.21.15.116:0/1065307522 >> 172.21.15.116:0/1065307522 conn(0x7fdc4006d4a0 msgr2=0x7fdc4006d880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.679+0000 7fdc45c48640 1 -- 172.21.15.116:0/1065307522 shutdown_connections 2023-11-28T17:10:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.679+0000 7fdc45c48640 1 -- 172.21.15.116:0/1065307522 wait complete. 2023-11-28T17:10:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.681+0000 7fdc45c48640 1 Processor -- start 2023-11-28T17:10:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.681+0000 7fdc45c48640 1 -- start start 2023-11-28T17:10:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.682+0000 7fdc45c48640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc28003970 0x7fdc4013e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:58.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.682+0000 7fdc45c48640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc40141ea0 0x7fdc4015d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.682+0000 7fdc45c48640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc4013d530 0x7fdc4013d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.682+0000 7fdc45c48640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc400cb750 con 0x7fdc40141ea0 2023-11-28T17:10:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.682+0000 7fdc45c48640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4013def0 con 0x7fdc28003970 2023-11-28T17:10:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.682+0000 7fdc45c48640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc400cbbe0 con 0x7fdc4013d530 2023-11-28T17:10:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.682+0000 7fdc3effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc40141ea0 0x7fdc4015d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:58.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.682+0000 7fdc3effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc40141ea0 0x7fdc4015d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54476/0 (socket says 172.21.15.116:54476) 2023-11-28T17:10:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.682+0000 7fdc3effd640 1 -- 172.21.15.116:0/280861843 learned_addr learned my addr 172.21.15.116:0/280861843 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.683+0000 7fdc3f7fe640 1 --2- 172.21.15.116:0/280861843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc28003970 0x7fdc4013e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.683+0000 7fdc3effd640 1 -- 172.21.15.116:0/280861843 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc4013d530 msgr2=0x7fdc4013d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.683+0000 7fdc3effd640 1 --2- 172.21.15.116:0/280861843 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc4013d530 0x7fdc4013d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.683+0000 7fdc3effd640 1 -- 172.21.15.116:0/280861843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc28003970 msgr2=0x7fdc4013e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:58.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.683+0000 7fdc3effd640 1 --2- 172.21.15.116:0/280861843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc28003970 0x7fdc4013e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.683+0000 7fdc3effd640 1 -- 172.21.15.116:0/280861843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc401425f0 con 0x7fdc40141ea0 2023-11-28T17:10:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.683+0000 7fdc3effd640 1 --2- 172.21.15.116:0/280861843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc40141ea0 0x7fdc4015d780 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400c500 tx=0x7fdc3400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.683+0000 7fdc27fff640 1 -- 172.21.15.116:0/280861843 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc340052d0 con 0x7fdc40141ea0 2023-11-28T17:10:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.683+0000 7fdc27fff640 1 -- 172.21.15.116:0/280861843 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc34005430 con 0x7fdc40141ea0 2023-11-28T17:10:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.683+0000 7fdc45c48640 1 -- 172.21.15.116:0/280861843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc400cb100 con 0x7fdc40141ea0 2023-11-28T17:10:58.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.683+0000 7fdc27fff640 1 -- 172.21.15.116:0/280861843 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34012870 con 0x7fdc40141ea0 2023-11-28T17:10:58.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.683+0000 7fdc45c48640 1 -- 172.21.15.116:0/280861843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc40142300 con 0x7fdc40141ea0 2023-11-28T17:10:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.684+0000 7fdc45c48640 1 -- 172.21.15.116:0/280861843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc400cb8c0 con 0x7fdc40141ea0 2023-11-28T17:10:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.685+0000 7fdc27fff640 1 -- 172.21.15.116:0/280861843 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fdc340129d0 con 0x7fdc40141ea0 2023-11-28T17:10:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.686+0000 7fdc27fff640 1 --2- 172.21.15.116:0/280861843 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdc2c06b2d0 0x7fdc2c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.686+0000 7fdc27fff640 1 -- 172.21.15.116:0/280861843 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdc34089890 con 0x7fdc40141ea0 2023-11-28T17:10:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.687+0000 7fdc3f7fe640 1 --2- 172.21.15.116:0/280861843 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdc2c06b2d0 0x7fdc2c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.687+0000 7fdc3f7fe640 1 --2- 172.21.15.116:0/280861843 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdc2c06b2d0 0x7fdc2c06d790 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdc30034000 tx=0x7fdc30009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.691+0000 7fdc27fff640 1 -- 172.21.15.116:0/280861843 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdc34058100 con 0x7fdc40141ea0 2023-11-28T17:10:58.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.882+0000 7fdc45c48640 1 -- 172.21.15.116:0/280861843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdc40067460 con 0x7fdc40141ea0 2023-11-28T17:10:58.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.883+0000 7fdc27fff640 1 -- 172.21.15.116:0/280861843 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdc34057c90 con 0x7fdc40141ea0 2023-11-28T17:10:58.885 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:10:58.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.886+0000 7fdc45c48640 1 -- 172.21.15.116:0/280861843 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdc2c06b2d0 msgr2=0x7fdc2c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:58.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.886+0000 7fdc45c48640 1 --2- 172.21.15.116:0/280861843 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdc2c06b2d0 0x7fdc2c06d790 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdc30034000 tx=0x7fdc30009030 comp rx=0 tx=0).stop 2023-11-28T17:10:58.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.886+0000 7fdc45c48640 1 -- 172.21.15.116:0/280861843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc40141ea0 msgr2=0x7fdc4015d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:58.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.886+0000 7fdc45c48640 1 --2- 172.21.15.116:0/280861843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc40141ea0 0x7fdc4015d780 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400c500 tx=0x7fdc3400c9c0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.886+0000 7fdc45c48640 1 -- 172.21.15.116:0/280861843 shutdown_connections 2023-11-28T17:10:58.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.886+0000 7fdc45c48640 1 --2- 172.21.15.116:0/280861843 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fdc2c06b2d0 0x7fdc2c06d790 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.886+0000 7fdc45c48640 1 --2- 172.21.15.116:0/280861843 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdc4013d530 0x7fdc4013d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.886+0000 7fdc45c48640 1 --2- 172.21.15.116:0/280861843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc40141ea0 0x7fdc4015d780 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.886+0000 7fdc45c48640 1 --2- 172.21.15.116:0/280861843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc28003970 0x7fdc4013e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.886+0000 7fdc45c48640 1 -- 172.21.15.116:0/280861843 >> 172.21.15.116:0/280861843 conn(0x7fdc4006d4a0 msgr2=0x7fdc400b0310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.886+0000 7fdc45c48640 1 -- 172.21.15.116:0/280861843 shutdown_connections 2023-11-28T17:10:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:58.886+0000 7fdc45c48640 1 -- 172.21.15.116:0/280861843 wait complete. 2023-11-28T17:10:58.907 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:10:59.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.117+0000 7fbd98f1d640 1 Processor -- start 2023-11-28T17:10:59.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.117+0000 7fbd98f1d640 1 -- start start 2023-11-28T17:10:59.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.119+0000 7fbd98f1d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbd9415b0c0 0x7fbd9415ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:59.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.119+0000 7fbd98f1d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd940cd320 con 0x7fbd940ce940 2023-11-28T17:10:59.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.119+0000 7fbd98f1d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd940ced80 con 0x7fbd94072250 2023-11-28T17:10:59.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.119+0000 7fbd98f1d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd940cc710 con 0x7fbd9415b0c0 2023-11-28T17:10:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.120+0000 7fbd8bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbd940ce940 0x7fbd94071b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54182/0 (socket says 172.21.15.116:54182) 2023-11-28T17:10:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.120+0000 7fbd8bfff640 1 -- 172.21.15.116:0/4147284860 learned_addr learned my addr 172.21.15.116:0/4147284860 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.120+0000 7fbd9259c640 1 --2- 172.21.15.116:0/4147284860 >> v2:172.21.15.156:3300/0 conn(0x7fbd9415b0c0 0x7fbd9415ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.120+0000 7fbd91d9b640 1 -- 172.21.15.116:0/4147284860 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221574298 0 0) 0x7fbd940cd320 con 0x7fbd940ce940 2023-11-28T17:10:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.120+0000 7fbd91d9b640 1 -- 172.21.15.116:0/4147284860 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd80003520 con 0x7fbd940ce940 2023-11-28T17:10:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.120+0000 7fbd91d9b640 1 -- 172.21.15.116:0/4147284860 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1994117815 0 0) 0x7fbd80003520 con 0x7fbd940ce940 2023-11-28T17:10:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.120+0000 7fbd91d9b640 1 -- 172.21.15.116:0/4147284860 >> v2:172.21.15.156:3300/0 conn(0x7fbd9415b0c0 msgr2=0x7fbd9415ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd91d9b640 1 --2- 172.21.15.116:0/4147284860 >> v2:172.21.15.156:3300/0 conn(0x7fbd9415b0c0 0x7fbd9415ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd91d9b640 1 -- 172.21.15.116:0/4147284860 >> v1:172.21.15.156:6789/0 conn(0x7fbd94072250 legacy=0x7fbd9415b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd91d9b640 1 -- 172.21.15.116:0/4147284860 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd940cc9f0 con 0x7fbd940ce940 2023-11-28T17:10:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd91d9b640 1 -- 172.21.15.116:0/4147284860 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fbd7c001670 con 0x7fbd940ce940 2023-11-28T17:10:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd91d9b640 1 -- 172.21.15.116:0/4147284860 >> v1:172.21.15.116:6789/0 conn(0x7fbd940ce940 legacy=0x7fbd94071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd91d9b640 1 --2- 172.21.15.116:0/4147284860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd800039c0 0x7fbd80003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd91d9b640 1 --2- 172.21.15.116:0/4147284860 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd800047c0 0x7fbd80004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd91d9b640 1 --2- 172.21.15.116:0/4147284860 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd800051d0 0x7fbd8000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd91d9b640 1 -- 172.21.15.116:0/4147284860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd940cc710 con 0x7fbd800039c0 2023-11-28T17:10:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd91d9b640 1 -- 172.21.15.116:0/4147284860 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8000c120 con 0x7fbd800051d0 2023-11-28T17:10:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd91d9b640 1 -- 172.21.15.116:0/4147284860 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8000c2e0 con 0x7fbd800047c0 2023-11-28T17:10:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd9259c640 1 --2- 172.21.15.116:0/4147284860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd800039c0 0x7fbd80003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:59.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd9259c640 1 -- 172.21.15.116:0/4147284860 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd800047c0 msgr2=0x7fbd80004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:59.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd9259c640 1 --2- 172.21.15.116:0/4147284860 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd800047c0 0x7fbd80004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd9259c640 1 -- 172.21.15.116:0/4147284860 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd800051d0 msgr2=0x7fbd8000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:59.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd9259c640 1 --2- 172.21.15.116:0/4147284860 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd800051d0 0x7fbd8000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd92d9d640 1 -- 172.21.15.116:0/4147284860 reap_dead start 2023-11-28T17:10:59.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.121+0000 7fbd9259c640 1 -- 172.21.15.116:0/4147284860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd80003520 con 0x7fbd800039c0 2023-11-28T17:10:59.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.122+0000 7fbd9259c640 1 --2- 172.21.15.116:0/4147284860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd800039c0 0x7fbd80003de0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fbd84009a90 tx=0x7fbd800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb45caf7590fb67 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:59.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.123+0000 7fbd91d9b640 1 -- 172.21.15.116:0/4147284860 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8402c6b0 con 0x7fbd800039c0 2023-11-28T17:10:59.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.123+0000 7fbd91d9b640 1 -- 172.21.15.116:0/4147284860 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8402c810 con 0x7fbd800039c0 2023-11-28T17:10:59.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.123+0000 7fbd91d9b640 1 -- 172.21.15.116:0/4147284860 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8402cba0 con 0x7fbd800039c0 2023-11-28T17:10:59.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.123+0000 7fbd98f1d640 1 -- 172.21.15.116:0/4147284860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd800039c0 msgr2=0x7fbd80003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:59.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.123+0000 7fbd98f1d640 1 --2- 172.21.15.116:0/4147284860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd800039c0 0x7fbd80003de0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fbd84009a90 tx=0x7fbd800057f0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.123+0000 7fbd98f1d640 1 -- 172.21.15.116:0/4147284860 shutdown_connections 2023-11-28T17:10:59.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.123+0000 7fbd98f1d640 1 --2- 172.21.15.116:0/4147284860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd800039c0 0x7fbd80003de0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.123+0000 7fbd98f1d640 1 -- 172.21.15.116:0/4147284860 >> 172.21.15.116:0/4147284860 conn(0x7fbd940b0cf0 msgr2=0x7fbd940b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:59.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.123+0000 7fbd98f1d640 1 -- 172.21.15.116:0/4147284860 shutdown_connections 2023-11-28T17:10:59.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.124+0000 7fbd98f1d640 1 -- 172.21.15.116:0/4147284860 wait complete. 2023-11-28T17:10:59.127 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.125+0000 7fbd98f1d640 1 Processor -- start 2023-11-28T17:10:59.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.125+0000 7fbd98f1d640 1 -- start start 2023-11-28T17:10:59.129 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.126+0000 7fbd98f1d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd800039c0 0x7fbd9414d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:59.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.126+0000 7fbd98f1d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd94155150 0x7fbd941f6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:59.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.126+0000 7fbd98f1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd941f6d50 0x7fbd940b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:59.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.126+0000 7fbd98f1d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd941f71d0 con 0x7fbd941f6d50 2023-11-28T17:10:59.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.126+0000 7fbd98f1d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd941540b0 con 0x7fbd800039c0 2023-11-28T17:10:59.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.126+0000 7fbd98f1d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd94154390 con 0x7fbd94155150 2023-11-28T17:10:59.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.127+0000 7fbd92d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd941f6d50 0x7fbd940b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:59.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.127+0000 7fbd92d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd941f6d50 0x7fbd940b2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54480/0 (socket says 172.21.15.116:54480) 2023-11-28T17:10:59.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.127+0000 7fbd92d9d640 1 -- 172.21.15.116:0/1782498816 learned_addr learned my addr 172.21.15.116:0/1782498816 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:59.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.127+0000 7fbd9259c640 1 --2- 172.21.15.116:0/1782498816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd800039c0 0x7fbd9414d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:59.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.127+0000 7fbd8bfff640 1 --2- 172.21.15.116:0/1782498816 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd94155150 0x7fbd941f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:59.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.127+0000 7fbd92d9d640 1 -- 172.21.15.116:0/1782498816 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd94155150 msgr2=0x7fbd941f6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:59.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.127+0000 7fbd92d9d640 1 --2- 172.21.15.116:0/1782498816 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd94155150 0x7fbd941f6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.127+0000 7fbd92d9d640 1 -- 172.21.15.116:0/1782498816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd800039c0 msgr2=0x7fbd9414d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:59.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.127+0000 7fbd92d9d640 1 --2- 172.21.15.116:0/1782498816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd800039c0 0x7fbd9414d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.127+0000 7fbd92d9d640 1 -- 172.21.15.116:0/1782498816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9415b620 con 0x7fbd941f6d50 2023-11-28T17:10:59.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.127+0000 7fbd92d9d640 1 --2- 172.21.15.116:0/1782498816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd941f6d50 0x7fbd940b2f90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbd940cb950 tx=0x7fbd8c00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:59.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.128+0000 7fbd8affd640 1 -- 172.21.15.116:0/1782498816 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c007070 con 0x7fbd941f6d50 2023-11-28T17:10:59.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.128+0000 7fbd8affd640 1 -- 172.21.15.116:0/1782498816 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c0071d0 con 0x7fbd941f6d50 2023-11-28T17:10:59.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.128+0000 7fbd98f1d640 1 -- 172.21.15.116:0/1782498816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd94154670 con 0x7fbd941f6d50 2023-11-28T17:10:59.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.128+0000 7fbd8affd640 1 -- 172.21.15.116:0/1782498816 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c007580 con 0x7fbd941f6d50 2023-11-28T17:10:59.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.128+0000 7fbd98f1d640 1 -- 172.21.15.116:0/1782498816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd94154c30 con 0x7fbd941f6d50 2023-11-28T17:10:59.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.129+0000 7fbd98f1d640 1 -- 172.21.15.116:0/1782498816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd94154e50 con 0x7fbd941f6d50 2023-11-28T17:10:59.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.130+0000 7fbd8affd640 1 -- 172.21.15.116:0/1782498816 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fbd8c0076e0 con 0x7fbd941f6d50 2023-11-28T17:10:59.132 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.131+0000 7fbd8affd640 1 --2- 172.21.15.116:0/1782498816 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbd7406b2d0 0x7fbd7406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:59.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.131+0000 7fbd8affd640 1 -- 172.21.15.116:0/1782498816 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbd8c0890a0 con 0x7fbd941f6d50 2023-11-28T17:10:59.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.132+0000 7fbd9259c640 1 --2- 172.21.15.116:0/1782498816 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbd7406b2d0 0x7fbd7406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:59.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.132+0000 7fbd9259c640 1 --2- 172.21.15.116:0/1782498816 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbd7406b2d0 0x7fbd7406d790 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbd84036000 tx=0x7fbd84030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:59.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.136+0000 7fbd8affd640 1 -- 172.21.15.116:0/1782498816 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd8c053e90 con 0x7fbd941f6d50 2023-11-28T17:10:59.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.329+0000 7fbd98f1d640 1 -- 172.21.15.116:0/1782498816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbd940cbe90 con 0x7fbd941f6d50 2023-11-28T17:10:59.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.330+0000 7fbd8affd640 1 -- 172.21.15.116:0/1782498816 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbd8c0170a0 con 0x7fbd941f6d50 2023-11-28T17:10:59.331 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:10:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.332+0000 7fbd98f1d640 1 -- 172.21.15.116:0/1782498816 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbd7406b2d0 msgr2=0x7fbd7406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.332+0000 7fbd98f1d640 1 --2- 172.21.15.116:0/1782498816 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbd7406b2d0 0x7fbd7406d790 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbd84036000 tx=0x7fbd84030000 comp rx=0 tx=0).stop 2023-11-28T17:10:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.332+0000 7fbd98f1d640 1 -- 172.21.15.116:0/1782498816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd941f6d50 msgr2=0x7fbd940b2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.332+0000 7fbd98f1d640 1 --2- 172.21.15.116:0/1782498816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd941f6d50 0x7fbd940b2f90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbd940cb950 tx=0x7fbd8c00c980 comp rx=0 tx=0).stop 2023-11-28T17:10:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.333+0000 7fbd98f1d640 1 -- 172.21.15.116:0/1782498816 shutdown_connections 2023-11-28T17:10:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.333+0000 7fbd98f1d640 1 --2- 172.21.15.116:0/1782498816 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbd7406b2d0 0x7fbd7406d790 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.333+0000 7fbd98f1d640 1 --2- 172.21.15.116:0/1782498816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd941f6d50 0x7fbd940b2f90 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.333+0000 7fbd98f1d640 1 --2- 172.21.15.116:0/1782498816 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd94155150 0x7fbd941f6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.333+0000 7fbd98f1d640 1 --2- 172.21.15.116:0/1782498816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd800039c0 0x7fbd9414d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.333+0000 7fbd98f1d640 1 -- 172.21.15.116:0/1782498816 >> 172.21.15.116:0/1782498816 conn(0x7fbd940b0cf0 msgr2=0x7fbd9415c850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.333+0000 7fbd98f1d640 1 -- 172.21.15.116:0/1782498816 shutdown_connections 2023-11-28T17:10:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.333+0000 7fbd98f1d640 1 -- 172.21.15.116:0/1782498816 wait complete. 2023-11-28T17:10:59.353 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:10:59.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.563+0000 7f3f9f59e640 1 Processor -- start 2023-11-28T17:10:59.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.563+0000 7f3f9f59e640 1 -- start start 2023-11-28T17:10:59.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.565+0000 7f3f9f59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3fa00d0700 0x7f3fa00d1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:59.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.565+0000 7f3f9f59e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa00d0da0 con 0x7f3fa0160b80 2023-11-28T17:10:59.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.565+0000 7f3f9f59e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa00d1080 con 0x7f3fa00b4010 2023-11-28T17:10:59.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.565+0000 7f3f9f59e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa0160f60 con 0x7f3fa00d0700 2023-11-28T17:10:59.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.565+0000 7f3f9e59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3fa0160b80 0x7f3fa00d1360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54190/0 (socket says 172.21.15.116:54190) 2023-11-28T17:10:59.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.565+0000 7f3f9e59c640 1 -- 172.21.15.116:0/3521179408 learned_addr learned my addr 172.21.15.116:0/3521179408 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:10:59.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.565+0000 7f3f97fff640 1 --2- 172.21.15.116:0/3521179408 >> v2:172.21.15.116:3300/0 conn(0x7f3fa00d0700 0x7f3fa00d1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:59.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.566+0000 7f3f9dd9b640 1 -- 172.21.15.116:0/3521179408 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582629928 0 0) 0x7f3fa00d0da0 con 0x7f3fa0160b80 2023-11-28T17:10:59.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.566+0000 7f3f9dd9b640 1 -- 172.21.15.116:0/3521179408 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f8c003520 con 0x7f3fa0160b80 2023-11-28T17:10:59.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.566+0000 7f3f97fff640 1 -- 172.21.15.116:0/3521179408 >> v1:172.21.15.156:6789/0 conn(0x7f3fa00b4010 legacy=0x7f3fa00c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:10:59.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.566+0000 7f3f97fff640 1 -- 172.21.15.116:0/3521179408 >> v1:172.21.15.116:6789/0 conn(0x7f3fa0160b80 legacy=0x7f3fa00d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:59.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.566+0000 7f3f97fff640 1 -- 172.21.15.116:0/3521179408 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa00ce780 con 0x7f3fa00d0700 2023-11-28T17:10:59.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.567+0000 7f3f97fff640 1 --2- 172.21.15.116:0/3521179408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fa00d0700 0x7f3fa00d1a70 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3f880097a0 tx=0x7f3f8802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e149ba38d99e770e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:59.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.567+0000 7f3f9dd9b640 1 -- 172.21.15.116:0/3521179408 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88031830 con 0x7f3fa00d0700 2023-11-28T17:10:59.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.567+0000 7f3f9dd9b640 1 -- 172.21.15.116:0/3521179408 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f88031990 con 0x7f3fa00d0700 2023-11-28T17:10:59.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.567+0000 7f3f9dd9b640 1 -- 172.21.15.116:0/3521179408 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88031d20 con 0x7f3fa00d0700 2023-11-28T17:10:59.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.568+0000 7f3f9f59e640 1 -- 172.21.15.116:0/3521179408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fa00d0700 msgr2=0x7f3fa00d1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:59.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.568+0000 7f3f9f59e640 1 --2- 172.21.15.116:0/3521179408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fa00d0700 0x7f3fa00d1a70 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3f880097a0 tx=0x7f3f8802b8d0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.568+0000 7f3f9f59e640 1 -- 172.21.15.116:0/3521179408 shutdown_connections 2023-11-28T17:10:59.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.568+0000 7f3f9f59e640 1 --2- 172.21.15.116:0/3521179408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fa00d0700 0x7f3fa00d1a70 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.568+0000 7f3f9f59e640 1 -- 172.21.15.116:0/3521179408 >> 172.21.15.116:0/3521179408 conn(0x7f3fa00d2fc0 msgr2=0x7f3fa00d33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:59.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.568+0000 7f3f9f59e640 1 -- 172.21.15.116:0/3521179408 shutdown_connections 2023-11-28T17:10:59.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.568+0000 7f3f9f59e640 1 -- 172.21.15.116:0/3521179408 wait complete. 2023-11-28T17:10:59.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.570+0000 7f3f9f59e640 1 Processor -- start 2023-11-28T17:10:59.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.570+0000 7f3f9f59e640 1 -- start start 2023-11-28T17:10:59.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.571+0000 7f3f9f59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fa00b4010 0x7f3fa014e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:59.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.571+0000 7f3f9f59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fa0160b80 0x7f3fa00b7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:59.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.571+0000 7f3f9f59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fa00b8440 0x7f3fa015dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:59.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.571+0000 7f3f9f59e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa00b65f0 con 0x7f3fa0160b80 2023-11-28T17:10:59.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.571+0000 7f3f9f59e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa00ca090 con 0x7f3fa00b4010 2023-11-28T17:10:59.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.571+0000 7f3f9f59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa00ca650 con 0x7f3fa00b8440 2023-11-28T17:10:59.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.571+0000 7f3f97fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fa0160b80 0x7f3fa00b7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:59.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.571+0000 7f3f97fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fa0160b80 0x7f3fa00b7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54496/0 (socket says 172.21.15.116:54496) 2023-11-28T17:10:59.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.571+0000 7f3f97fff640 1 -- 172.21.15.116:0/93445141 learned_addr learned my addr 172.21.15.116:0/93445141 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:10:59.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.572+0000 7f3f9e59c640 1 --2- 172.21.15.116:0/93445141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fa00b4010 0x7f3fa014e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:59.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.572+0000 7f3f97fff640 1 -- 172.21.15.116:0/93445141 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fa00b8440 msgr2=0x7f3fa015dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:59.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.572+0000 7f3f97fff640 1 --2- 172.21.15.116:0/93445141 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fa00b8440 0x7f3fa015dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.572+0000 7f3f97fff640 1 -- 172.21.15.116:0/93445141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fa00b4010 msgr2=0x7f3fa014e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:59.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.572+0000 7f3f97fff640 1 --2- 172.21.15.116:0/93445141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fa00b4010 0x7f3fa014e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.572+0000 7f3f97fff640 1 -- 172.21.15.116:0/93445141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f88009400 con 0x7f3fa0160b80 2023-11-28T17:10:59.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.572+0000 7f3f9e59c640 1 --2- 172.21.15.116:0/93445141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fa00b4010 0x7f3fa014e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:10:59.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.572+0000 7f3f97fff640 1 --2- 172.21.15.116:0/93445141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fa0160b80 0x7f3fa00b7810 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3f8802bee0 tx=0x7f3f880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:59.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.572+0000 7f3f96ffd640 1 -- 172.21.15.116:0/93445141 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8803a6f0 con 0x7f3fa0160b80 2023-11-28T17:10:59.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.572+0000 7f3f96ffd640 1 -- 172.21.15.116:0/93445141 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f8803a870 con 0x7f3fa0160b80 2023-11-28T17:10:59.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.572+0000 7f3f9f59e640 1 -- 172.21.15.116:0/93445141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa00b7d50 con 0x7f3fa0160b80 2023-11-28T17:10:59.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.572+0000 7f3f96ffd640 1 -- 172.21.15.116:0/93445141 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8803ac20 con 0x7f3fa0160b80 2023-11-28T17:10:59.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.573+0000 7f3f9f59e640 1 -- 172.21.15.116:0/93445141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa00b6b20 con 0x7f3fa0160b80 2023-11-28T17:10:59.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.574+0000 7f3f9f59e640 1 -- 172.21.15.116:0/93445141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa0067980 con 0x7f3fa0160b80 2023-11-28T17:10:59.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.575+0000 7f3f96ffd640 1 -- 172.21.15.116:0/93445141 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3f8804c3b0 con 0x7f3fa0160b80 2023-11-28T17:10:59.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.575+0000 7f3f96ffd640 1 --2- 172.21.15.116:0/93445141 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3f8006b2d0 0x7f3f8006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:10:59.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.576+0000 7f3f96ffd640 1 -- 172.21.15.116:0/93445141 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3f880ab9c0 con 0x7f3fa0160b80 2023-11-28T17:10:59.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.576+0000 7f3f9e59c640 1 --2- 172.21.15.116:0/93445141 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3f8006b2d0 0x7f3f8006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:10:59.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.576+0000 7f3f9e59c640 1 --2- 172.21.15.116:0/93445141 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3f8006b2d0 0x7f3f8006d790 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3f900011e0 tx=0x7f3f90009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:10:59.581 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.580+0000 7f3f96ffd640 1 -- 172.21.15.116:0/93445141 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f88076960 con 0x7f3fa0160b80 2023-11-28T17:10:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.770+0000 7f3f9f59e640 1 -- 172.21.15.116:0/93445141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3fa00a0eb0 con 0x7f3fa0160b80 2023-11-28T17:10:59.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.771+0000 7f3f96ffd640 1 -- 172.21.15.116:0/93445141 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3f88035020 con 0x7f3fa0160b80 2023-11-28T17:10:59.774 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:10:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.774+0000 7f3f9f59e640 1 -- 172.21.15.116:0/93445141 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3f8006b2d0 msgr2=0x7f3f8006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.774+0000 7f3f9f59e640 1 --2- 172.21.15.116:0/93445141 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3f8006b2d0 0x7f3f8006d790 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3f900011e0 tx=0x7f3f90009040 comp rx=0 tx=0).stop 2023-11-28T17:10:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.774+0000 7f3f9f59e640 1 -- 172.21.15.116:0/93445141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fa0160b80 msgr2=0x7f3fa00b7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:10:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.774+0000 7f3f9f59e640 1 --2- 172.21.15.116:0/93445141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fa0160b80 0x7f3fa00b7810 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f3f8802bee0 tx=0x7f3f880046c0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.774+0000 7f3f9f59e640 1 -- 172.21.15.116:0/93445141 shutdown_connections 2023-11-28T17:10:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.774+0000 7f3f9f59e640 1 --2- 172.21.15.116:0/93445141 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3f8006b2d0 0x7f3f8006d790 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.774+0000 7f3f9f59e640 1 --2- 172.21.15.116:0/93445141 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fa00b8440 0x7f3fa015dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.774+0000 7f3f9f59e640 1 --2- 172.21.15.116:0/93445141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fa0160b80 0x7f3fa00b7810 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.774+0000 7f3f9f59e640 1 --2- 172.21.15.116:0/93445141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fa00b4010 0x7f3fa014e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:10:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.774+0000 7f3f9f59e640 1 -- 172.21.15.116:0/93445141 >> 172.21.15.116:0/93445141 conn(0x7f3fa00d2fc0 msgr2=0x7f3fa015fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:10:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.774+0000 7f3f9f59e640 1 -- 172.21.15.116:0/93445141 shutdown_connections 2023-11-28T17:10:59.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:10:59.774+0000 7f3f9f59e640 1 -- 172.21.15.116:0/93445141 wait complete. 2023-11-28T17:10:59.794 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:11:00.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.004+0000 7f4ac34ee640 1 Processor -- start 2023-11-28T17:11:00.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.005+0000 7f4ac34ee640 1 -- start start 2023-11-28T17:11:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.007+0000 7f4ac34ee640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4abc0b3940 0x7f4abc06c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.007+0000 7f4ac34ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4abc0b3d20 0x7f4abc06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.007+0000 7f4ac34ee640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4abc0b29c0 con 0x7f4abc161d80 2023-11-28T17:11:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.007+0000 7f4ac34ee640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4abc162af0 con 0x7f4abc0b3940 2023-11-28T17:11:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.007+0000 7f4ac34ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc0b2090 con 0x7f4abc0b3d20 2023-11-28T17:11:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.007+0000 7f4ac1a8b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4abc161d80 0x7f4abc162210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54204/0 (socket says 172.21.15.116:54204) 2023-11-28T17:11:00.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.007+0000 7f4ac1a8b640 1 -- 172.21.15.116:0/2456429890 learned_addr learned my addr 172.21.15.116:0/2456429890 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.007+0000 7f4ac128a640 1 --2- 172.21.15.116:0/2456429890 >> v2:172.21.15.156:3300/0 conn(0x7f4abc0b3940 0x7f4abc06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.007+0000 7f4ac0a89640 1 --2- 172.21.15.116:0/2456429890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4abc0b3d20 0x7f4abc06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.007+0000 7f4ab3fff640 1 -- 172.21.15.116:0/2456429890 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756466862 0 0) 0x7f4abc0b29c0 con 0x7f4abc161d80 2023-11-28T17:11:00.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 -- 172.21.15.116:0/2456429890 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa8003520 con 0x7f4abc161d80 2023-11-28T17:11:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 -- 172.21.15.116:0/2456429890 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 41762952 0 0) 0x7f4aa8003520 con 0x7f4abc161d80 2023-11-28T17:11:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 -- 172.21.15.116:0/2456429890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4abc0b3d20 msgr2=0x7f4abc06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 --2- 172.21.15.116:0/2456429890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4abc0b3d20 0x7f4abc06c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 -- 172.21.15.116:0/2456429890 >> v2:172.21.15.156:3300/0 conn(0x7f4abc0b3940 msgr2=0x7f4abc06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 --2- 172.21.15.116:0/2456429890 >> v2:172.21.15.156:3300/0 conn(0x7f4abc0b3940 0x7f4abc06c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 -- 172.21.15.116:0/2456429890 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc0b2370 con 0x7f4abc161d80 2023-11-28T17:11:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 -- 172.21.15.116:0/2456429890 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f4ab40019f0 con 0x7f4abc161d80 2023-11-28T17:11:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 -- 172.21.15.116:0/2456429890 >> v1:172.21.15.116:6789/0 conn(0x7f4abc161d80 legacy=0x7f4abc162210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 --2- 172.21.15.116:0/2456429890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa8003970 0x7f4aa8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 --2- 172.21.15.116:0/2456429890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa80047c0 0x7f4aa8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 --2- 172.21.15.116:0/2456429890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4abc0b3d20 0x7f4aa8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 -- 172.21.15.116:0/2456429890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc162af0 con 0x7f4aa80047c0 2023-11-28T17:11:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 -- 172.21.15.116:0/2456429890 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc0b2090 con 0x7f4aa8003970 2023-11-28T17:11:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ab3fff640 1 -- 172.21.15.116:0/2456429890 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa800b980 con 0x7f4abc0b3d20 2023-11-28T17:11:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.008+0000 7f4ac0a89640 1 --2- 172.21.15.116:0/2456429890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa80047c0 0x7f4aa8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.009+0000 7f4ac1a8b640 1 --2- 172.21.15.116:0/2456429890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4abc0b3d20 0x7f4aa8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.009+0000 7f4ac128a640 1 --2- 172.21.15.116:0/2456429890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa8003970 0x7f4aa8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.009+0000 7f4ac0a89640 1 -- 172.21.15.116:0/2456429890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4abc0b3d20 msgr2=0x7f4aa8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.009+0000 7f4ac0a89640 1 --2- 172.21.15.116:0/2456429890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4abc0b3d20 0x7f4aa8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.009+0000 7f4ac0a89640 1 -- 172.21.15.116:0/2456429890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa8003970 msgr2=0x7f4aa8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.009+0000 7f4ac0a89640 1 --2- 172.21.15.116:0/2456429890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa8003970 0x7f4aa8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.009+0000 7f4ac0a89640 1 -- 172.21.15.116:0/2456429890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa8003520 con 0x7f4aa80047c0 2023-11-28T17:11:00.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.009+0000 7f4ac128a640 1 --2- 172.21.15.116:0/2456429890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa8003970 0x7f4aa8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.009+0000 7f4ac1a8b640 1 --2- 172.21.15.116:0/2456429890 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4abc0b3d20 0x7f4aa8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.010+0000 7f4ac0a89640 1 --2- 172.21.15.116:0/2456429890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa80047c0 0x7f4aa8004c40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4aac009a90 tx=0x7f4aa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=774f3758863a0bb5 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.010+0000 7f4ab3fff640 1 -- 172.21.15.116:0/2456429890 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac02c680 con 0x7f4aa80047c0 2023-11-28T17:11:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.010+0000 7f4ab3fff640 1 -- 172.21.15.116:0/2456429890 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aac02c7e0 con 0x7f4aa80047c0 2023-11-28T17:11:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.010+0000 7f4ab3fff640 1 -- 172.21.15.116:0/2456429890 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac02cb70 con 0x7f4aa80047c0 2023-11-28T17:11:00.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.011+0000 7f4ac34ee640 1 -- 172.21.15.116:0/2456429890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa80047c0 msgr2=0x7f4aa8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.011+0000 7f4ac34ee640 1 --2- 172.21.15.116:0/2456429890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa80047c0 0x7f4aa8004c40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4aac009a90 tx=0x7f4aa80057f0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.011+0000 7f4ac1a8b640 1 -- 172.21.15.116:0/2456429890 reap_dead start 2023-11-28T17:11:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.011+0000 7f4ac34ee640 1 -- 172.21.15.116:0/2456429890 shutdown_connections 2023-11-28T17:11:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.011+0000 7f4ac34ee640 1 -- 172.21.15.116:0/2456429890 >> 172.21.15.116:0/2456429890 conn(0x7f4abc06a4c0 msgr2=0x7f4abc06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.011+0000 7f4ac34ee640 1 -- 172.21.15.116:0/2456429890 shutdown_connections 2023-11-28T17:11:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.011+0000 7f4ac34ee640 1 -- 172.21.15.116:0/2456429890 wait complete. 2023-11-28T17:11:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.013+0000 7f4ac34ee640 1 Processor -- start 2023-11-28T17:11:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.013+0000 7f4ac34ee640 1 -- start start 2023-11-28T17:11:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.014+0000 7f4ac34ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa80047c0 0x7f4abc0cd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.014+0000 7f4ac34ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4abc0a8460 0x7f4abc1560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.014+0000 7f4ac34ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc160dc0 0x7f4abc161240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.014+0000 7f4ac34ee640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc161780 con 0x7f4abc160dc0 2023-11-28T17:11:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.014+0000 7f4ac34ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc0a97c0 con 0x7f4aa80047c0 2023-11-28T17:11:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.014+0000 7f4ac34ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc0a88e0 con 0x7f4abc0a8460 2023-11-28T17:11:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.014+0000 7f4ac1a8b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc160dc0 0x7f4abc161240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.014+0000 7f4ac1a8b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc160dc0 0x7f4abc161240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54516/0 (socket says 172.21.15.116:54516) 2023-11-28T17:11:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.014+0000 7f4ac1a8b640 1 -- 172.21.15.116:0/4089851513 learned_addr learned my addr 172.21.15.116:0/4089851513 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.014+0000 7f4ac1a8b640 1 -- 172.21.15.116:0/4089851513 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4abc0a8460 msgr2=0x7f4abc1560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.014+0000 7f4ac1a8b640 1 --2- 172.21.15.116:0/4089851513 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4abc0a8460 0x7f4abc1560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.014+0000 7f4ac1a8b640 1 -- 172.21.15.116:0/4089851513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa80047c0 msgr2=0x7f4abc0cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:00.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.014+0000 7f4ac128a640 1 --2- 172.21.15.116:0/4089851513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa80047c0 0x7f4abc0cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.015+0000 7f4ac1a8b640 1 --2- 172.21.15.116:0/4089851513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa80047c0 0x7f4abc0cd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.015+0000 7f4ac1a8b640 1 -- 172.21.15.116:0/4089851513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc1565f0 con 0x7f4abc160dc0 2023-11-28T17:11:00.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.015+0000 7f4ac1a8b640 1 --2- 172.21.15.116:0/4089851513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc160dc0 0x7f4abc161240 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4abc162160 tx=0x7f4ab400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:00.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.015+0000 7f4ab1ffb640 1 -- 172.21.15.116:0/4089851513 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab4001990 con 0x7f4abc160dc0 2023-11-28T17:11:00.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.015+0000 7f4ab1ffb640 1 -- 172.21.15.116:0/4089851513 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab400bb80 con 0x7f4abc160dc0 2023-11-28T17:11:00.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.015+0000 7f4ab1ffb640 1 -- 172.21.15.116:0/4089851513 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab4007070 con 0x7f4abc160dc0 2023-11-28T17:11:00.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.015+0000 7f4ac34ee640 1 -- 172.21.15.116:0/4089851513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4abc0a8bb0 con 0x7f4abc160dc0 2023-11-28T17:11:00.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.015+0000 7f4ac34ee640 1 -- 172.21.15.116:0/4089851513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4abc0a9170 con 0x7f4abc160dc0 2023-11-28T17:11:00.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.016+0000 7f4ac34ee640 1 -- 172.21.15.116:0/4089851513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4abc0ac700 con 0x7f4abc160dc0 2023-11-28T17:11:00.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.017+0000 7f4ab1ffb640 1 -- 172.21.15.116:0/4089851513 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4ab40071d0 con 0x7f4abc160dc0 2023-11-28T17:11:00.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.018+0000 7f4ab1ffb640 1 --2- 172.21.15.116:0/4089851513 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4aa006b2d0 0x7f4aa006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.018+0000 7f4ab1ffb640 1 -- 172.21.15.116:0/4089851513 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4ab4089fc0 con 0x7f4abc160dc0 2023-11-28T17:11:00.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.018+0000 7f4ac128a640 1 --2- 172.21.15.116:0/4089851513 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4aa006b2d0 0x7f4aa006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.019+0000 7f4ac128a640 1 --2- 172.21.15.116:0/4089851513 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4aa006b2d0 0x7f4aa006d790 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009a40 tx=0x7f4ab8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:00.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.023+0000 7f4ab1ffb640 1 -- 172.21.15.116:0/4089851513 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ab4054db0 con 0x7f4abc160dc0 2023-11-28T17:11:00.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.252+0000 7f4ac34ee640 1 -- 172.21.15.116:0/4089851513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4abc0693d0 con 0x7f4abc160dc0 2023-11-28T17:11:00.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.253+0000 7f4ab1ffb640 1 -- 172.21.15.116:0/4089851513 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f4ab40583d0 con 0x7f4abc160dc0 2023-11-28T17:11:00.254 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:00.255 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/491158514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":491158514},{"type":"v1","addr":"172.21.15.156:6834","nonce":491158514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3927645596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3927645596},{"type":"v1","addr":"172.21.15.116:6835","nonce":3927645596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4738,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/141796020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":141796020},{"type":"v1","addr":"172.21.15.156:6835","nonce":141796020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:10:19.725913+0000","modified":"2023-11-28T17:10:22.808343+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6837/3177126349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3177126349},{"type":"v1","addr":"172.21.15.116:6837","nonce":3177126349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T17:11:00.255 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 15 2023-11-28T17:11:00.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.255+0000 7f4ac34ee640 1 -- 172.21.15.116:0/4089851513 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4aa006b2d0 msgr2=0x7f4aa006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:00.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.256+0000 7f4ac34ee640 1 --2- 172.21.15.116:0/4089851513 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4aa006b2d0 0x7f4aa006d790 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009a40 tx=0x7f4ab8008040 comp rx=0 tx=0).stop 2023-11-28T17:11:00.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.256+0000 7f4ac34ee640 1 -- 172.21.15.116:0/4089851513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc160dc0 msgr2=0x7f4abc161240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:00.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.256+0000 7f4ac34ee640 1 --2- 172.21.15.116:0/4089851513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc160dc0 0x7f4abc161240 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4abc162160 tx=0x7f4ab400b4a0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.256+0000 7f4ac34ee640 1 -- 172.21.15.116:0/4089851513 shutdown_connections 2023-11-28T17:11:00.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.256+0000 7f4ac34ee640 1 --2- 172.21.15.116:0/4089851513 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4aa006b2d0 0x7f4aa006d790 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.256+0000 7f4ac34ee640 1 --2- 172.21.15.116:0/4089851513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4abc160dc0 0x7f4abc161240 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.256+0000 7f4ac34ee640 1 --2- 172.21.15.116:0/4089851513 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4abc0a8460 0x7f4abc1560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.256+0000 7f4ac34ee640 1 --2- 172.21.15.116:0/4089851513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa80047c0 0x7f4abc0cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.256+0000 7f4ac34ee640 1 -- 172.21.15.116:0/4089851513 >> 172.21.15.116:0/4089851513 conn(0x7f4abc06a4c0 msgr2=0x7f4abc156a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:00.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.256+0000 7f4ac34ee640 1 -- 172.21.15.116:0/4089851513 shutdown_connections 2023-11-28T17:11:00.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.256+0000 7f4ac34ee640 1 -- 172.21.15.116:0/4089851513 wait complete. 2023-11-28T17:11:00.277 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:11:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.489+0000 7f209fba9640 1 Processor -- start 2023-11-28T17:11:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.490+0000 7f209fba9640 1 -- start start 2023-11-28T17:11:00.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.491+0000 7f209fba9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f209815d510 0x7f20980d1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.491+0000 7f209fba9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20980d0310 0x7f20980d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.491+0000 7f209fba9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20980ce110 con 0x7f20980ceed0 2023-11-28T17:11:00.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.491+0000 7f209fba9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20980ceab0 con 0x7f209815d510 2023-11-28T17:11:00.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.491+0000 7f209fba9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20980d0740 con 0x7f20980d0310 2023-11-28T17:11:00.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.491+0000 7f209d945640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f209815d510 0x7f20980d1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.491+0000 7f209d945640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f209815d510 0x7f20980d1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54532/0 (socket says 172.21.15.116:54532) 2023-11-28T17:11:00.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.491+0000 7f209d945640 1 -- 172.21.15.116:0/390482382 learned_addr learned my addr 172.21.15.116:0/390482382 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:00.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.492+0000 7f209e146640 1 --2- 172.21.15.116:0/390482382 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20980d0310 0x7f20980d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.492+0000 7f209d945640 1 -- 172.21.15.116:0/390482382 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20980d0310 msgr2=0x7f20980d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:00.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.492+0000 7f209d945640 1 --2- 172.21.15.116:0/390482382 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20980d0310 0x7f20980d1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.492+0000 7f209d945640 1 -- 172.21.15.116:0/390482382 >> v1:172.21.15.156:6789/0 conn(0x7f20980ceed0 legacy=0x7f20980cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:00.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.492+0000 7f209d945640 1 -- 172.21.15.116:0/390482382 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20980ce3f0 con 0x7f209815d510 2023-11-28T17:11:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.493+0000 7f209d945640 1 --2- 172.21.15.116:0/390482382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f209815d510 0x7f20980d1730 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f208c009e50 tx=0x7f208c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1f5700b4987f2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.493+0000 7f209c943640 1 -- 172.21.15.116:0/390482382 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c02ca80 con 0x7f209815d510 2023-11-28T17:11:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.493+0000 7f209c943640 1 -- 172.21.15.116:0/390482382 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208c02cbe0 con 0x7f209815d510 2023-11-28T17:11:00.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.493+0000 7f209c943640 1 -- 172.21.15.116:0/390482382 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208c0343f0 con 0x7f209815d510 2023-11-28T17:11:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.494+0000 7f209fba9640 1 -- 172.21.15.116:0/390482382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f209815d510 msgr2=0x7f20980d1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.494+0000 7f209fba9640 1 --2- 172.21.15.116:0/390482382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f209815d510 0x7f20980d1730 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f208c009e50 tx=0x7f208c02bee0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.494+0000 7f209fba9640 1 -- 172.21.15.116:0/390482382 shutdown_connections 2023-11-28T17:11:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.494+0000 7f209fba9640 1 --2- 172.21.15.116:0/390482382 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20980d0310 0x7f20980d1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.494+0000 7f209fba9640 1 --2- 172.21.15.116:0/390482382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f209815d510 0x7f20980d1730 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.494+0000 7f209fba9640 1 -- 172.21.15.116:0/390482382 >> 172.21.15.116:0/390482382 conn(0x7f209809c460 msgr2=0x7f20980d3190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.494+0000 7f209fba9640 1 -- 172.21.15.116:0/390482382 shutdown_connections 2023-11-28T17:11:00.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.494+0000 7f209fba9640 1 -- 172.21.15.116:0/390482382 wait complete. 2023-11-28T17:11:00.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.496+0000 7f209fba9640 1 Processor -- start 2023-11-28T17:11:00.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.496+0000 7f209fba9640 1 -- start start 2023-11-28T17:11:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.497+0000 7f209fba9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20980ceed0 0x7f20980ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.497+0000 7f209fba9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20980d0310 0x7f2098159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.497+0000 7f209fba9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f209815d510 0x7f20981f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.497+0000 7f209fba9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2098159cb0 con 0x7f209815d510 2023-11-28T17:11:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.497+0000 7f209fba9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20980c7f00 con 0x7f20980ceed0 2023-11-28T17:11:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.497+0000 7f209fba9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20980ca260 con 0x7f20980d0310 2023-11-28T17:11:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.497+0000 7f209e146640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f209815d510 0x7f20981f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.497+0000 7f209e146640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f209815d510 0x7f20981f2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54540/0 (socket says 172.21.15.116:54540) 2023-11-28T17:11:00.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.497+0000 7f209e146640 1 -- 172.21.15.116:0/3411473152 learned_addr learned my addr 172.21.15.116:0/3411473152 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.497+0000 7f209d945640 1 --2- 172.21.15.116:0/3411473152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20980ceed0 0x7f20980ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.497+0000 7f209e146640 1 -- 172.21.15.116:0/3411473152 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20980d0310 msgr2=0x7f2098159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.497+0000 7f209e146640 1 --2- 172.21.15.116:0/3411473152 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20980d0310 0x7f2098159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.497+0000 7f209e146640 1 -- 172.21.15.116:0/3411473152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20980ceed0 msgr2=0x7f20980ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.498+0000 7f209e146640 1 --2- 172.21.15.116:0/3411473152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20980ceed0 0x7f20980ccb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.498+0000 7f209e146640 1 -- 172.21.15.116:0/3411473152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c009b30 con 0x7f209815d510 2023-11-28T17:11:00.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.498+0000 7f209e146640 1 --2- 172.21.15.116:0/3411473152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f209815d510 0x7f20981f2e00 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f209000c7f0 tx=0x7f209000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.498+0000 7f20967fc640 1 -- 172.21.15.116:0/3411473152 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090004220 con 0x7f209815d510 2023-11-28T17:11:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.498+0000 7f20967fc640 1 -- 172.21.15.116:0/3411473152 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2090004380 con 0x7f209815d510 2023-11-28T17:11:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.498+0000 7f20967fc640 1 -- 172.21.15.116:0/3411473152 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090007860 con 0x7f209815d510 2023-11-28T17:11:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.498+0000 7f209fba9640 1 -- 172.21.15.116:0/3411473152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20980ca540 con 0x7f209815d510 2023-11-28T17:11:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.498+0000 7f209fba9640 1 -- 172.21.15.116:0/3411473152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20980cab00 con 0x7f209815d510 2023-11-28T17:11:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.499+0000 7f209fba9640 1 -- 172.21.15.116:0/3411473152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20980cad20 con 0x7f209815d510 2023-11-28T17:11:00.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.500+0000 7f20967fc640 1 -- 172.21.15.116:0/3411473152 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2090007bd0 con 0x7f209815d510 2023-11-28T17:11:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.501+0000 7f20967fc640 1 --2- 172.21.15.116:0/3411473152 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f207c06b2a0 0x7f207c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.501+0000 7f20967fc640 1 -- 172.21.15.116:0/3411473152 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2090089ba0 con 0x7f209815d510 2023-11-28T17:11:00.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.501+0000 7f209d945640 1 --2- 172.21.15.116:0/3411473152 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f207c06b2a0 0x7f207c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.503 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.502+0000 7f209d945640 1 --2- 172.21.15.116:0/3411473152 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f207c06b2a0 0x7f207c06d760 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f208c035000 tx=0x7f208c030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:00.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.506+0000 7f20967fc640 1 -- 172.21.15.116:0/3411473152 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2090054980 con 0x7f209815d510 2023-11-28T17:11:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.697+0000 7f209fba9640 1 -- 172.21.15.116:0/3411473152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2098067460 con 0x7f209815d510 2023-11-28T17:11:00.701 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.700+0000 7f20967fc640 1 -- 172.21.15.116:0/3411473152 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16770 (secure 0 0 0) 0x7f209001e090 con 0x7f209815d510 2023-11-28T17:11:00.702 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:00.702 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":36,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:10:44.069312+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:10:17.338444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-28T17:10:18.766338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.1d","mappings":[{"from":4,"to":5}]},{"pgid":"5.37","mappings":[{"from":4,"to":5}]},{"pgid":"5.3e","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:11:00.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.703+0000 7f209fba9640 1 -- 172.21.15.116:0/3411473152 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f207c06b2a0 msgr2=0x7f207c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:00.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.703+0000 7f209fba9640 1 --2- 172.21.15.116:0/3411473152 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f207c06b2a0 0x7f207c06d760 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f208c035000 tx=0x7f208c030000 comp rx=0 tx=0).stop 2023-11-28T17:11:00.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.703+0000 7f209fba9640 1 -- 172.21.15.116:0/3411473152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f209815d510 msgr2=0x7f20981f2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:00.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.703+0000 7f209fba9640 1 --2- 172.21.15.116:0/3411473152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f209815d510 0x7f20981f2e00 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f209000c7f0 tx=0x7f209000ccb0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.704+0000 7f209fba9640 1 -- 172.21.15.116:0/3411473152 shutdown_connections 2023-11-28T17:11:00.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.704+0000 7f209fba9640 1 --2- 172.21.15.116:0/3411473152 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f207c06b2a0 0x7f207c06d760 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.704+0000 7f209fba9640 1 --2- 172.21.15.116:0/3411473152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f209815d510 0x7f20981f2e00 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.704+0000 7f209fba9640 1 --2- 172.21.15.116:0/3411473152 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20980d0310 0x7f2098159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.704+0000 7f209fba9640 1 --2- 172.21.15.116:0/3411473152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20980ceed0 0x7f20980ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.704+0000 7f209fba9640 1 -- 172.21.15.116:0/3411473152 >> 172.21.15.116:0/3411473152 conn(0x7f209809c460 msgr2=0x7f209806c730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:00.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.704+0000 7f209fba9640 1 -- 172.21.15.116:0/3411473152 shutdown_connections 2023-11-28T17:11:00.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.704+0000 7f209fba9640 1 -- 172.21.15.116:0/3411473152 wait complete. 2023-11-28T17:11:00.725 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-28T17:11:00.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:11:00.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.934+0000 7f7a05885640 1 Processor -- start 2023-11-28T17:11:00.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.934+0000 7f7a05885640 1 -- start start 2023-11-28T17:11:00.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.935+0000 7f7a05885640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a00159f30 0x7f7a000701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.935+0000 7f7a05885640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7a0015ae90 0x7f7a000ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.935+0000 7f7a05885640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a000ceb90 0x7f7a000cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.935+0000 7f7a05885640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a00070730 con 0x7f7a000ceb90 2023-11-28T17:11:00.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.935+0000 7f7a05885640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a00070ee0 con 0x7f7a0015ae90 2023-11-28T17:11:00.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.935+0000 7f7a05885640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a000cf550 con 0x7f7a00159f30 2023-11-28T17:11:00.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.936+0000 7f7a05084640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a000ceb90 0x7f7a000cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.936+0000 7f7a05084640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a000ceb90 0x7f7a000cf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54544/0 (socket says 172.21.15.116:54544) 2023-11-28T17:11:00.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.936+0000 7f7a05084640 1 -- 172.21.15.116:0/3489726052 learned_addr learned my addr 172.21.15.116:0/3489726052 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:00.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.936+0000 7f79fffff640 1 --2- 172.21.15.116:0/3489726052 >> v2:172.21.15.156:3300/0 conn(0x7f7a0015ae90 0x7f7a000ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.936+0000 7f7a04883640 1 --2- 172.21.15.116:0/3489726052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a00159f30 0x7f7a000701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.936+0000 7f7a05084640 1 -- 172.21.15.116:0/3489726052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a00159f30 msgr2=0x7f7a000701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:00.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.936+0000 7f7a05084640 1 --2- 172.21.15.116:0/3489726052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a00159f30 0x7f7a000701f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.936+0000 7f7a05084640 1 -- 172.21.15.116:0/3489726052 >> v2:172.21.15.156:3300/0 conn(0x7f7a0015ae90 msgr2=0x7f7a000ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:00.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.936+0000 7f7a05084640 1 --2- 172.21.15.116:0/3489726052 >> v2:172.21.15.156:3300/0 conn(0x7f7a0015ae90 0x7f7a000ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.936+0000 7f7a05084640 1 -- 172.21.15.116:0/3489726052 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a000cd8e0 con 0x7f7a000ceb90 2023-11-28T17:11:00.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.937+0000 7f7a05084640 1 --2- 172.21.15.116:0/3489726052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a000ceb90 0x7f7a000cf010 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7a0006fee0 tx=0x7f79f802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a81a6436190f3a1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:00.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.938+0000 7f79ff7fe640 1 -- 172.21.15.116:0/3489726052 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f802ca70 con 0x7f7a000ceb90 2023-11-28T17:11:00.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.938+0000 7f79ff7fe640 1 -- 172.21.15.116:0/3489726052 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f802cbd0 con 0x7f7a000ceb90 2023-11-28T17:11:00.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.938+0000 7f79ff7fe640 1 -- 172.21.15.116:0/3489726052 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f80343f0 con 0x7f7a000ceb90 2023-11-28T17:11:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.938+0000 7f7a05885640 1 -- 172.21.15.116:0/3489726052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a000ceb90 msgr2=0x7f7a000cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.938+0000 7f7a05885640 1 --2- 172.21.15.116:0/3489726052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a000ceb90 0x7f7a000cf010 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7a0006fee0 tx=0x7f79f802bed0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.938+0000 7f7a05885640 1 -- 172.21.15.116:0/3489726052 shutdown_connections 2023-11-28T17:11:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.938+0000 7f7a05885640 1 --2- 172.21.15.116:0/3489726052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a000ceb90 0x7f7a000cf010 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.938+0000 7f7a05885640 1 --2- 172.21.15.116:0/3489726052 >> v2:172.21.15.156:3300/0 conn(0x7f7a0015ae90 0x7f7a000ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.938+0000 7f7a05885640 1 --2- 172.21.15.116:0/3489726052 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a00159f30 0x7f7a000701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.938+0000 7f7a05885640 1 -- 172.21.15.116:0/3489726052 >> 172.21.15.116:0/3489726052 conn(0x7f7a000b0950 msgr2=0x7f7a000b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.938+0000 7f7a05885640 1 -- 172.21.15.116:0/3489726052 shutdown_connections 2023-11-28T17:11:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.938+0000 7f7a05885640 1 -- 172.21.15.116:0/3489726052 wait complete. 2023-11-28T17:11:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.940+0000 7f7a05885640 1 Processor -- start 2023-11-28T17:11:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.940+0000 7f7a05885640 1 -- start start 2023-11-28T17:11:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.941+0000 7f7a05885640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a000ceb90 0x7f7a001f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.941+0000 7f7a05885640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a00159f30 0x7f7a00150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.941+0000 7f7a05885640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a0015ae90 0x7f7a000ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.941+0000 7f7a05885640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0015b500 con 0x7f7a00159f30 2023-11-28T17:11:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.941+0000 7f7a05885640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a001514b0 con 0x7f7a0015ae90 2023-11-28T17:11:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.941+0000 7f7a05885640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a001f64e0 con 0x7f7a000ceb90 2023-11-28T17:11:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.942+0000 7f79fffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a00159f30 0x7f7a00150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.942+0000 7f79fffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a00159f30 0x7f7a00150dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54548/0 (socket says 172.21.15.116:54548) 2023-11-28T17:11:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.942+0000 7f79fffff640 1 -- 172.21.15.116:0/3116029449 learned_addr learned my addr 172.21.15.116:0/3116029449 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.942+0000 7f79fffff640 1 -- 172.21.15.116:0/3116029449 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a000ceb90 msgr2=0x7f7a001f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.942+0000 7f7a04883640 1 --2- 172.21.15.116:0/3116029449 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a000ceb90 0x7f7a001f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.942+0000 7f7a05084640 1 --2- 172.21.15.116:0/3116029449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a0015ae90 0x7f7a000ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.942+0000 7f79fffff640 1 --2- 172.21.15.116:0/3116029449 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a000ceb90 0x7f7a001f3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.942+0000 7f79fffff640 1 -- 172.21.15.116:0/3116029449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a0015ae90 msgr2=0x7f7a000ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.942+0000 7f79fffff640 1 --2- 172.21.15.116:0/3116029449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a0015ae90 0x7f7a000ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.942+0000 7f79fffff640 1 -- 172.21.15.116:0/3116029449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f8009b30 con 0x7f7a00159f30 2023-11-28T17:11:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.942+0000 7f79fffff640 1 --2- 172.21.15.116:0/3116029449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a00159f30 0x7f7a00150dd0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00d5d0 tx=0x7f79ec00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.943+0000 7f79fd7fa640 1 -- 172.21.15.116:0/3116029449 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec004240 con 0x7f7a00159f30 2023-11-28T17:11:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.943+0000 7f79fd7fa640 1 -- 172.21.15.116:0/3116029449 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ec0043a0 con 0x7f7a00159f30 2023-11-28T17:11:00.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.943+0000 7f7a05885640 1 -- 172.21.15.116:0/3116029449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a001f67c0 con 0x7f7a00159f30 2023-11-28T17:11:00.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.943+0000 7f79fd7fa640 1 -- 172.21.15.116:0/3116029449 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec003830 con 0x7f7a00159f30 2023-11-28T17:11:00.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.943+0000 7f7a05885640 1 -- 172.21.15.116:0/3116029449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a001f6d80 con 0x7f7a00159f30 2023-11-28T17:11:00.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.944+0000 7f7a05885640 1 -- 172.21.15.116:0/3116029449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a0009d0b0 con 0x7f7a00159f30 2023-11-28T17:11:00.946 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.945+0000 7f79fd7fa640 1 -- 172.21.15.116:0/3116029449 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f79ec003990 con 0x7f7a00159f30 2023-11-28T17:11:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.946+0000 7f79fd7fa640 1 --2- 172.21.15.116:0/3116029449 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f79e406b2a0 0x7f79e406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.946+0000 7f79fd7fa640 1 -- 172.21.15.116:0/3116029449 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f79ec089e30 con 0x7f7a00159f30 2023-11-28T17:11:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.946+0000 7f7a04883640 1 --2- 172.21.15.116:0/3116029449 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f79e406b2a0 0x7f79e406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.947+0000 7f7a04883640 1 --2- 172.21.15.116:0/3116029449 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f79e406b2a0 0x7f79e406d760 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f79f4006fd0 tx=0x7f79f4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:00.952 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:00.951+0000 7f79fd7fa640 1 -- 172.21.15.116:0/3116029449 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f79ec054b80 con 0x7f7a00159f30 2023-11-28T17:11:01.181 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.180+0000 7f7a05885640 1 -- 172.21.15.116:0/3116029449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a0004e960 con 0x7f7a00159f30 2023-11-28T17:11:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.181+0000 7f79fd7fa640 1 -- 172.21.15.116:0/3116029449 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f79ec0581a0 con 0x7f7a00159f30 2023-11-28T17:11:01.182 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:01.182 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/491158514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":491158514},{"type":"v1","addr":"172.21.15.156:6834","nonce":491158514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3927645596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3927645596},{"type":"v1","addr":"172.21.15.116:6835","nonce":3927645596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4738,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/141796020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":141796020},{"type":"v1","addr":"172.21.15.156:6835","nonce":141796020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:10:19.725913+0000","modified":"2023-11-28T17:10:22.808343+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6837/3177126349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3177126349},{"type":"v1","addr":"172.21.15.116:6837","nonce":3177126349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T17:11:01.182 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 15 2023-11-28T17:11:01.184 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.183+0000 7f7a05885640 1 -- 172.21.15.116:0/3116029449 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f79e406b2a0 msgr2=0x7f79e406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:01.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.183+0000 7f7a05885640 1 --2- 172.21.15.116:0/3116029449 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f79e406b2a0 0x7f79e406d760 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f79f4006fd0 tx=0x7f79f4008040 comp rx=0 tx=0).stop 2023-11-28T17:11:01.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.183+0000 7f7a05885640 1 -- 172.21.15.116:0/3116029449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a00159f30 msgr2=0x7f7a00150dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:01.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.183+0000 7f7a05885640 1 --2- 172.21.15.116:0/3116029449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a00159f30 0x7f7a00150dd0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00d5d0 tx=0x7f79ec00da90 comp rx=0 tx=0).stop 2023-11-28T17:11:01.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.183+0000 7f7a05885640 1 -- 172.21.15.116:0/3116029449 shutdown_connections 2023-11-28T17:11:01.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.183+0000 7f7a05885640 1 --2- 172.21.15.116:0/3116029449 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f79e406b2a0 0x7f79e406d760 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.183+0000 7f7a05885640 1 --2- 172.21.15.116:0/3116029449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a0015ae90 0x7f7a000ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.183+0000 7f7a05885640 1 --2- 172.21.15.116:0/3116029449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a00159f30 0x7f7a00150dd0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.183+0000 7f7a05885640 1 --2- 172.21.15.116:0/3116029449 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a000ceb90 0x7f7a001f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.185 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.183+0000 7f7a05885640 1 -- 172.21.15.116:0/3116029449 >> 172.21.15.116:0/3116029449 conn(0x7f7a000b0950 msgr2=0x7f7a000733f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:01.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.183+0000 7f7a05885640 1 -- 172.21.15.116:0/3116029449 shutdown_connections 2023-11-28T17:11:01.186 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.184+0000 7f7a05885640 1 -- 172.21.15.116:0/3116029449 wait complete. 2023-11-28T17:11:01.204 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:11:01.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.413+0000 7eff10f24640 1 Processor -- start 2023-11-28T17:11:01.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.413+0000 7eff10f24640 1 -- start start 2023-11-28T17:11:01.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.414+0000 7eff10f24640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7eff0c06c960 0x7eff0c15a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:01.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.415+0000 7eff10f24640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0c0aea20 con 0x7eff0c13cee0 2023-11-28T17:11:01.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.415+0000 7eff10f24640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0c13d2c0 con 0x7eff0c140200 2023-11-28T17:11:01.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.415+0000 7eff10f24640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff0c15b3a0 con 0x7eff0c06c960 2023-11-28T17:11:01.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.415+0000 7eff0a59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7eff0c13cee0 0x7eff0c06c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54214/0 (socket says 172.21.15.116:54214) 2023-11-28T17:11:01.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.415+0000 7eff0a59c640 1 -- 172.21.15.116:0/1595233377 learned_addr learned my addr 172.21.15.116:0/1595233377 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:01.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.415+0000 7eff03fff640 1 --2- 172.21.15.116:0/1595233377 >> v2:172.21.15.116:3300/0 conn(0x7eff0c06c960 0x7eff0c15a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:01.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.415+0000 7eff09d9b640 1 -- 172.21.15.116:0/1595233377 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045422516 0 0) 0x7eff0c0aea20 con 0x7eff0c13cee0 2023-11-28T17:11:01.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.415+0000 7eff09d9b640 1 -- 172.21.15.116:0/1595233377 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efef8003520 con 0x7eff0c13cee0 2023-11-28T17:11:01.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.415+0000 7eff03fff640 1 -- 172.21.15.116:0/1595233377 >> v1:172.21.15.156:6789/0 conn(0x7eff0c140200 legacy=0x7eff0c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:01.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.415+0000 7eff03fff640 1 -- 172.21.15.116:0/1595233377 >> v1:172.21.15.116:6789/0 conn(0x7eff0c13cee0 legacy=0x7eff0c06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:01.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.415+0000 7eff03fff640 1 -- 172.21.15.116:0/1595233377 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0c15b0c0 con 0x7eff0c06c960 2023-11-28T17:11:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.417+0000 7eff03fff640 1 --2- 172.21.15.116:0/1595233377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c06c960 0x7eff0c15a490 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7efef40097a0 tx=0x7efef402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d66a8c4a842b48 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.417+0000 7eff09d9b640 1 -- 172.21.15.116:0/1595233377 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef402e070 con 0x7eff0c06c960 2023-11-28T17:11:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.417+0000 7eff09d9b640 1 -- 172.21.15.116:0/1595233377 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef40315a0 con 0x7eff0c06c960 2023-11-28T17:11:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.417+0000 7eff09d9b640 1 -- 172.21.15.116:0/1595233377 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4031930 con 0x7eff0c06c960 2023-11-28T17:11:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.418+0000 7eff10f24640 1 -- 172.21.15.116:0/1595233377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c06c960 msgr2=0x7eff0c15a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.418+0000 7eff10f24640 1 --2- 172.21.15.116:0/1595233377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c06c960 0x7eff0c15a490 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7efef40097a0 tx=0x7efef402b840 comp rx=0 tx=0).stop 2023-11-28T17:11:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.418+0000 7eff10f24640 1 -- 172.21.15.116:0/1595233377 shutdown_connections 2023-11-28T17:11:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.418+0000 7eff10f24640 1 --2- 172.21.15.116:0/1595233377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c06c960 0x7eff0c15a490 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.418+0000 7eff10f24640 1 -- 172.21.15.116:0/1595233377 >> 172.21.15.116:0/1595233377 conn(0x7eff0c06a4c0 msgr2=0x7eff0c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.418+0000 7eff10f24640 1 -- 172.21.15.116:0/1595233377 shutdown_connections 2023-11-28T17:11:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.418+0000 7eff10f24640 1 -- 172.21.15.116:0/1595233377 wait complete. 2023-11-28T17:11:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.420+0000 7eff10f24640 1 Processor -- start 2023-11-28T17:11:01.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.420+0000 7eff10f24640 1 -- start start 2023-11-28T17:11:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.421+0000 7eff10f24640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff0c06c960 0x7eff0c14f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.421+0000 7eff10f24640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c13cee0 0x7eff0c1594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.421+0000 7eff10f24640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff0c140200 0x7eff0c159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.421+0000 7eff10f24640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c1f79d0 con 0x7eff0c13cee0 2023-11-28T17:11:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.421+0000 7eff10f24640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c1f6d50 con 0x7eff0c06c960 2023-11-28T17:11:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.421+0000 7eff10f24640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c1f7030 con 0x7eff0c140200 2023-11-28T17:11:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.421+0000 7eff03fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c13cee0 0x7eff0c1594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.421+0000 7eff03fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c13cee0 0x7eff0c1594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54578/0 (socket says 172.21.15.116:54578) 2023-11-28T17:11:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.421+0000 7eff03fff640 1 -- 172.21.15.116:0/485076620 learned_addr learned my addr 172.21.15.116:0/485076620 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.422+0000 7eff0a59c640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff0c06c960 0x7eff0c14f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.422+0000 7eff0ad9d640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff0c140200 0x7eff0c159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.422+0000 7eff03fff640 1 -- 172.21.15.116:0/485076620 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff0c140200 msgr2=0x7eff0c159a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.422+0000 7eff03fff640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff0c140200 0x7eff0c159a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.422+0000 7eff03fff640 1 -- 172.21.15.116:0/485076620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff0c06c960 msgr2=0x7eff0c14f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.422+0000 7eff03fff640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff0c06c960 0x7eff0c14f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.422+0000 7eff03fff640 1 -- 172.21.15.116:0/485076620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef4009400 con 0x7eff0c13cee0 2023-11-28T17:11:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.422+0000 7eff0a59c640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff0c06c960 0x7eff0c14f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.422+0000 7eff03fff640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c13cee0 0x7eff0c1594d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7efef4000c00 tx=0x7efef40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.422+0000 7eff02ffd640 1 -- 172.21.15.116:0/485076620 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4032630 con 0x7eff0c13cee0 2023-11-28T17:11:01.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.422+0000 7eff02ffd640 1 -- 172.21.15.116:0/485076620 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef40327b0 con 0x7eff0c13cee0 2023-11-28T17:11:01.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.422+0000 7eff10f24640 1 -- 172.21.15.116:0/485076620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0c1f7310 con 0x7eff0c13cee0 2023-11-28T17:11:01.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.422+0000 7eff02ffd640 1 -- 172.21.15.116:0/485076620 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4032b60 con 0x7eff0c13cee0 2023-11-28T17:11:01.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.423+0000 7eff10f24640 1 -- 172.21.15.116:0/485076620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0c14fec0 con 0x7eff0c13cee0 2023-11-28T17:11:01.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.424+0000 7eff10f24640 1 -- 172.21.15.116:0/485076620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0c0693d0 con 0x7eff0c13cee0 2023-11-28T17:11:01.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.425+0000 7eff02ffd640 1 -- 172.21.15.116:0/485076620 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7efef4032cc0 con 0x7eff0c13cee0 2023-11-28T17:11:01.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.425+0000 7eff02ffd640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7efeec06b2a0 0x7efeec06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:01.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.426+0000 7eff02ffd640 1 -- 172.21.15.116:0/485076620 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7efef40ab530 con 0x7eff0c13cee0 2023-11-28T17:11:01.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.426+0000 7eff0a59c640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7efeec06b2a0 0x7efeec06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:01.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.427+0000 7eff0a59c640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7efeec06b2a0 0x7efeec06d760 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efefc005880 tx=0x7efefc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:01.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.431+0000 7eff02ffd640 1 -- 172.21.15.116:0/485076620 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efef40763d0 con 0x7eff0c13cee0 2023-11-28T17:11:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.654+0000 7eff10f24640 1 -- 172.21.15.116:0/485076620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff0c13d2c0 con 0x7eff0c13cee0 2023-11-28T17:11:01.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.655+0000 7eff02ffd640 1 -- 172.21.15.116:0/485076620 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4919 (secure 0 0 0) 0x7eff0c13d2c0 con 0x7eff0c13cee0 2023-11-28T17:11:01.657 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:01.657 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/491158514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":491158514},{"type":"v1","addr":"172.21.15.156:6834","nonce":491158514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3927645596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3927645596},{"type":"v1","addr":"172.21.15.116:6835","nonce":3927645596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4738,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/141796020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":141796020},{"type":"v1","addr":"172.21.15.156:6835","nonce":141796020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:10:19.725913+0000","modified":"2023-11-28T17:10:22.808343+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5202},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5202":{"gid":5202,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6837/3177126349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3177126349},{"type":"v1","addr":"172.21.15.116:6837","nonce":3177126349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-28T17:11:01.658 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 15 2023-11-28T17:11:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.657+0000 7eff10f24640 1 -- 172.21.15.116:0/485076620 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7efeec06b2a0 msgr2=0x7efeec06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.657+0000 7eff10f24640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7efeec06b2a0 0x7efeec06d760 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efefc005880 tx=0x7efefc009040 comp rx=0 tx=0).stop 2023-11-28T17:11:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.657+0000 7eff10f24640 1 -- 172.21.15.116:0/485076620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c13cee0 msgr2=0x7eff0c1594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.657+0000 7eff10f24640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c13cee0 0x7eff0c1594d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7efef4000c00 tx=0x7efef40046c0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.658+0000 7eff10f24640 1 -- 172.21.15.116:0/485076620 shutdown_connections 2023-11-28T17:11:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.658+0000 7eff10f24640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7efeec06b2a0 0x7efeec06d760 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.658+0000 7eff10f24640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff0c140200 0x7eff0c159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.658+0000 7eff10f24640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff0c13cee0 0x7eff0c1594d0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.658+0000 7eff10f24640 1 --2- 172.21.15.116:0/485076620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff0c06c960 0x7eff0c14f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.658+0000 7eff10f24640 1 -- 172.21.15.116:0/485076620 >> 172.21.15.116:0/485076620 conn(0x7eff0c06a4c0 msgr2=0x7eff0c15c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.658+0000 7eff10f24640 1 -- 172.21.15.116:0/485076620 shutdown_connections 2023-11-28T17:11:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.658+0000 7eff10f24640 1 -- 172.21.15.116:0/485076620 wait complete. 2023-11-28T17:11:01.678 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:11:01.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.888+0000 7f09c00da640 1 Processor -- start 2023-11-28T17:11:01.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.888+0000 7f09c00da640 1 -- start start 2023-11-28T17:11:01.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.889+0000 7f09c00da640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f09b8144430 0x7f09b815f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:01.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.890+0000 7f09c00da640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09b806c250 0x7f09b806c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:01.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.890+0000 7f09c00da640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b8070bd0 con 0x7f09b80703f0 2023-11-28T17:11:01.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.890+0000 7f09c00da640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b8144810 con 0x7f09b8144430 2023-11-28T17:11:01.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.890+0000 7f09c00da640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b806cc20 con 0x7f09b806c250 2023-11-28T17:11:01.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.890+0000 7f09bde76640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f09b8144430 0x7f09b815f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:01.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.890+0000 7f09bde76640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f09b8144430 0x7f09b815f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54594/0 (socket says 172.21.15.116:54594) 2023-11-28T17:11:01.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.890+0000 7f09bd675640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f09b80703f0 0x7f09b8071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54222/0 (socket says 172.21.15.116:54222) 2023-11-28T17:11:01.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.890+0000 7f09bde76640 1 -- 172.21.15.116:0/1896436407 learned_addr learned my addr 172.21.15.116:0/1896436407 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:01.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.890+0000 7f09be677640 1 --2- 172.21.15.116:0/1896436407 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09b806c250 0x7f09b806c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:01.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.890+0000 7f09bde76640 1 -- 172.21.15.116:0/1896436407 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09b806c250 msgr2=0x7f09b806c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:01.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.891+0000 7f09bde76640 1 --2- 172.21.15.116:0/1896436407 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09b806c250 0x7f09b806c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.891+0000 7f09bce74640 1 -- 172.21.15.116:0/1896436407 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269179397 0 0) 0x7f09b8070bd0 con 0x7f09b80703f0 2023-11-28T17:11:01.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.891+0000 7f09bde76640 1 -- 172.21.15.116:0/1896436407 >> v1:172.21.15.116:6789/0 conn(0x7f09b80703f0 legacy=0x7f09b8071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:01.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.891+0000 7f09bde76640 1 -- 172.21.15.116:0/1896436407 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b8070eb0 con 0x7f09b8144430 2023-11-28T17:11:01.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.891+0000 7f09be677640 1 --2- 172.21.15.116:0/1896436407 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09b806c250 0x7f09b806c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:01.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.892+0000 7f09bde76640 1 --2- 172.21.15.116:0/1896436407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b8144430 0x7f09b815f7b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f09ac009870 tx=0x7f09ac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95485c712e6eff0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:01.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.892+0000 7f09bce74640 1 -- 172.21.15.116:0/1896436407 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac030840 con 0x7f09b8144430 2023-11-28T17:11:01.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.892+0000 7f09bce74640 1 -- 172.21.15.116:0/1896436407 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ac0309a0 con 0x7f09b8144430 2023-11-28T17:11:01.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.892+0000 7f09bce74640 1 -- 172.21.15.116:0/1896436407 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac030d30 con 0x7f09b8144430 2023-11-28T17:11:01.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.893+0000 7f09c00da640 1 -- 172.21.15.116:0/1896436407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b8144430 msgr2=0x7f09b815f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:01.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.893+0000 7f09c00da640 1 --2- 172.21.15.116:0/1896436407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b8144430 0x7f09b815f7b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f09ac009870 tx=0x7f09ac02b980 comp rx=0 tx=0).stop 2023-11-28T17:11:01.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.893+0000 7f09c00da640 1 -- 172.21.15.116:0/1896436407 shutdown_connections 2023-11-28T17:11:01.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.893+0000 7f09c00da640 1 --2- 172.21.15.116:0/1896436407 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09b806c250 0x7f09b806c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.893+0000 7f09c00da640 1 --2- 172.21.15.116:0/1896436407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b8144430 0x7f09b815f7b0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.893+0000 7f09c00da640 1 -- 172.21.15.116:0/1896436407 >> 172.21.15.116:0/1896436407 conn(0x7f09b806a4c0 msgr2=0x7f09b806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:01.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.893+0000 7f09c00da640 1 -- 172.21.15.116:0/1896436407 shutdown_connections 2023-11-28T17:11:01.898 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.893+0000 7f09c00da640 1 -- 172.21.15.116:0/1896436407 wait complete. 2023-11-28T17:11:01.902 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.895+0000 7f09c00da640 1 Processor -- start 2023-11-28T17:11:01.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.895+0000 7f09c00da640 1 -- start start 2023-11-28T17:11:01.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.896+0000 7f09c00da640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09b806c250 0x7f09b813c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:01.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.896+0000 7f09c00da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b80703f0 0x7f09b80cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:01.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.896+0000 7f09c00da640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09b8144430 0x7f09b813e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:01.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.896+0000 7f09c00da640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b813deb0 con 0x7f09b80703f0 2023-11-28T17:11:01.904 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.896+0000 7f09c00da640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b813cdd0 con 0x7f09b8144430 2023-11-28T17:11:01.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.896+0000 7f09c00da640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b813d270 con 0x7f09b806c250 2023-11-28T17:11:01.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.896+0000 7f09bd675640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b80703f0 0x7f09b80cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:01.905 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.896+0000 7f09bd675640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b80703f0 0x7f09b80cadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54608/0 (socket says 172.21.15.116:54608) 2023-11-28T17:11:01.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.896+0000 7f09bd675640 1 -- 172.21.15.116:0/2146509421 learned_addr learned my addr 172.21.15.116:0/2146509421 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:01.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.896+0000 7f09bd675640 1 -- 172.21.15.116:0/2146509421 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09b806c250 msgr2=0x7f09b813c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:01.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.896+0000 7f09bd675640 1 --2- 172.21.15.116:0/2146509421 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09b806c250 0x7f09b813c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.896+0000 7f09bd675640 1 -- 172.21.15.116:0/2146509421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09b8144430 msgr2=0x7f09b813e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:01.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.896+0000 7f09bd675640 1 --2- 172.21.15.116:0/2146509421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09b8144430 0x7f09b813e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:01.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.896+0000 7f09bd675640 1 -- 172.21.15.116:0/2146509421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac0094d0 con 0x7f09b80703f0 2023-11-28T17:11:01.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.897+0000 7f09bd675640 1 --2- 172.21.15.116:0/2146509421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b80703f0 0x7f09b80cadc0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f09b400c890 tx=0x7f09b400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:01.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.897+0000 7f09aa7fc640 1 -- 172.21.15.116:0/2146509421 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b400a640 con 0x7f09b80703f0 2023-11-28T17:11:01.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.897+0000 7f09aa7fc640 1 -- 172.21.15.116:0/2146509421 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b400a7c0 con 0x7f09b80703f0 2023-11-28T17:11:01.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.897+0000 7f09aa7fc640 1 -- 172.21.15.116:0/2146509421 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b400ab70 con 0x7f09b80703f0 2023-11-28T17:11:01.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.897+0000 7f09c00da640 1 -- 172.21.15.116:0/2146509421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b813d510 con 0x7f09b80703f0 2023-11-28T17:11:01.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.897+0000 7f09c00da640 1 -- 172.21.15.116:0/2146509421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b813dad0 con 0x7f09b80703f0 2023-11-28T17:11:01.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.898+0000 7f09c00da640 1 -- 172.21.15.116:0/2146509421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b813e120 con 0x7f09b80703f0 2023-11-28T17:11:01.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.899+0000 7f09aa7fc640 1 -- 172.21.15.116:0/2146509421 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f09b400acd0 con 0x7f09b80703f0 2023-11-28T17:11:01.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.900+0000 7f09aa7fc640 1 --2- 172.21.15.116:0/2146509421 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f099c06b2a0 0x7f099c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:01.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.900+0000 7f09aa7fc640 1 -- 172.21.15.116:0/2146509421 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f09b4089180 con 0x7f09b80703f0 2023-11-28T17:11:01.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.900+0000 7f09bde76640 1 --2- 172.21.15.116:0/2146509421 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f099c06b2a0 0x7f099c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:01.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.901+0000 7f09bde76640 1 --2- 172.21.15.116:0/2146509421 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f099c06b2a0 0x7f099c06d760 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f09ac036000 tx=0x7f09ac03b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:01.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:01.904+0000 7f09aa7fc640 1 -- 172.21.15.116:0/2146509421 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09b4053f60 con 0x7f09b80703f0 2023-11-28T17:11:02.085 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.083+0000 7f09c00da640 1 -- 172.21.15.116:0/2146509421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09b80693d0 con 0x7f09b80703f0 2023-11-28T17:11:02.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.084+0000 7f09aa7fc640 1 -- 172.21.15.116:0/2146509421 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16770 (secure 0 0 0) 0x7f09b4057580 con 0x7f09b80703f0 2023-11-28T17:11:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:02.086 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":36,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:10:44.069312+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:10:17.338444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-28T17:10:18.766338+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.1d","mappings":[{"from":4,"to":5}]},{"pgid":"5.37","mappings":[{"from":4,"to":5}]},{"pgid":"5.3e","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:11:02.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.087+0000 7f09c00da640 1 -- 172.21.15.116:0/2146509421 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f099c06b2a0 msgr2=0x7f099c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:02.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.087+0000 7f09c00da640 1 --2- 172.21.15.116:0/2146509421 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f099c06b2a0 0x7f099c06d760 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f09ac036000 tx=0x7f09ac03b040 comp rx=0 tx=0).stop 2023-11-28T17:11:02.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.087+0000 7f09c00da640 1 -- 172.21.15.116:0/2146509421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b80703f0 msgr2=0x7f09b80cadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:02.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.087+0000 7f09c00da640 1 --2- 172.21.15.116:0/2146509421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b80703f0 0x7f09b80cadc0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f09b400c890 tx=0x7f09b400cd50 comp rx=0 tx=0).stop 2023-11-28T17:11:02.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.087+0000 7f09c00da640 1 -- 172.21.15.116:0/2146509421 shutdown_connections 2023-11-28T17:11:02.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.087+0000 7f09c00da640 1 --2- 172.21.15.116:0/2146509421 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f099c06b2a0 0x7f099c06d760 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.087+0000 7f09c00da640 1 --2- 172.21.15.116:0/2146509421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09b8144430 0x7f09b813e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.087+0000 7f09c00da640 1 --2- 172.21.15.116:0/2146509421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b80703f0 0x7f09b80cadc0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.087+0000 7f09c00da640 1 --2- 172.21.15.116:0/2146509421 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09b806c250 0x7f09b813c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.087+0000 7f09c00da640 1 -- 172.21.15.116:0/2146509421 >> 172.21.15.116:0/2146509421 conn(0x7f09b806a4c0 msgr2=0x7f09b815ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:02.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.087+0000 7f09c00da640 1 -- 172.21.15.116:0/2146509421 shutdown_connections 2023-11-28T17:11:02.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.087+0000 7f09c00da640 1 -- 172.21.15.116:0/2146509421 wait complete. 2023-11-28T17:11:02.106 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-28T17:11:02.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.313+0000 7f6b34475640 1 Processor -- start 2023-11-28T17:11:02.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.314+0000 7f6b34475640 1 -- start start 2023-11-28T17:11:02.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.315+0000 7f6b34475640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6b2c15f7b0 0x7f6b2c071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:02.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.315+0000 7f6b34475640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2c15fc10 con 0x7f6b2c144430 2023-11-28T17:11:02.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.315+0000 7f6b34475640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2c070bd0 con 0x7f6b2c0721b0 2023-11-28T17:11:02.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.315+0000 7f6b34475640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b2c144810 con 0x7f6b2c15f7b0 2023-11-28T17:11:02.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.315+0000 7f6b32211640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6b2c144430 0x7f6b2c0703f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54234/0 (socket says 172.21.15.116:54234) 2023-11-28T17:11:02.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.315+0000 7f6b32211640 1 -- 172.21.15.116:0/1529424451 learned_addr learned my addr 172.21.15.116:0/1529424451 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:02.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 -- 172.21.15.116:0/1529424451 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551729302 0 0) 0x7f6b2c15fc10 con 0x7f6b2c144430 2023-11-28T17:11:02.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 -- 172.21.15.116:0/1529424451 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b18003520 con 0x7f6b2c144430 2023-11-28T17:11:02.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b31a10640 1 --2- 172.21.15.116:0/1529424451 >> v2:172.21.15.156:3300/0 conn(0x7f6b2c15f7b0 0x7f6b2c071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:02.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 -- 172.21.15.116:0/1529424451 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2560408015 0 0) 0x7f6b18003520 con 0x7f6b2c144430 2023-11-28T17:11:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 -- 172.21.15.116:0/1529424451 >> v2:172.21.15.156:3300/0 conn(0x7f6b2c15f7b0 msgr2=0x7f6b2c071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 --2- 172.21.15.116:0/1529424451 >> v2:172.21.15.156:3300/0 conn(0x7f6b2c15f7b0 0x7f6b2c071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 -- 172.21.15.116:0/1529424451 >> v1:172.21.15.156:6789/0 conn(0x7f6b2c0721b0 legacy=0x7f6b2c06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 -- 172.21.15.116:0/1529424451 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c070eb0 con 0x7f6b2c144430 2023-11-28T17:11:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 -- 172.21.15.116:0/1529424451 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f6b28001670 con 0x7f6b2c144430 2023-11-28T17:11:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 -- 172.21.15.116:0/1529424451 >> v1:172.21.15.116:6789/0 conn(0x7f6b2c144430 legacy=0x7f6b2c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 --2- 172.21.15.116:0/1529424451 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b18003970 0x7f6b18003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 --2- 172.21.15.116:0/1529424451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b180047c0 0x7f6b18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 --2- 172.21.15.116:0/1529424451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b18005180 0x7f6b1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 -- 172.21.15.116:0/1529424451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c144810 con 0x7f6b18005180 2023-11-28T17:11:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 -- 172.21.15.116:0/1529424451 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1800c0b0 con 0x7f6b180047c0 2023-11-28T17:11:02.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.316+0000 7f6b3120f640 1 -- 172.21.15.116:0/1529424451 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1800c270 con 0x7f6b18003970 2023-11-28T17:11:02.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.317+0000 7f6b32a12640 1 --2- 172.21.15.116:0/1529424451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b18005180 0x7f6b1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:02.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.317+0000 7f6b31a10640 1 --2- 172.21.15.116:0/1529424451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b180047c0 0x7f6b18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:02.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.317+0000 7f6b32211640 1 --2- 172.21.15.116:0/1529424451 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b18003970 0x7f6b18003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:02.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.317+0000 7f6b32a12640 1 -- 172.21.15.116:0/1529424451 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b18003970 msgr2=0x7f6b18003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:02.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.317+0000 7f6b32a12640 1 --2- 172.21.15.116:0/1529424451 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b18003970 0x7f6b18003d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.317+0000 7f6b32a12640 1 -- 172.21.15.116:0/1529424451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b180047c0 msgr2=0x7f6b18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:02.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.317+0000 7f6b32a12640 1 --2- 172.21.15.116:0/1529424451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b180047c0 0x7f6b18004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.317+0000 7f6b32a12640 1 -- 172.21.15.116:0/1529424451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b18003520 con 0x7f6b18005180 2023-11-28T17:11:02.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.318+0000 7f6b32a12640 1 -- 172.21.15.116:0/1529424451 reap_dead start 2023-11-28T17:11:02.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.318+0000 7f6b32a12640 1 --2- 172.21.15.116:0/1529424451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b18005180 0x7f6b1800b980 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c067fd0 tx=0x7f6b180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2b3872d662916f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:02.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.318+0000 7f6b3120f640 1 -- 172.21.15.116:0/1529424451 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24006d50 con 0x7f6b18005180 2023-11-28T17:11:02.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.319+0000 7f6b3120f640 1 -- 172.21.15.116:0/1529424451 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b24031450 con 0x7f6b18005180 2023-11-28T17:11:02.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.319+0000 7f6b3120f640 1 -- 172.21.15.116:0/1529424451 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24031800 con 0x7f6b18005180 2023-11-28T17:11:02.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.319+0000 7f6b34475640 1 -- 172.21.15.116:0/1529424451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b18005180 msgr2=0x7f6b1800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:02.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.319+0000 7f6b34475640 1 --2- 172.21.15.116:0/1529424451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b18005180 0x7f6b1800b980 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c067fd0 tx=0x7f6b180057f0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.319+0000 7f6b34475640 1 -- 172.21.15.116:0/1529424451 shutdown_connections 2023-11-28T17:11:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.319+0000 7f6b34475640 1 --2- 172.21.15.116:0/1529424451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b18005180 0x7f6b1800b980 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.319+0000 7f6b34475640 1 -- 172.21.15.116:0/1529424451 >> 172.21.15.116:0/1529424451 conn(0x7f6b2c06a4c0 msgr2=0x7f6b2c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.319+0000 7f6b34475640 1 -- 172.21.15.116:0/1529424451 shutdown_connections 2023-11-28T17:11:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.319+0000 7f6b34475640 1 -- 172.21.15.116:0/1529424451 wait complete. 2023-11-28T17:11:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.321+0000 7f6b34475640 1 Processor -- start 2023-11-28T17:11:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.321+0000 7f6b34475640 1 -- start start 2023-11-28T17:11:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.322+0000 7f6b34475640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b18005180 0x7f6b2c0cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.322+0000 7f6b34475640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b2c12f210 0x7f6b2c0c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.322+0000 7f6b34475640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b2c15a7d0 0x7f6b2c13e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:02.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.322+0000 7f6b34475640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c0c9ee0 con 0x7f6b2c15a7d0 2023-11-28T17:11:02.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.322+0000 7f6b34475640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c0ca140 con 0x7f6b2c12f210 2023-11-28T17:11:02.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.322+0000 7f6b34475640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c0ca420 con 0x7f6b18005180 2023-11-28T17:11:02.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.322+0000 7f6b32a12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b2c15a7d0 0x7f6b2c13e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:02.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.322+0000 7f6b32a12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b2c15a7d0 0x7f6b2c13e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54626/0 (socket says 172.21.15.116:54626) 2023-11-28T17:11:02.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.322+0000 7f6b32a12640 1 -- 172.21.15.116:0/1630279964 learned_addr learned my addr 172.21.15.116:0/1630279964 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:02.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.323+0000 7f6b32211640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b18005180 0x7f6b2c0cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:02.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.323+0000 7f6b31a10640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b2c12f210 0x7f6b2c0c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:02.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.323+0000 7f6b32a12640 1 -- 172.21.15.116:0/1630279964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b18005180 msgr2=0x7f6b2c0cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:02.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.323+0000 7f6b32a12640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b18005180 0x7f6b2c0cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.323+0000 7f6b32a12640 1 -- 172.21.15.116:0/1630279964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b2c12f210 msgr2=0x7f6b2c0c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:02.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.323+0000 7f6b32a12640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b2c12f210 0x7f6b2c0c99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.323+0000 7f6b32a12640 1 -- 172.21.15.116:0/1630279964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c0ca700 con 0x7f6b2c15a7d0 2023-11-28T17:11:02.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.323+0000 7f6b31a10640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b2c12f210 0x7f6b2c0c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:02.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.323+0000 7f6b32a12640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b2c15a7d0 0x7f6b2c13e450 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6b2402bf10 tx=0x7f6b24003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.323+0000 7f6b22ffd640 1 -- 172.21.15.116:0/1630279964 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24031bf0 con 0x7f6b2c15a7d0 2023-11-28T17:11:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.324+0000 7f6b22ffd640 1 -- 172.21.15.116:0/1630279964 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2403e070 con 0x7f6b2c15a7d0 2023-11-28T17:11:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.324+0000 7f6b22ffd640 1 -- 172.21.15.116:0/1630279964 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24043480 con 0x7f6b2c15a7d0 2023-11-28T17:11:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.324+0000 7f6b34475640 1 -- 172.21.15.116:0/1630279964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2c0ca9e0 con 0x7f6b2c15a7d0 2023-11-28T17:11:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.324+0000 7f6b34475640 1 -- 172.21.15.116:0/1630279964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2c12f690 con 0x7f6b2c15a7d0 2023-11-28T17:11:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.325+0000 7f6b34475640 1 -- 172.21.15.116:0/1630279964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b2c0caf00 con 0x7f6b2c15a7d0 2023-11-28T17:11:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.326+0000 7f6b22ffd640 1 -- 172.21.15.116:0/1630279964 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6b24040030 con 0x7f6b2c15a7d0 2023-11-28T17:11:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.327+0000 7f6b22ffd640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6b1006b2d0 0x7f6b1006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.327+0000 7f6b22ffd640 1 -- 172.21.15.116:0/1630279964 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6b240aacc0 con 0x7f6b2c15a7d0 2023-11-28T17:11:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.327+0000 7f6b32211640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6b1006b2d0 0x7f6b1006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:02.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.328+0000 7f6b32211640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6b1006b2d0 0x7f6b1006d790 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6b280018e0 tx=0x7f6b28009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:02.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.332+0000 7f6b22ffd640 1 -- 172.21.15.116:0/1630279964 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b2c0caf00 con 0x7f6b2c15a7d0 2023-11-28T17:11:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.544+0000 7f6b34475640 1 -- 172.21.15.116:0/1630279964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6b2c0caf00 con 0x7f6b2c15a7d0 2023-11-28T17:11:02.546 INFO:tasks.ceph.mon.a.smithi116.stderr:2023-11-28T17:11:02.545+0000 7f2312418640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-28T17:11:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.549+0000 7f6b22ffd640 1 -- 172.21.15.116:0/1630279964 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6b2c0caf00 con 0x7f6b2c15a7d0 2023-11-28T17:11:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-28T17:11:02.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.553+0000 7f6b34475640 1 -- 172.21.15.116:0/1630279964 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6b1006b2d0 msgr2=0x7f6b1006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:02.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.553+0000 7f6b34475640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6b1006b2d0 0x7f6b1006d790 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6b280018e0 tx=0x7f6b28009040 comp rx=0 tx=0).stop 2023-11-28T17:11:02.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.553+0000 7f6b34475640 1 -- 172.21.15.116:0/1630279964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b2c15a7d0 msgr2=0x7f6b2c13e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:02.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.553+0000 7f6b34475640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b2c15a7d0 0x7f6b2c13e450 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6b2402bf10 tx=0x7f6b24003a70 comp rx=0 tx=0).stop 2023-11-28T17:11:02.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.554+0000 7f6b34475640 1 -- 172.21.15.116:0/1630279964 shutdown_connections 2023-11-28T17:11:02.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.554+0000 7f6b34475640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6b1006b2d0 0x7f6b1006d790 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.554+0000 7f6b34475640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b2c15a7d0 0x7f6b2c13e450 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.554+0000 7f6b34475640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b2c12f210 0x7f6b2c0c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.554+0000 7f6b34475640 1 --2- 172.21.15.116:0/1630279964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b18005180 0x7f6b2c0cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.554+0000 7f6b34475640 1 -- 172.21.15.116:0/1630279964 >> 172.21.15.116:0/1630279964 conn(0x7f6b2c06a4c0 msgr2=0x7f6b2c15eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:02.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.554+0000 7f6b34475640 1 -- 172.21.15.116:0/1630279964 shutdown_connections 2023-11-28T17:11:02.556 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.554+0000 7f6b34475640 1 -- 172.21.15.116:0/1630279964 wait complete. 2023-11-28T17:11:02.563 INFO:tasks.ceph.mds.c.smithi116.stderr: -1927> 2023-11-28T17:10:21.239+0000 7f56ff6d0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T17:11:02.575 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-28T17:11:02.627 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2023-11-28T17:11:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.766+0000 7f9cdace0640 1 Processor -- start 2023-11-28T17:11:02.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.766+0000 7f9cdace0640 1 -- start start 2023-11-28T17:11:02.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.768+0000 7f9cdace0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd413cee0 0x7f9cd413d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:02.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.768+0000 7f9cdace0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9cd4072430 0x7f9cd413fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:02.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.768+0000 7f9cdace0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd413d2c0 con 0x7f9cd413f640 2023-11-28T17:11:02.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.768+0000 7f9cdace0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd41400e0 con 0x7f9cd4072430 2023-11-28T17:11:02.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.768+0000 7f9cdace0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd41403c0 con 0x7f9cd413cee0 2023-11-28T17:11:02.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.768+0000 7f9cd927d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9cd4072430 0x7f9cd413fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:02.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.769+0000 7f9cd927d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9cd4072430 0x7f9cd413fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54664/0 (socket says 172.21.15.116:54664) 2023-11-28T17:11:02.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.769+0000 7f9cd927d640 1 -- 172.21.15.116:0/36565722 learned_addr learned my addr 172.21.15.116:0/36565722 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:02.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.769+0000 7f9cd8a7c640 1 --2- 172.21.15.116:0/36565722 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd413cee0 0x7f9cd413d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:02.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.769+0000 7f9cd927d640 1 -- 172.21.15.116:0/36565722 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd413cee0 msgr2=0x7f9cd413d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:02.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.769+0000 7f9cd927d640 1 --2- 172.21.15.116:0/36565722 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd413cee0 0x7f9cd413d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.769+0000 7f9cd927d640 1 -- 172.21.15.116:0/36565722 >> v1:172.21.15.156:6789/0 conn(0x7f9cd413f640 legacy=0x7f9cd4071d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:02.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.769+0000 7f9cd927d640 1 -- 172.21.15.116:0/36565722 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd40cdb50 con 0x7f9cd4072430 2023-11-28T17:11:02.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.770+0000 7f9cd927d640 1 --2- 172.21.15.116:0/36565722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd4072430 0x7f9cd413fa80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009870 tx=0x7f9ccc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1f25d25b5d0c77 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:02.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.770+0000 7f9cd37fe640 1 -- 172.21.15.116:0/36565722 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc0317c0 con 0x7f9cd4072430 2023-11-28T17:11:02.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.770+0000 7f9cd37fe640 1 -- 172.21.15.116:0/36565722 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ccc031920 con 0x7f9cd4072430 2023-11-28T17:11:02.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.770+0000 7f9cd37fe640 1 -- 172.21.15.116:0/36565722 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc031cb0 con 0x7f9cd4072430 2023-11-28T17:11:02.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.771+0000 7f9cdace0640 1 -- 172.21.15.116:0/36565722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd4072430 msgr2=0x7f9cd413fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:02.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.771+0000 7f9cdace0640 1 --2- 172.21.15.116:0/36565722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd4072430 0x7f9cd413fa80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009870 tx=0x7f9ccc02b860 comp rx=0 tx=0).stop 2023-11-28T17:11:02.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.771+0000 7f9cdace0640 1 -- 172.21.15.116:0/36565722 shutdown_connections 2023-11-28T17:11:02.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.771+0000 7f9cdace0640 1 --2- 172.21.15.116:0/36565722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd4072430 0x7f9cd413fa80 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.771+0000 7f9cdace0640 1 --2- 172.21.15.116:0/36565722 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd413cee0 0x7f9cd413d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.771+0000 7f9cdace0640 1 -- 172.21.15.116:0/36565722 >> 172.21.15.116:0/36565722 conn(0x7f9cd40b0cf0 msgr2=0x7f9cd40b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:02.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.771+0000 7f9cdace0640 1 -- 172.21.15.116:0/36565722 shutdown_connections 2023-11-28T17:11:02.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.771+0000 7f9cdace0640 1 -- 172.21.15.116:0/36565722 wait complete. 2023-11-28T17:11:02.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.773+0000 7f9cdace0640 1 Processor -- start 2023-11-28T17:11:02.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.773+0000 7f9cdace0640 1 -- start start 2023-11-28T17:11:02.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.774+0000 7f9cdace0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cd4072430 0x7f9cd41380d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:02.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.774+0000 7f9cdace0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd413cee0 0x7f9cd4138610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:02.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.774+0000 7f9cdace0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd413f640 0x7f9cd413e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:02.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.774+0000 7f9cdace0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd415cf10 con 0x7f9cd413f640 2023-11-28T17:11:02.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.774+0000 7f9cdace0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd415c290 con 0x7f9cd4072430 2023-11-28T17:11:02.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.774+0000 7f9cdace0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd415c570 con 0x7f9cd413cee0 2023-11-28T17:11:02.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.774+0000 7f9cd927d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd413f640 0x7f9cd413e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.774+0000 7f9cd927d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd413f640 0x7f9cd413e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54678/0 (socket says 172.21.15.116:54678) 2023-11-28T17:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.774+0000 7f9cd927d640 1 -- 172.21.15.116:0/1378455522 learned_addr learned my addr 172.21.15.116:0/1378455522 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.775+0000 7f9cd3fff640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd413cee0 0x7f9cd4138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.775+0000 7f9cd927d640 1 -- 172.21.15.116:0/1378455522 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd413cee0 msgr2=0x7f9cd4138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.775+0000 7f9cd927d640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd413cee0 0x7f9cd4138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.775+0000 7f9cd8a7c640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cd4072430 0x7f9cd41380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.775+0000 7f9cd927d640 1 -- 172.21.15.116:0/1378455522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cd4072430 msgr2=0x7f9cd41380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.775+0000 7f9cd927d640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cd4072430 0x7f9cd41380d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.775+0000 7f9cd927d640 1 -- 172.21.15.116:0/1378455522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc0094d0 con 0x7f9cd413f640 2023-11-28T17:11:02.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.775+0000 7f9cd3fff640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd413cee0 0x7f9cd4138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:02.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.775+0000 7f9cd927d640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd413f640 0x7f9cd413e530 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc0090b0 tx=0x7f9ccc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:02.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.775+0000 7f9cd17fa640 1 -- 172.21.15.116:0/1378455522 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc032860 con 0x7f9cd413f640 2023-11-28T17:11:02.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.775+0000 7f9cd17fa640 1 -- 172.21.15.116:0/1378455522 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ccc0329e0 con 0x7f9cd413f640 2023-11-28T17:11:02.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.775+0000 7f9cd17fa640 1 -- 172.21.15.116:0/1378455522 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc032d90 con 0x7f9cd413f640 2023-11-28T17:11:02.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.775+0000 7f9cdace0640 1 -- 172.21.15.116:0/1378455522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd415c850 con 0x7f9cd413f640 2023-11-28T17:11:02.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.776+0000 7f9cdace0640 1 -- 172.21.15.116:0/1378455522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd415d330 con 0x7f9cd413f640 2023-11-28T17:11:02.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.776+0000 7f9cdace0640 1 -- 172.21.15.116:0/1378455522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd406c9d0 con 0x7f9cd413f640 2023-11-28T17:11:02.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.778+0000 7f9cd17fa640 1 -- 172.21.15.116:0/1378455522 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f9ccc04e450 con 0x7f9cd413f640 2023-11-28T17:11:02.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.778+0000 7f9cd17fa640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9cb406b2a0 0x7f9cb406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:02.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.778+0000 7f9cd17fa640 1 -- 172.21.15.116:0/1378455522 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f9ccc0abea0 con 0x7f9cd413f640 2023-11-28T17:11:02.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.779+0000 7f9cd8a7c640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9cb406b2a0 0x7f9cb406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:02.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.779+0000 7f9cd8a7c640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9cb406b2a0 0x7f9cb406d760 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9cd41f6a70 tx=0x7f9cc400b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:02.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:02.783+0000 7f9cd17fa640 1 -- 172.21.15.116:0/1378455522 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ccc076c30 con 0x7f9cd413f640 2023-11-28T17:11:03.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.012+0000 7f9cdace0640 1 -- 172.21.15.116:0/1378455522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9cd415d050 con 0x7f9cd413f640 2023-11-28T17:11:03.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.553+0000 7f9cd17fa640 1 -- 172.21.15.116:0/1378455522 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9cd415d050 con 0x7f9cd413f640 2023-11-28T17:11:03.559 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.558+0000 7f9cdace0640 1 -- 172.21.15.116:0/1378455522 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9cb406b2a0 msgr2=0x7f9cb406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:03.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.558+0000 7f9cdace0640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9cb406b2a0 0x7f9cb406d760 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9cd41f6a70 tx=0x7f9cc400b040 comp rx=0 tx=0).stop 2023-11-28T17:11:03.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.558+0000 7f9cdace0640 1 -- 172.21.15.116:0/1378455522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd413f640 msgr2=0x7f9cd413e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:03.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.558+0000 7f9cdace0640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd413f640 0x7f9cd413e530 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc0090b0 tx=0x7f9ccc0046c0 comp rx=0 tx=0).stop 2023-11-28T17:11:03.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.558+0000 7f9cdace0640 1 -- 172.21.15.116:0/1378455522 shutdown_connections 2023-11-28T17:11:03.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.558+0000 7f9cdace0640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9cb406b2a0 0x7f9cb406d760 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:03.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.558+0000 7f9cdace0640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cd413f640 0x7f9cd413e530 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:03.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.558+0000 7f9cdace0640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cd413cee0 0x7f9cd4138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:03.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.558+0000 7f9cdace0640 1 --2- 172.21.15.116:0/1378455522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cd4072430 0x7f9cd41380d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:03.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.558+0000 7f9cdace0640 1 -- 172.21.15.116:0/1378455522 >> 172.21.15.116:0/1378455522 conn(0x7f9cd40b0cf0 msgr2=0x7f9cd41599c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:03.560 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.558+0000 7f9cdace0640 1 -- 172.21.15.116:0/1378455522 shutdown_connections 2023-11-28T17:11:03.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.559+0000 7f9cdace0640 1 -- 172.21.15.116:0/1378455522 wait complete. 2023-11-28T17:11:03.582 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-28T17:11:03.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.791+0000 7f7a5359e640 1 Processor -- start 2023-11-28T17:11:03.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.791+0000 7f7a5359e640 1 -- start start 2023-11-28T17:11:03.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.792+0000 7f7a5359e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7a5415d3a0 0x7f7a540cb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:03.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.792+0000 7f7a5359e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a540d1500 0x7f7a5406bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:03.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.792+0000 7f7a5359e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5406c4d0 con 0x7f7a540ce2b0 2023-11-28T17:11:03.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.792+0000 7f7a5359e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a540cf9e0 con 0x7f7a540d1500 2023-11-28T17:11:03.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.792+0000 7f7a5359e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a5406c9f0 con 0x7f7a5415d3a0 2023-11-28T17:11:03.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.793+0000 7f7a52d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a540d1500 0x7f7a5406bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:03.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.793+0000 7f7a52d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a540d1500 0x7f7a5406bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54688/0 (socket says 172.21.15.116:54688) 2023-11-28T17:11:03.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.793+0000 7f7a52d9d640 1 -- 172.21.15.116:0/3328168430 learned_addr learned my addr 172.21.15.116:0/3328168430 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:03.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.793+0000 7f7a5259c640 1 --2- 172.21.15.116:0/3328168430 >> v2:172.21.15.156:3300/0 conn(0x7f7a5415d3a0 0x7f7a540cb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:03.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.793+0000 7f7a52d9d640 1 -- 172.21.15.116:0/3328168430 >> v2:172.21.15.156:3300/0 conn(0x7f7a5415d3a0 msgr2=0x7f7a540cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:03.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.793+0000 7f7a52d9d640 1 --2- 172.21.15.116:0/3328168430 >> v2:172.21.15.156:3300/0 conn(0x7f7a5415d3a0 0x7f7a540cb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:03.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.793+0000 7f7a52d9d640 1 -- 172.21.15.116:0/3328168430 >> v1:172.21.15.116:6789/0 conn(0x7f7a540ce2b0 legacy=0x7f7a540d0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:03.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.793+0000 7f7a52d9d640 1 -- 172.21.15.116:0/3328168430 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a540cf0f0 con 0x7f7a540d1500 2023-11-28T17:11:03.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.794+0000 7f7a52d9d640 1 --2- 172.21.15.116:0/3328168430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a540d1500 0x7f7a5406bf90 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7a5406e2b0 tx=0x7f7a4c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8402b33d8ab8757 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:03.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.794+0000 7f7a5159a640 1 -- 172.21.15.116:0/3328168430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c030860 con 0x7f7a540d1500 2023-11-28T17:11:03.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.794+0000 7f7a5159a640 1 -- 172.21.15.116:0/3328168430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4c0309c0 con 0x7f7a540d1500 2023-11-28T17:11:03.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.794+0000 7f7a5159a640 1 -- 172.21.15.116:0/3328168430 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c030d50 con 0x7f7a540d1500 2023-11-28T17:11:03.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.795+0000 7f7a5359e640 1 -- 172.21.15.116:0/3328168430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a540d1500 msgr2=0x7f7a5406bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:03.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.795+0000 7f7a5359e640 1 --2- 172.21.15.116:0/3328168430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a540d1500 0x7f7a5406bf90 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7a5406e2b0 tx=0x7f7a4c02b9a0 comp rx=0 tx=0).stop 2023-11-28T17:11:03.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.795+0000 7f7a5359e640 1 -- 172.21.15.116:0/3328168430 shutdown_connections 2023-11-28T17:11:03.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.795+0000 7f7a5359e640 1 --2- 172.21.15.116:0/3328168430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a540d1500 0x7f7a5406bf90 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:03.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.795+0000 7f7a5359e640 1 --2- 172.21.15.116:0/3328168430 >> v2:172.21.15.156:3300/0 conn(0x7f7a5415d3a0 0x7f7a540cb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:03.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.795+0000 7f7a5359e640 1 -- 172.21.15.116:0/3328168430 >> 172.21.15.116:0/3328168430 conn(0x7f7a54099a00 msgr2=0x7f7a54099e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:03.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.795+0000 7f7a5359e640 1 -- 172.21.15.116:0/3328168430 shutdown_connections 2023-11-28T17:11:03.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.795+0000 7f7a5359e640 1 -- 172.21.15.116:0/3328168430 wait complete. 2023-11-28T17:11:03.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.797+0000 7f7a5359e640 1 Processor -- start 2023-11-28T17:11:03.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.797+0000 7f7a5359e640 1 -- start start 2023-11-28T17:11:03.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.798+0000 7f7a5359e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a540ce2b0 0x7f7a541509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:03.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.798+0000 7f7a5359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a540d1500 0x7f7a54150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:03.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.798+0000 7f7a5359e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a5415d3a0 0x7f7a5415ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:03.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.798+0000 7f7a5359e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5415db10 con 0x7f7a540d1500 2023-11-28T17:11:03.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.798+0000 7f7a5359e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a54150ee0 con 0x7f7a540ce2b0 2023-11-28T17:11:03.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.798+0000 7f7a5359e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5415ddd0 con 0x7f7a5415d3a0 2023-11-28T17:11:03.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.798+0000 7f7a51d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a540d1500 0x7f7a54150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:03.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.798+0000 7f7a51d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a540d1500 0x7f7a54150110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54690/0 (socket says 172.21.15.116:54690) 2023-11-28T17:11:03.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.798+0000 7f7a51d9b640 1 -- 172.21.15.116:0/60047989 learned_addr learned my addr 172.21.15.116:0/60047989 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:03.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.799+0000 7f7a52d9d640 1 --2- 172.21.15.116:0/60047989 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a5415d3a0 0x7f7a5415ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:03.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.799+0000 7f7a51d9b640 1 -- 172.21.15.116:0/60047989 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a5415d3a0 msgr2=0x7f7a5415ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:03.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.799+0000 7f7a5259c640 1 --2- 172.21.15.116:0/60047989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a540ce2b0 0x7f7a541509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:03.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.799+0000 7f7a51d9b640 1 --2- 172.21.15.116:0/60047989 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a5415d3a0 0x7f7a5415ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:03.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.799+0000 7f7a51d9b640 1 -- 172.21.15.116:0/60047989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a540ce2b0 msgr2=0x7f7a541509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:03.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.799+0000 7f7a51d9b640 1 --2- 172.21.15.116:0/60047989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a540ce2b0 0x7f7a541509a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:03.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.799+0000 7f7a51d9b640 1 -- 172.21.15.116:0/60047989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c0094d0 con 0x7f7a540d1500 2023-11-28T17:11:03.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.799+0000 7f7a51d9b640 1 --2- 172.21.15.116:0/60047989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a540d1500 0x7f7a54150110 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c00c400 tx=0x7f7a3c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:03.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.799+0000 7f7a4affd640 1 -- 172.21.15.116:0/60047989 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c0052d0 con 0x7f7a540d1500 2023-11-28T17:11:03.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.799+0000 7f7a4affd640 1 -- 172.21.15.116:0/60047989 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3c005430 con 0x7f7a540d1500 2023-11-28T17:11:03.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.799+0000 7f7a4affd640 1 -- 172.21.15.116:0/60047989 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c003830 con 0x7f7a540d1500 2023-11-28T17:11:03.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.799+0000 7f7a5359e640 1 -- 172.21.15.116:0/60047989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5415e0b0 con 0x7f7a540d1500 2023-11-28T17:11:03.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.799+0000 7f7a5359e640 1 -- 172.21.15.116:0/60047989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a54150650 con 0x7f7a540d1500 2023-11-28T17:11:03.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.801+0000 7f7a5359e640 1 -- 172.21.15.116:0/60047989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a540cf370 con 0x7f7a540d1500 2023-11-28T17:11:03.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.802+0000 7f7a4affd640 1 -- 172.21.15.116:0/60047989 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f7a3c003990 con 0x7f7a540d1500 2023-11-28T17:11:03.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.803+0000 7f7a4affd640 1 --2- 172.21.15.116:0/60047989 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7a3406b2a0 0x7f7a3406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:03.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.803+0000 7f7a4affd640 1 -- 172.21.15.116:0/60047989 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f7a3c0890d0 con 0x7f7a540d1500 2023-11-28T17:11:03.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.803+0000 7f7a5259c640 1 --2- 172.21.15.116:0/60047989 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7a3406b2a0 0x7f7a3406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:03.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.804+0000 7f7a5259c640 1 --2- 172.21.15.116:0/60047989 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7a3406b2a0 0x7f7a3406d760 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009f30 tx=0x7f7a4400b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:03.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.807+0000 7f7a4affd640 1 -- 172.21.15.116:0/60047989 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a540cf370 con 0x7f7a540d1500 2023-11-28T17:11:03.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:03.996+0000 7f7a5359e640 1 -- 172.21.15.116:0/60047989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7a5406c4d0 con 0x7f7a540d1500 2023-11-28T17:11:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.566+0000 7f7a4affd640 1 -- 172.21.15.116:0/60047989 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v38) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7a5406c4d0 con 0x7f7a540d1500 2023-11-28T17:11:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' removed 2023-11-28T17:11:04.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.570+0000 7f7a5359e640 1 -- 172.21.15.116:0/60047989 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7a3406b2a0 msgr2=0x7f7a3406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:04.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.570+0000 7f7a5359e640 1 --2- 172.21.15.116:0/60047989 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7a3406b2a0 0x7f7a3406d760 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009f30 tx=0x7f7a4400b040 comp rx=0 tx=0).stop 2023-11-28T17:11:04.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.570+0000 7f7a5359e640 1 -- 172.21.15.116:0/60047989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a540d1500 msgr2=0x7f7a54150110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:04.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.570+0000 7f7a5359e640 1 --2- 172.21.15.116:0/60047989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a540d1500 0x7f7a54150110 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c00c400 tx=0x7f7a3c00c8c0 comp rx=0 tx=0).stop 2023-11-28T17:11:04.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.570+0000 7f7a5359e640 1 -- 172.21.15.116:0/60047989 shutdown_connections 2023-11-28T17:11:04.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.570+0000 7f7a5359e640 1 --2- 172.21.15.116:0/60047989 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7a3406b2a0 0x7f7a3406d760 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:04.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.570+0000 7f7a5359e640 1 --2- 172.21.15.116:0/60047989 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a5415d3a0 0x7f7a5415ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:04.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.570+0000 7f7a5359e640 1 --2- 172.21.15.116:0/60047989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a540d1500 0x7f7a54150110 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:04.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.570+0000 7f7a5359e640 1 --2- 172.21.15.116:0/60047989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a540ce2b0 0x7f7a541509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:04.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.570+0000 7f7a5359e640 1 -- 172.21.15.116:0/60047989 >> 172.21.15.116:0/60047989 conn(0x7f7a54099a00 msgr2=0x7f7a540cc920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:04.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.570+0000 7f7a5359e640 1 -- 172.21.15.116:0/60047989 shutdown_connections 2023-11-28T17:11:04.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.571+0000 7f7a5359e640 1 -- 172.21.15.116:0/60047989 wait complete. 2023-11-28T17:11:04.600 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-28T17:11:04.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.802+0000 7fb53e173640 1 Processor -- start 2023-11-28T17:11:04.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.803+0000 7fb53e173640 1 -- start start 2023-11-28T17:11:04.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.804+0000 7fb53e173640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb53815a240 0x7fb5380ced70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:04.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.804+0000 7fb53e173640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb538071420 con 0x7fb5380ce160 2023-11-28T17:11:04.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.804+0000 7fb53e173640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb538071cc0 con 0x7fb538072150 2023-11-28T17:11:04.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.804+0000 7fb53e173640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb53815a620 con 0x7fb53815a240 2023-11-28T17:11:04.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.804+0000 7fb537fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb5380ce160 0x7fb53806efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54250/0 (socket says 172.21.15.116:54250) 2023-11-28T17:11:04.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.804+0000 7fb537fff640 1 -- 172.21.15.116:0/3359511543 learned_addr learned my addr 172.21.15.116:0/3359511543 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:04.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 -- 172.21.15.116:0/3359511543 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059990141 0 0) 0x7fb538071420 con 0x7fb5380ce160 2023-11-28T17:11:04.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 -- 172.21.15.116:0/3359511543 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb524003520 con 0x7fb5380ce160 2023-11-28T17:11:04.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5377fe640 1 --2- 172.21.15.116:0/3359511543 >> v2:172.21.15.156:3300/0 conn(0x7fb53815a240 0x7fb5380ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:04.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 -- 172.21.15.116:0/3359511543 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1804721294 0 0) 0x7fb524003520 con 0x7fb5380ce160 2023-11-28T17:11:04.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 -- 172.21.15.116:0/3359511543 >> v2:172.21.15.156:3300/0 conn(0x7fb53815a240 msgr2=0x7fb5380ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:04.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 --2- 172.21.15.116:0/3359511543 >> v2:172.21.15.156:3300/0 conn(0x7fb53815a240 0x7fb5380ced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:04.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 -- 172.21.15.116:0/3359511543 >> v1:172.21.15.156:6789/0 conn(0x7fb538072150 legacy=0x7fb53806e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:04.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 -- 172.21.15.116:0/3359511543 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53806f6e0 con 0x7fb5380ce160 2023-11-28T17:11:04.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 -- 172.21.15.116:0/3359511543 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fb530001670 con 0x7fb5380ce160 2023-11-28T17:11:04.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 -- 172.21.15.116:0/3359511543 >> v1:172.21.15.116:6789/0 conn(0x7fb5380ce160 legacy=0x7fb53806efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:04.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 --2- 172.21.15.116:0/3359511543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5240039c0 0x7fb524003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:04.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 --2- 172.21.15.116:0/3359511543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5240047c0 0x7fb524004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:04.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 --2- 172.21.15.116:0/3359511543 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb524005380 0x7fb52400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:04.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 -- 172.21.15.116:0/3359511543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53815a620 con 0x7fb5240047c0 2023-11-28T17:11:04.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 -- 172.21.15.116:0/3359511543 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52400c240 con 0x7fb5240039c0 2023-11-28T17:11:04.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.805+0000 7fb5367fc640 1 -- 172.21.15.116:0/3359511543 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52400e610 con 0x7fb524005380 2023-11-28T17:11:04.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.806+0000 7fb536ffd640 1 --2- 172.21.15.116:0/3359511543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5240047c0 0x7fb524004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:04.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.806+0000 7fb5377fe640 1 --2- 172.21.15.116:0/3359511543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5240039c0 0x7fb524003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:04.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.806+0000 7fb536ffd640 1 -- 172.21.15.116:0/3359511543 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb524005380 msgr2=0x7fb52400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:04.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.806+0000 7fb536ffd640 1 --2- 172.21.15.116:0/3359511543 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb524005380 0x7fb52400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:04.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.806+0000 7fb536ffd640 1 -- 172.21.15.116:0/3359511543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5240039c0 msgr2=0x7fb524003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:04.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.806+0000 7fb536ffd640 1 --2- 172.21.15.116:0/3359511543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5240039c0 0x7fb524003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:04.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.806+0000 7fb536ffd640 1 -- 172.21.15.116:0/3359511543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb524003520 con 0x7fb5240047c0 2023-11-28T17:11:04.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.806+0000 7fb537fff640 1 -- 172.21.15.116:0/3359511543 reap_dead start 2023-11-28T17:11:04.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.807+0000 7fb536ffd640 1 --2- 172.21.15.116:0/3359511543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5240047c0 0x7fb524004c40 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb520003db0 tx=0x7fb5240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b5a89691ca7525 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:04.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.807+0000 7fb5367fc640 1 -- 172.21.15.116:0/3359511543 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520032600 con 0x7fb5240047c0 2023-11-28T17:11:04.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.808+0000 7fb5367fc640 1 -- 172.21.15.116:0/3359511543 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52002e050 con 0x7fb5240047c0 2023-11-28T17:11:04.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.808+0000 7fb5367fc640 1 -- 172.21.15.116:0/3359511543 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5200329e0 con 0x7fb5240047c0 2023-11-28T17:11:04.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.808+0000 7fb53e173640 1 -- 172.21.15.116:0/3359511543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5240047c0 msgr2=0x7fb524004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.808+0000 7fb53e173640 1 --2- 172.21.15.116:0/3359511543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5240047c0 0x7fb524004c40 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb520003db0 tx=0x7fb5240057f0 comp rx=0 tx=0).stop 2023-11-28T17:11:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.808+0000 7fb53e173640 1 -- 172.21.15.116:0/3359511543 shutdown_connections 2023-11-28T17:11:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.808+0000 7fb53e173640 1 --2- 172.21.15.116:0/3359511543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5240047c0 0x7fb524004c40 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.808+0000 7fb53e173640 1 -- 172.21.15.116:0/3359511543 >> 172.21.15.116:0/3359511543 conn(0x7fb5380b1d70 msgr2=0x7fb5380b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.808+0000 7fb53e173640 1 -- 172.21.15.116:0/3359511543 shutdown_connections 2023-11-28T17:11:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.808+0000 7fb53e173640 1 -- 172.21.15.116:0/3359511543 wait complete. 2023-11-28T17:11:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.810+0000 7fb53e173640 1 Processor -- start 2023-11-28T17:11:04.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.810+0000 7fb53e173640 1 -- start start 2023-11-28T17:11:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.811+0000 7fb53e173640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5240047c0 0x7fb53814f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.811+0000 7fb53e173640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb53815b490 0x7fb53815b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.811+0000 7fb53e173640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb538153fe0 0x7fb538154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.811+0000 7fb53e173640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53815be50 con 0x7fb53815b490 2023-11-28T17:11:04.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.811+0000 7fb53e173640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb538154d50 con 0x7fb5240047c0 2023-11-28T17:11:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.811+0000 7fb53e173640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53814f850 con 0x7fb538153fe0 2023-11-28T17:11:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.811+0000 7fb536ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb53815b490 0x7fb53815b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.811+0000 7fb536ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb53815b490 0x7fb53815b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54698/0 (socket says 172.21.15.116:54698) 2023-11-28T17:11:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.812+0000 7fb536ffd640 1 -- 172.21.15.116:0/2910350548 learned_addr learned my addr 172.21.15.116:0/2910350548 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.812+0000 7fb5377fe640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5240047c0 0x7fb53814f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.812+0000 7fb537fff640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb538153fe0 0x7fb538154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.812+0000 7fb536ffd640 1 -- 172.21.15.116:0/2910350548 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb538153fe0 msgr2=0x7fb538154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.812+0000 7fb536ffd640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb538153fe0 0x7fb538154460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.812+0000 7fb536ffd640 1 -- 172.21.15.116:0/2910350548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5240047c0 msgr2=0x7fb53814f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:04.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.812+0000 7fb536ffd640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5240047c0 0x7fb53814f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.812+0000 7fb536ffd640 1 -- 172.21.15.116:0/2910350548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53814fb30 con 0x7fb53815b490 2023-11-28T17:11:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.812+0000 7fb5377fe640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5240047c0 0x7fb53814f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.812+0000 7fb536ffd640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb53815b490 0x7fb53815b910 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb520035040 tx=0x7fb5200036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.812+0000 7fb52ffff640 1 -- 172.21.15.116:0/2910350548 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52003c410 con 0x7fb53815b490 2023-11-28T17:11:04.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.813+0000 7fb52ffff640 1 -- 172.21.15.116:0/2910350548 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52003c570 con 0x7fb53815b490 2023-11-28T17:11:04.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.813+0000 7fb52ffff640 1 -- 172.21.15.116:0/2910350548 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52003c900 con 0x7fb53815b490 2023-11-28T17:11:04.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.813+0000 7fb53e173640 1 -- 172.21.15.116:0/2910350548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53814fe10 con 0x7fb53815b490 2023-11-28T17:11:04.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.813+0000 7fb53e173640 1 -- 172.21.15.116:0/2910350548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5381549a0 con 0x7fb53815b490 2023-11-28T17:11:04.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.814+0000 7fb53e173640 1 -- 172.21.15.116:0/2910350548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb538069360 con 0x7fb53815b490 2023-11-28T17:11:04.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.815+0000 7fb52ffff640 1 -- 172.21.15.116:0/2910350548 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb520039050 con 0x7fb53815b490 2023-11-28T17:11:04.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.816+0000 7fb52ffff640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb51806b2a0 0x7fb51806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:04.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.816+0000 7fb52ffff640 1 -- 172.21.15.116:0/2910350548 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5473+0+0 (secure 0 0 0) 0x7fb5200aaf00 con 0x7fb53815b490 2023-11-28T17:11:04.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.816+0000 7fb5377fe640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb51806b2a0 0x7fb51806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:04.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.817+0000 7fb5377fe640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb51806b2a0 0x7fb51806d760 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb5381f1c10 tx=0x7fb52800b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:04.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:04.820+0000 7fb52ffff640 1 -- 172.21.15.116:0/2910350548 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb520075ef0 con 0x7fb53815b490 2023-11-28T17:11:05.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.009+0000 7fb53e173640 1 -- 172.21.15.116:0/2910350548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb53815d590 con 0x7fb53815b490 2023-11-28T17:11:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.576+0000 7fb52ffff640 1 -- 172.21.15.116:0/2910350548 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v39) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb52004b020 con 0x7fb53815b490 2023-11-28T17:11:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' removed 2023-11-28T17:11:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.580+0000 7fb53e173640 1 -- 172.21.15.116:0/2910350548 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb51806b2a0 msgr2=0x7fb51806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.580+0000 7fb53e173640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb51806b2a0 0x7fb51806d760 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb5381f1c10 tx=0x7fb52800b040 comp rx=0 tx=0).stop 2023-11-28T17:11:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.580+0000 7fb53e173640 1 -- 172.21.15.116:0/2910350548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb53815b490 msgr2=0x7fb53815b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.580+0000 7fb53e173640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb53815b490 0x7fb53815b910 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb520035040 tx=0x7fb5200036f0 comp rx=0 tx=0).stop 2023-11-28T17:11:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.580+0000 7fb53e173640 1 -- 172.21.15.116:0/2910350548 shutdown_connections 2023-11-28T17:11:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.580+0000 7fb53e173640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb51806b2a0 0x7fb51806d760 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.580+0000 7fb53e173640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb538153fe0 0x7fb538154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.580+0000 7fb53e173640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb53815b490 0x7fb53815b910 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.580+0000 7fb53e173640 1 --2- 172.21.15.116:0/2910350548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb5240047c0 0x7fb53814f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.580+0000 7fb53e173640 1 -- 172.21.15.116:0/2910350548 >> 172.21.15.116:0/2910350548 conn(0x7fb5380b1d70 msgr2=0x7fb5380b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.581+0000 7fb53e173640 1 -- 172.21.15.116:0/2910350548 shutdown_connections 2023-11-28T17:11:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.581+0000 7fb53e173640 1 -- 172.21.15.116:0/2910350548 wait complete. 2023-11-28T17:11:05.604 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds ms_async_reap_threshold 2023-11-28T17:11:05.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.805+0000 7fe6e646e640 1 Processor -- start 2023-11-28T17:11:05.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.805+0000 7fe6e646e640 1 -- start start 2023-11-28T17:11:05.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.807+0000 7fe6e646e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe6e00d0700 0x7fe6e00d1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:05.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.807+0000 7fe6e646e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e00d0da0 con 0x7fe6e0160b80 2023-11-28T17:11:05.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.807+0000 7fe6e646e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e00d1080 con 0x7fe6e00b4010 2023-11-28T17:11:05.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.807+0000 7fe6e646e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e0160f60 con 0x7fe6e00d0700 2023-11-28T17:11:05.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.807+0000 7fe6e546c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe6e0160b80 0x7fe6e00d1360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54256/0 (socket says 172.21.15.116:54256) 2023-11-28T17:11:05.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.807+0000 7fe6e546c640 1 -- 172.21.15.116:0/2706752232 learned_addr learned my addr 172.21.15.116:0/2706752232 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:05.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 -- 172.21.15.116:0/2706752232 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575144333 0 0) 0x7fe6e00d0da0 con 0x7fe6e0160b80 2023-11-28T17:11:05.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 -- 172.21.15.116:0/2706752232 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6cc003520 con 0x7fe6e0160b80 2023-11-28T17:11:05.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6e4c6b640 1 --2- 172.21.15.116:0/2706752232 >> v2:172.21.15.156:3300/0 conn(0x7fe6e00d0700 0x7fe6e00d1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:05.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 -- 172.21.15.116:0/2706752232 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 83449944 0 0) 0x7fe6cc003520 con 0x7fe6e0160b80 2023-11-28T17:11:05.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 -- 172.21.15.116:0/2706752232 >> v2:172.21.15.156:3300/0 conn(0x7fe6e00d0700 msgr2=0x7fe6e00d1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:05.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 --2- 172.21.15.116:0/2706752232 >> v2:172.21.15.156:3300/0 conn(0x7fe6e00d0700 0x7fe6e00d1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:05.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 -- 172.21.15.116:0/2706752232 >> v1:172.21.15.156:6789/0 conn(0x7fe6e00b4010 legacy=0x7fe6e00c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:05.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 -- 172.21.15.116:0/2706752232 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e00ce780 con 0x7fe6e0160b80 2023-11-28T17:11:05.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 -- 172.21.15.116:0/2706752232 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fe6dc001670 con 0x7fe6e0160b80 2023-11-28T17:11:05.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 -- 172.21.15.116:0/2706752232 >> v1:172.21.15.116:6789/0 conn(0x7fe6e0160b80 legacy=0x7fe6e00d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:05.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 --2- 172.21.15.116:0/2706752232 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6cc003970 0x7fe6cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:05.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 --2- 172.21.15.116:0/2706752232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc0047c0 0x7fe6cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:05.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 --2- 172.21.15.116:0/2706752232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6cc005180 0x7fe6cc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:05.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 -- 172.21.15.116:0/2706752232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc00c0a0 con 0x7fe6cc0047c0 2023-11-28T17:11:05.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 -- 172.21.15.116:0/2706752232 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc00c210 con 0x7fe6cc005180 2023-11-28T17:11:05.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6d7fff640 1 -- 172.21.15.116:0/2706752232 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc00c3d0 con 0x7fe6cc003970 2023-11-28T17:11:05.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.808+0000 7fe6e4c6b640 1 --2- 172.21.15.116:0/2706752232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc0047c0 0x7fe6cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:05.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.809+0000 7fe6e4c6b640 1 -- 172.21.15.116:0/2706752232 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6cc003970 msgr2=0x7fe6cc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:05.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.809+0000 7fe6e4c6b640 1 --2- 172.21.15.116:0/2706752232 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6cc003970 0x7fe6cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.809+0000 7fe6e4c6b640 1 -- 172.21.15.116:0/2706752232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6cc005180 msgr2=0x7fe6cc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.809+0000 7fe6e4c6b640 1 --2- 172.21.15.116:0/2706752232 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6cc005180 0x7fe6cc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.809+0000 7fe6e4c6b640 1 -- 172.21.15.116:0/2706752232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc003520 con 0x7fe6cc0047c0 2023-11-28T17:11:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.809+0000 7fe6e5c6d640 1 -- 172.21.15.116:0/2706752232 reap_dead start 2023-11-28T17:11:05.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.810+0000 7fe6e4c6b640 1 --2- 172.21.15.116:0/2706752232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc0047c0 0x7fe6cc004c40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000ba30 tx=0x7fe6cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3aa9945bbc2ec50 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:05.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.810+0000 7fe6d7fff640 1 -- 172.21.15.116:0/2706752232 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0032680 con 0x7fe6cc0047c0 2023-11-28T17:11:05.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.810+0000 7fe6d7fff640 1 -- 172.21.15.116:0/2706752232 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d00327e0 con 0x7fe6cc0047c0 2023-11-28T17:11:05.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.810+0000 7fe6d7fff640 1 -- 172.21.15.116:0/2706752232 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0032b70 con 0x7fe6cc0047c0 2023-11-28T17:11:05.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.811+0000 7fe6e646e640 1 -- 172.21.15.116:0/2706752232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc0047c0 msgr2=0x7fe6cc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:05.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.811+0000 7fe6e646e640 1 --2- 172.21.15.116:0/2706752232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc0047c0 0x7fe6cc004c40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000ba30 tx=0x7fe6cc0057f0 comp rx=0 tx=0).stop 2023-11-28T17:11:05.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.811+0000 7fe6e646e640 1 -- 172.21.15.116:0/2706752232 shutdown_connections 2023-11-28T17:11:05.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.811+0000 7fe6e646e640 1 --2- 172.21.15.116:0/2706752232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc0047c0 0x7fe6cc004c40 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:05.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.811+0000 7fe6e646e640 1 -- 172.21.15.116:0/2706752232 >> 172.21.15.116:0/2706752232 conn(0x7fe6e00d2fc0 msgr2=0x7fe6e00d33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:05.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.811+0000 7fe6e646e640 1 -- 172.21.15.116:0/2706752232 shutdown_connections 2023-11-28T17:11:05.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.811+0000 7fe6e646e640 1 -- 172.21.15.116:0/2706752232 wait complete. 2023-11-28T17:11:05.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.813+0000 7fe6e646e640 1 Processor -- start 2023-11-28T17:11:05.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.813+0000 7fe6e646e640 1 -- start start 2023-11-28T17:11:05.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.814+0000 7fe6e646e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc0047c0 0x7fe6e00cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:05.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.814+0000 7fe6e646e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6e01f39f0 0x7fe6e01f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:05.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.814+0000 7fe6e646e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6e00c7dd0 0x7fe6e0149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:05.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.814+0000 7fe6e646e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e01f3e70 con 0x7fe6cc0047c0 2023-11-28T17:11:05.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.814+0000 7fe6e646e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e01f6a10 con 0x7fe6e00c7dd0 2023-11-28T17:11:05.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.814+0000 7fe6e646e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e01f6c90 con 0x7fe6e01f39f0 2023-11-28T17:11:05.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.814+0000 7fe6e546c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc0047c0 0x7fe6e00cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.814+0000 7fe6e546c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc0047c0 0x7fe6e00cb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54720/0 (socket says 172.21.15.116:54720) 2023-11-28T17:11:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.814+0000 7fe6e546c640 1 -- 172.21.15.116:0/2733317558 learned_addr learned my addr 172.21.15.116:0/2733317558 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.814+0000 7fe6e546c640 1 -- 172.21.15.116:0/2733317558 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6e01f39f0 msgr2=0x7fe6e01f64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:05.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.815+0000 7fe6e546c640 1 --2- 172.21.15.116:0/2733317558 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6e01f39f0 0x7fe6e01f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:05.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.815+0000 7fe6e546c640 1 -- 172.21.15.116:0/2733317558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6e00c7dd0 msgr2=0x7fe6e0149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:05.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.815+0000 7fe6e546c640 1 --2- 172.21.15.116:0/2733317558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6e00c7dd0 0x7fe6e0149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:05.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.815+0000 7fe6e546c640 1 -- 172.21.15.116:0/2733317558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e01f6f70 con 0x7fe6cc0047c0 2023-11-28T17:11:05.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.815+0000 7fe6e546c640 1 --2- 172.21.15.116:0/2733317558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc0047c0 0x7fe6e00cb5a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00c840 tx=0x7fe6dc00cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:05.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.815+0000 7fe6d5ffb640 1 -- 172.21.15.116:0/2733317558 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc0074e0 con 0x7fe6cc0047c0 2023-11-28T17:11:05.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.815+0000 7fe6d5ffb640 1 -- 172.21.15.116:0/2733317558 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6dc007640 con 0x7fe6cc0047c0 2023-11-28T17:11:05.825 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.815+0000 7fe6d5ffb640 1 -- 172.21.15.116:0/2733317558 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc004a20 con 0x7fe6cc0047c0 2023-11-28T17:11:05.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.815+0000 7fe6e646e640 1 -- 172.21.15.116:0/2733317558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e01f7250 con 0x7fe6cc0047c0 2023-11-28T17:11:05.826 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.815+0000 7fe6e646e640 1 -- 172.21.15.116:0/2733317558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e00b5ec0 con 0x7fe6cc0047c0 2023-11-28T17:11:05.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.816+0000 7fe6e646e640 1 -- 172.21.15.116:0/2733317558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e0067980 con 0x7fe6cc0047c0 2023-11-28T17:11:05.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.817+0000 7fe6d5ffb640 1 -- 172.21.15.116:0/2733317558 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe6dc004b80 con 0x7fe6cc0047c0 2023-11-28T17:11:05.827 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.818+0000 7fe6d5ffb640 1 --2- 172.21.15.116:0/2733317558 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe6c406b2a0 0x7fe6c406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:05.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.818+0000 7fe6d5ffb640 1 -- 172.21.15.116:0/2733317558 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe6dc088e60 con 0x7fe6cc0047c0 2023-11-28T17:11:05.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.819+0000 7fe6e4c6b640 1 --2- 172.21.15.116:0/2733317558 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe6c406b2a0 0x7fe6c406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:05.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.819+0000 7fe6e4c6b640 1 --2- 172.21.15.116:0/2733317558 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe6c406b2a0 0x7fe6c406d760 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000ba00 tx=0x7fe6d0032000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:05.828 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:05.823+0000 7fe6d5ffb640 1 -- 172.21.15.116:0/2733317558 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6dc053fd0 con 0x7fe6cc0047c0 2023-11-28T17:11:06.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.013+0000 7fe6e646e640 1 -- 172.21.15.116:0/2733317558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "ms_async_reap_threshold"} v 0) v1 -- 0x7fe6e00a0eb0 con 0x7fe6cc0047c0 2023-11-28T17:11:06.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.021+0000 7fe6d5ffb640 1 -- 172.21.15.116:0/2733317558 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "ms_async_reap_threshold"}]=0 v12) v1 ==== 106+0+0 (secure 0 0 0) 0x7fe6dc018070 con 0x7fe6cc0047c0 2023-11-28T17:11:06.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.024+0000 7fe6e646e640 1 -- 172.21.15.116:0/2733317558 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe6c406b2a0 msgr2=0x7fe6c406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:06.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.024+0000 7fe6e646e640 1 --2- 172.21.15.116:0/2733317558 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe6c406b2a0 0x7fe6c406d760 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000ba00 tx=0x7fe6d0032000 comp rx=0 tx=0).stop 2023-11-28T17:11:06.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.024+0000 7fe6e646e640 1 -- 172.21.15.116:0/2733317558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc0047c0 msgr2=0x7fe6e00cb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:06.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.024+0000 7fe6e646e640 1 --2- 172.21.15.116:0/2733317558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc0047c0 0x7fe6e00cb5a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00c840 tx=0x7fe6dc00cd00 comp rx=0 tx=0).stop 2023-11-28T17:11:06.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.025+0000 7fe6e646e640 1 -- 172.21.15.116:0/2733317558 shutdown_connections 2023-11-28T17:11:06.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.025+0000 7fe6e646e640 1 --2- 172.21.15.116:0/2733317558 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe6c406b2a0 0x7fe6c406d760 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.025+0000 7fe6e646e640 1 --2- 172.21.15.116:0/2733317558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6e00c7dd0 0x7fe6e0149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.025+0000 7fe6e646e640 1 --2- 172.21.15.116:0/2733317558 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe6e01f39f0 0x7fe6e01f64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.025+0000 7fe6e646e640 1 --2- 172.21.15.116:0/2733317558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6cc0047c0 0x7fe6e00cb5a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.025+0000 7fe6e646e640 1 -- 172.21.15.116:0/2733317558 >> 172.21.15.116:0/2733317558 conn(0x7fe6e00d2fc0 msgr2=0x7fe6e00cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.025+0000 7fe6e646e640 1 -- 172.21.15.116:0/2733317558 shutdown_connections 2023-11-28T17:11:06.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.025+0000 7fe6e646e640 1 -- 172.21.15.116:0/2733317558 wait complete. 2023-11-28T17:11:06.045 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-28T17:11:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.259+0000 7f8404a3d640 1 Processor -- start 2023-11-28T17:11:06.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.259+0000 7f8404a3d640 1 -- start start 2023-11-28T17:11:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.260+0000 7f8404a3d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f84000ceb10 0x7f840006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.260+0000 7f8404a3d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f840006ed00 0x7f840015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.260+0000 7f8404a3d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84000cef90 con 0x7f84000d0a60 2023-11-28T17:11:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.261+0000 7f8404a3d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f840006f190 con 0x7f84000ceb10 2023-11-28T17:11:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.261+0000 7f8404a3d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f840015b990 con 0x7f840006ed00 2023-11-28T17:11:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.261+0000 7f83f7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f84000ceb10 0x7f840006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.261+0000 7f83f7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f84000ceb10 0x7f840006e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54728/0 (socket says 172.21.15.116:54728) 2023-11-28T17:11:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.261+0000 7f83f7fff640 1 -- 172.21.15.116:0/2184455032 learned_addr learned my addr 172.21.15.116:0/2184455032 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.261+0000 7f83f7fff640 1 -- 172.21.15.116:0/2184455032 >> v2:172.21.15.156:3300/0 conn(0x7f840006ed00 msgr2=0x7f840015ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.261+0000 7f83f7fff640 1 --2- 172.21.15.116:0/2184455032 >> v2:172.21.15.156:3300/0 conn(0x7f840006ed00 0x7f840015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.261+0000 7f83f7fff640 1 -- 172.21.15.116:0/2184455032 >> v1:172.21.15.156:6789/0 conn(0x7f84000d0a60 legacy=0x7f84000ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.261+0000 7f83f7fff640 1 -- 172.21.15.116:0/2184455032 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840015bc70 con 0x7f84000ceb10 2023-11-28T17:11:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.262+0000 7f83f7fff640 1 --2- 172.21.15.116:0/2184455032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84000ceb10 0x7f840006e7c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f84001591a0 tx=0x7f83e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439b886ba1c5bd85 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.263+0000 7f83fdd9b640 1 -- 172.21.15.116:0/2184455032 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e8031870 con 0x7f84000ceb10 2023-11-28T17:11:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.263+0000 7f83fdd9b640 1 -- 172.21.15.116:0/2184455032 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e80319d0 con 0x7f84000ceb10 2023-11-28T17:11:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.263+0000 7f83fdd9b640 1 -- 172.21.15.116:0/2184455032 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e8031d60 con 0x7f84000ceb10 2023-11-28T17:11:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.263+0000 7f8404a3d640 1 -- 172.21.15.116:0/2184455032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84000ceb10 msgr2=0x7f840006e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.263+0000 7f8404a3d640 1 --2- 172.21.15.116:0/2184455032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84000ceb10 0x7f840006e7c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f84001591a0 tx=0x7f83e802b910 comp rx=0 tx=0).stop 2023-11-28T17:11:06.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.263+0000 7f8404a3d640 1 -- 172.21.15.116:0/2184455032 shutdown_connections 2023-11-28T17:11:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.263+0000 7f8404a3d640 1 --2- 172.21.15.116:0/2184455032 >> v2:172.21.15.156:3300/0 conn(0x7f840006ed00 0x7f840015ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.263+0000 7f8404a3d640 1 --2- 172.21.15.116:0/2184455032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84000ceb10 0x7f840006e7c0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.263+0000 7f8404a3d640 1 -- 172.21.15.116:0/2184455032 >> 172.21.15.116:0/2184455032 conn(0x7f84000d0f40 msgr2=0x7f84000d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.263+0000 7f8404a3d640 1 -- 172.21.15.116:0/2184455032 shutdown_connections 2023-11-28T17:11:06.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.264+0000 7f8404a3d640 1 -- 172.21.15.116:0/2184455032 wait complete. 2023-11-28T17:11:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.265+0000 7f8404a3d640 1 Processor -- start 2023-11-28T17:11:06.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.265+0000 7f8404a3d640 1 -- start start 2023-11-28T17:11:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.266+0000 7f8404a3d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f840006ed00 0x7f8400156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.266+0000 7f8404a3d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84000ceb10 0x7f84001f0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.266+0000 7f8404a3d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84000d0a60 0x7f840015a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:06.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.266+0000 7f8404a3d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8400156bc0 con 0x7f84000d0a60 2023-11-28T17:11:06.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.266+0000 7f8404a3d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84000b3e30 con 0x7f84000ceb10 2023-11-28T17:11:06.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.266+0000 7f8404a3d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84000b4110 con 0x7f840006ed00 2023-11-28T17:11:06.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.267+0000 7f83fed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84000d0a60 0x7f840015a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:06.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.267+0000 7f83fed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84000d0a60 0x7f840015a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54734/0 (socket says 172.21.15.116:54734) 2023-11-28T17:11:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.267+0000 7f83fed9d640 1 -- 172.21.15.116:0/2545275930 learned_addr learned my addr 172.21.15.116:0/2545275930 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:06.272 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.267+0000 7f83f7fff640 1 --2- 172.21.15.116:0/2545275930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84000ceb10 0x7f84001f0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.267+0000 7f83fe59c640 1 --2- 172.21.15.116:0/2545275930 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f840006ed00 0x7f8400156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.267+0000 7f83fed9d640 1 -- 172.21.15.116:0/2545275930 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f840006ed00 msgr2=0x7f8400156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:06.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.267+0000 7f83fed9d640 1 --2- 172.21.15.116:0/2545275930 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f840006ed00 0x7f8400156680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.267+0000 7f83fed9d640 1 -- 172.21.15.116:0/2545275930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84000ceb10 msgr2=0x7f84001f0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.267+0000 7f83fed9d640 1 --2- 172.21.15.116:0/2545275930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84000ceb10 0x7f84001f0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.267+0000 7f83fed9d640 1 -- 172.21.15.116:0/2545275930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e80094d0 con 0x7f84000d0a60 2023-11-28T17:11:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.267+0000 7f83fed9d640 1 --2- 172.21.15.116:0/2545275930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84000d0a60 0x7f840015a320 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f83f800b970 tx=0x7f83f800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:06.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.268+0000 7f83f6ffd640 1 -- 172.21.15.116:0/2545275930 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f800cb40 con 0x7f84000d0a60 2023-11-28T17:11:06.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.268+0000 7f83f6ffd640 1 -- 172.21.15.116:0/2545275930 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f800ccc0 con 0x7f84000d0a60 2023-11-28T17:11:06.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.268+0000 7f83f6ffd640 1 -- 172.21.15.116:0/2545275930 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f8014680 con 0x7f84000d0a60 2023-11-28T17:11:06.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.268+0000 7f8404a3d640 1 -- 172.21.15.116:0/2545275930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84000b43f0 con 0x7f84000d0a60 2023-11-28T17:11:06.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.268+0000 7f8404a3d640 1 -- 172.21.15.116:0/2545275930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84000b49b0 con 0x7f84000d0a60 2023-11-28T17:11:06.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.269+0000 7f8404a3d640 1 -- 172.21.15.116:0/2545275930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84000b4ba0 con 0x7f84000d0a60 2023-11-28T17:11:06.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.270+0000 7f83f6ffd640 1 -- 172.21.15.116:0/2545275930 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f83f801d3b0 con 0x7f84000d0a60 2023-11-28T17:11:06.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.271+0000 7f83f6ffd640 1 --2- 172.21.15.116:0/2545275930 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f83e006b2a0 0x7f83e006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:06.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.271+0000 7f83f6ffd640 1 -- 172.21.15.116:0/2545275930 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f83f80897f0 con 0x7f84000d0a60 2023-11-28T17:11:06.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.271+0000 7f83fe59c640 1 --2- 172.21.15.116:0/2545275930 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f83e006b2a0 0x7f83e006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:06.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.272+0000 7f83fe59c640 1 --2- 172.21.15.116:0/2545275930 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f83e006b2a0 0x7f83e006d760 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f83f0007fd0 tx=0x7f83f0009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:06.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.275+0000 7f83f6ffd640 1 -- 172.21.15.116:0/2545275930 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83f80549f0 con 0x7f84000d0a60 2023-11-28T17:11:06.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.465+0000 7f8404a3d640 1 -- 172.21.15.116:0/2545275930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f8400067460 con 0x7f84000d0a60 2023-11-28T17:11:06.472 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.471+0000 7f83f6ffd640 1 -- 172.21.15.116:0/2545275930 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f83f80190a0 con 0x7f84000d0a60 2023-11-28T17:11:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.473+0000 7f8404a3d640 1 -- 172.21.15.116:0/2545275930 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f83e006b2a0 msgr2=0x7f83e006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.473+0000 7f8404a3d640 1 --2- 172.21.15.116:0/2545275930 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f83e006b2a0 0x7f83e006d760 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f83f0007fd0 tx=0x7f83f0009040 comp rx=0 tx=0).stop 2023-11-28T17:11:06.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.473+0000 7f8404a3d640 1 -- 172.21.15.116:0/2545275930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84000d0a60 msgr2=0x7f840015a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.473+0000 7f8404a3d640 1 --2- 172.21.15.116:0/2545275930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84000d0a60 0x7f840015a320 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f83f800b970 tx=0x7f83f800be30 comp rx=0 tx=0).stop 2023-11-28T17:11:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.474+0000 7f8404a3d640 1 -- 172.21.15.116:0/2545275930 shutdown_connections 2023-11-28T17:11:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.474+0000 7f8404a3d640 1 --2- 172.21.15.116:0/2545275930 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f83e006b2a0 0x7f83e006d760 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.474+0000 7f8404a3d640 1 --2- 172.21.15.116:0/2545275930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84000d0a60 0x7f840015a320 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.474+0000 7f8404a3d640 1 --2- 172.21.15.116:0/2545275930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84000ceb10 0x7f84001f0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.474+0000 7f8404a3d640 1 --2- 172.21.15.116:0/2545275930 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f840006ed00 0x7f8400156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.474+0000 7f8404a3d640 1 -- 172.21.15.116:0/2545275930 >> 172.21.15.116:0/2545275930 conn(0x7f84000d0f40 msgr2=0x7f84000cceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.474+0000 7f8404a3d640 1 -- 172.21.15.116:0/2545275930 shutdown_connections 2023-11-28T17:11:06.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.474+0000 7f8404a3d640 1 -- 172.21.15.116:0/2545275930 wait complete. 2023-11-28T17:11:06.494 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_mount_conn_close' 2023-11-28T17:11:06.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.701+0000 7ff5de54c640 1 Processor -- start 2023-11-28T17:11:06.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.701+0000 7ff5de54c640 1 -- start start 2023-11-28T17:11:06.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.703+0000 7ff5de54c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5d813cee0 0x7ff5d806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:06.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.703+0000 7ff5de54c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d80aea20 con 0x7ff5d8140200 2023-11-28T17:11:06.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.703+0000 7ff5de54c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d813d2c0 con 0x7ff5d815aba0 2023-11-28T17:11:06.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.703+0000 7ff5de54c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d815b3a0 con 0x7ff5d813cee0 2023-11-28T17:11:06.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.703+0000 7ff5d77fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff5d8140200 0x7ff5d815a490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54264/0 (socket says 172.21.15.116:54264) 2023-11-28T17:11:06.705 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.704+0000 7ff5d77fe640 1 -- 172.21.15.116:0/3782758327 learned_addr learned my addr 172.21.15.116:0/3782758327 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:06.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.704+0000 7ff5d7fff640 1 --2- 172.21.15.116:0/3782758327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5d813cee0 0x7ff5d806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:06.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.704+0000 7ff5d6ffd640 1 -- 172.21.15.116:0/3782758327 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245344197 0 0) 0x7ff5d80aea20 con 0x7ff5d8140200 2023-11-28T17:11:06.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.704+0000 7ff5d6ffd640 1 -- 172.21.15.116:0/3782758327 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c4005730 con 0x7ff5d8140200 2023-11-28T17:11:06.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.704+0000 7ff5d6ffd640 1 -- 172.21.15.116:0/3782758327 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 774214797 0 0) 0x7ff5c4005730 con 0x7ff5d8140200 2023-11-28T17:11:06.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.704+0000 7ff5d6ffd640 1 -- 172.21.15.116:0/3782758327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5d813cee0 msgr2=0x7ff5d806c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:06.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.704+0000 7ff5d6ffd640 1 --2- 172.21.15.116:0/3782758327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5d813cee0 0x7ff5d806c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.704+0000 7ff5d6ffd640 1 -- 172.21.15.116:0/3782758327 >> v1:172.21.15.156:6789/0 conn(0x7ff5d815aba0 legacy=0x7ff5d80cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:06.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.704+0000 7ff5d6ffd640 1 -- 172.21.15.116:0/3782758327 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d815b0c0 con 0x7ff5d8140200 2023-11-28T17:11:06.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.704+0000 7ff5d6ffd640 1 -- 172.21.15.116:0/3782758327 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7ff5c0001670 con 0x7ff5d8140200 2023-11-28T17:11:06.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.704+0000 7ff5d6ffd640 1 -- 172.21.15.116:0/3782758327 >> v1:172.21.15.116:6789/0 conn(0x7ff5d8140200 legacy=0x7ff5d815a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:06.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.705+0000 7ff5d6ffd640 1 --2- 172.21.15.116:0/3782758327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5d813cee0 0x7ff5c4005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:06.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.705+0000 7ff5d6ffd640 1 --2- 172.21.15.116:0/3782758327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5c4006130 0x7ff5c40069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:06.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.705+0000 7ff5d6ffd640 1 --2- 172.21.15.116:0/3782758327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c4006f10 0x7ff5c4007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:06.709 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.705+0000 7ff5d6ffd640 1 -- 172.21.15.116:0/3782758327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d815b3a0 con 0x7ff5c4006f10 2023-11-28T17:11:06.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.705+0000 7ff5d6ffd640 1 -- 172.21.15.116:0/3782758327 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c400dd20 con 0x7ff5c4006130 2023-11-28T17:11:06.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.705+0000 7ff5d6ffd640 1 -- 172.21.15.116:0/3782758327 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c400dee0 con 0x7ff5d813cee0 2023-11-28T17:11:06.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.705+0000 7ff5dcae9640 1 --2- 172.21.15.116:0/3782758327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c4006f10 0x7ff5c4007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:06.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.705+0000 7ff5d77fe640 1 --2- 172.21.15.116:0/3782758327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5c4006130 0x7ff5c40069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:06.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.705+0000 7ff5d7fff640 1 --2- 172.21.15.116:0/3782758327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5d813cee0 0x7ff5c4005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:06.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.705+0000 7ff5dcae9640 1 -- 172.21.15.116:0/3782758327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5d813cee0 msgr2=0x7ff5c4005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:06.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.705+0000 7ff5dcae9640 1 --2- 172.21.15.116:0/3782758327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5d813cee0 0x7ff5c4005bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.705+0000 7ff5dcae9640 1 -- 172.21.15.116:0/3782758327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5c4006130 msgr2=0x7ff5c40069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:06.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.705+0000 7ff5dcae9640 1 --2- 172.21.15.116:0/3782758327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5c4006130 0x7ff5c40069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.705+0000 7ff5dcae9640 1 -- 172.21.15.116:0/3782758327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c4005730 con 0x7ff5c4006f10 2023-11-28T17:11:06.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.706+0000 7ff5dcae9640 1 --2- 172.21.15.116:0/3782758327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c4006f10 0x7ff5c4007390 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0001580 tx=0x7ff5c4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48a174019d3e300 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:06.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.707+0000 7ff5d6ffd640 1 -- 172.21.15.116:0/3782758327 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d00052d0 con 0x7ff5c4006f10 2023-11-28T17:11:06.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.707+0000 7ff5d6ffd640 1 -- 172.21.15.116:0/3782758327 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d002e050 con 0x7ff5c4006f10 2023-11-28T17:11:06.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.707+0000 7ff5d6ffd640 1 -- 172.21.15.116:0/3782758327 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d00035f0 con 0x7ff5c4006f10 2023-11-28T17:11:06.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.707+0000 7ff5de54c640 1 -- 172.21.15.116:0/3782758327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c4006f10 msgr2=0x7ff5c4007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:06.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.707+0000 7ff5de54c640 1 --2- 172.21.15.116:0/3782758327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5c4006f10 0x7ff5c4007390 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0001580 tx=0x7ff5c4007a00 comp rx=0 tx=0).stop 2023-11-28T17:11:06.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.707+0000 7ff5dcae9640 1 -- 172.21.15.116:0/3782758327 reap_dead start 2023-11-28T17:11:06.714 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.707+0000 7ff5de54c640 1 -- 172.21.15.116:0/3782758327 shutdown_connections 2023-11-28T17:11:06.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.707+0000 7ff5de54c640 1 -- 172.21.15.116:0/3782758327 >> 172.21.15.116:0/3782758327 conn(0x7ff5d806a4c0 msgr2=0x7ff5d806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:06.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.707+0000 7ff5de54c640 1 -- 172.21.15.116:0/3782758327 shutdown_connections 2023-11-28T17:11:06.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.707+0000 7ff5de54c640 1 -- 172.21.15.116:0/3782758327 wait complete. 2023-11-28T17:11:06.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.709+0000 7ff5de54c640 1 Processor -- start 2023-11-28T17:11:06.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.709+0000 7ff5de54c640 1 -- start start 2023-11-28T17:11:06.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.710+0000 7ff5de54c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5c4006f10 0x7ff5d81509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:06.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.710+0000 7ff5de54c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5d8154bd0 0x7ff5d8155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:06.715 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.710+0000 7ff5de54c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d8156870 0x7ff5d8156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:06.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.710+0000 7ff5de54c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d8151e50 con 0x7ff5d8156870 2023-11-28T17:11:06.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.710+0000 7ff5de54c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d8150ef0 con 0x7ff5d8154bd0 2023-11-28T17:11:06.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.710+0000 7ff5de54c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d80afdf0 con 0x7ff5c4006f10 2023-11-28T17:11:06.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.710+0000 7ff5dcae9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d8156870 0x7ff5d8156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:06.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.710+0000 7ff5dcae9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d8156870 0x7ff5d8156d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54756/0 (socket says 172.21.15.116:54756) 2023-11-28T17:11:06.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.710+0000 7ff5dcae9640 1 -- 172.21.15.116:0/2988257848 learned_addr learned my addr 172.21.15.116:0/2988257848 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:06.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.711+0000 7ff5d7fff640 1 --2- 172.21.15.116:0/2988257848 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5c4006f10 0x7ff5d81509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:06.716 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.711+0000 7ff5d77fe640 1 --2- 172.21.15.116:0/2988257848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5d8154bd0 0x7ff5d8155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:06.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.711+0000 7ff5dcae9640 1 -- 172.21.15.116:0/2988257848 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5c4006f10 msgr2=0x7ff5d81509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:06.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.711+0000 7ff5dcae9640 1 --2- 172.21.15.116:0/2988257848 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5c4006f10 0x7ff5d81509b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.711+0000 7ff5dcae9640 1 -- 172.21.15.116:0/2988257848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5d8154bd0 msgr2=0x7ff5d8155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:06.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.711+0000 7ff5dcae9640 1 --2- 172.21.15.116:0/2988257848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5d8154bd0 0x7ff5d8155050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:06.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.711+0000 7ff5dcae9640 1 -- 172.21.15.116:0/2988257848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d80b00d0 con 0x7ff5d8156870 2023-11-28T17:11:06.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.711+0000 7ff5dcae9640 1 --2- 172.21.15.116:0/2988257848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d8156870 0x7ff5d8156d20 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00016a0 tx=0x7ff5d0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:06.717 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.711+0000 7ff5d4ff9640 1 -- 172.21.15.116:0/2988257848 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0007a30 con 0x7ff5d8156870 2023-11-28T17:11:06.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.711+0000 7ff5d4ff9640 1 -- 172.21.15.116:0/2988257848 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d00067d0 con 0x7ff5d8156870 2023-11-28T17:11:06.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.711+0000 7ff5d4ff9640 1 -- 172.21.15.116:0/2988257848 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0006b60 con 0x7ff5d8156870 2023-11-28T17:11:06.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.711+0000 7ff5de54c640 1 -- 172.21.15.116:0/2988257848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d8151a70 con 0x7ff5d8156870 2023-11-28T17:11:06.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.711+0000 7ff5de54c640 1 -- 172.21.15.116:0/2988257848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d80ae990 con 0x7ff5d8156870 2023-11-28T17:11:06.718 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.712+0000 7ff5de54c640 1 -- 172.21.15.116:0/2988257848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d8151f90 con 0x7ff5d8156870 2023-11-28T17:11:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.713+0000 7ff5d4ff9640 1 -- 172.21.15.116:0/2988257848 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff5d0006cc0 con 0x7ff5d8156870 2023-11-28T17:11:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.714+0000 7ff5d4ff9640 1 --2- 172.21.15.116:0/2988257848 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff5b806b2d0 0x7ff5b806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.714+0000 7ff5d4ff9640 1 -- 172.21.15.116:0/2988257848 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff5d00ab2b0 con 0x7ff5d8156870 2023-11-28T17:11:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.714+0000 7ff5d7fff640 1 --2- 172.21.15.116:0/2988257848 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff5b806b2d0 0x7ff5b806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.715+0000 7ff5d7fff640 1 --2- 172.21.15.116:0/2988257848 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff5b806b2d0 0x7ff5b806d790 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8003a50 tx=0x7ff5c800b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:06.720 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.719+0000 7ff5d4ff9640 1 -- 172.21.15.116:0/2988257848 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff5d8151f90 con 0x7ff5d8156870 2023-11-28T17:11:06.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:06.969+0000 7ff5de54c640 1 -- 172.21.15.116:0/2988257848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_mount_conn_close"]} v 0) v1 -- 0x7ff5d813d2c0 con 0x7ff5d8156870 2023-11-28T17:11:07.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.588+0000 7ff5d4ff9640 1 -- 172.21.15.116:0/2988257848 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_sessionmap.TestSessionMap.test_mount_conn_close"]}]=0 v139) v1 ==== 144+0+0 (secure 0 0 0) 0x7ff5d813d2c0 con 0x7ff5d8156870 2023-11-28T17:11:07.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.592+0000 7ff5de54c640 1 -- 172.21.15.116:0/2988257848 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff5b806b2d0 msgr2=0x7ff5b806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:07.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.592+0000 7ff5de54c640 1 --2- 172.21.15.116:0/2988257848 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff5b806b2d0 0x7ff5b806d790 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8003a50 tx=0x7ff5c800b040 comp rx=0 tx=0).stop 2023-11-28T17:11:07.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.592+0000 7ff5de54c640 1 -- 172.21.15.116:0/2988257848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d8156870 msgr2=0x7ff5d8156d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:07.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.592+0000 7ff5de54c640 1 --2- 172.21.15.116:0/2988257848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d8156870 0x7ff5d8156d20 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00016a0 tx=0x7ff5d0001120 comp rx=0 tx=0).stop 2023-11-28T17:11:07.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.592+0000 7ff5de54c640 1 -- 172.21.15.116:0/2988257848 shutdown_connections 2023-11-28T17:11:07.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.592+0000 7ff5de54c640 1 --2- 172.21.15.116:0/2988257848 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff5b806b2d0 0x7ff5b806d790 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:07.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.592+0000 7ff5de54c640 1 --2- 172.21.15.116:0/2988257848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d8156870 0x7ff5d8156d20 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:07.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.592+0000 7ff5de54c640 1 --2- 172.21.15.116:0/2988257848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5d8154bd0 0x7ff5d8155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:07.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.592+0000 7ff5de54c640 1 --2- 172.21.15.116:0/2988257848 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5c4006f10 0x7ff5d81509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:07.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.592+0000 7ff5de54c640 1 -- 172.21.15.116:0/2988257848 >> 172.21.15.116:0/2988257848 conn(0x7ff5d806a4c0 msgr2=0x7ff5d81596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:07.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.592+0000 7ff5de54c640 1 -- 172.21.15.116:0/2988257848 shutdown_connections 2023-11-28T17:11:07.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.593+0000 7ff5de54c640 1 -- 172.21.15.116:0/2988257848 wait complete. 2023-11-28T17:11:07.615 INFO:tasks.cephfs_test_runner:That when a client unmounts, the thread count on the MDS goes back ... ok 2023-11-28T17:11:07.615 INFO:tasks.cephfs_test_runner:Starting test: test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap) Check that mds evicts blocklisted client 2023-11-28T17:11:07.615 INFO:tasks.cephfs_test_runner:test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap) 2023-11-28T17:11:07.615 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:11:07.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.831+0000 7f82da51d640 1 Processor -- start 2023-11-28T17:11:07.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.831+0000 7f82da51d640 1 -- start start 2023-11-28T17:11:07.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.833+0000 7f82da51d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82d41401c0 0x7f82d4071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:07.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.833+0000 7f82da51d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d41405a0 con 0x7f82d406c960 2023-11-28T17:11:07.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.833+0000 7f82da51d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d4140880 con 0x7f82d4071710 2023-11-28T17:11:07.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.833+0000 7f82da51d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d415db00 con 0x7f82d41401c0 2023-11-28T17:11:07.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.833+0000 7f82d8aba640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f82d406c960 0x7f82d40b3430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54280/0 (socket says 172.21.15.116:54280) 2023-11-28T17:11:07.834 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.833+0000 7f82d8aba640 1 -- 172.21.15.116:0/1828813176 learned_addr learned my addr 172.21.15.116:0/1828813176 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:07.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.833+0000 7f82d2ffd640 1 -- 172.21.15.116:0/1828813176 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590879243 0 0) 0x7f82d41405a0 con 0x7f82d406c960 2023-11-28T17:11:07.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.833+0000 7f82d2ffd640 1 -- 172.21.15.116:0/1828813176 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc003520 con 0x7f82d406c960 2023-11-28T17:11:07.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d3fff640 1 --2- 172.21.15.116:0/1828813176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82d41401c0 0x7f82d4071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:07.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d2ffd640 1 -- 172.21.15.116:0/1828813176 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4211028640 0 0) 0x7f82bc003520 con 0x7f82d406c960 2023-11-28T17:11:07.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d2ffd640 1 -- 172.21.15.116:0/1828813176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82d41401c0 msgr2=0x7f82d4071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:07.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d2ffd640 1 --2- 172.21.15.116:0/1828813176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82d41401c0 0x7f82d4071eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:07.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d2ffd640 1 -- 172.21.15.116:0/1828813176 >> v1:172.21.15.156:6789/0 conn(0x7f82d4071710 legacy=0x7f82d406c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:07.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d2ffd640 1 -- 172.21.15.116:0/1828813176 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d415dde0 con 0x7f82d406c960 2023-11-28T17:11:07.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d2ffd640 1 -- 172.21.15.116:0/1828813176 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f82cc0019f0 con 0x7f82d406c960 2023-11-28T17:11:07.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d2ffd640 1 -- 172.21.15.116:0/1828813176 >> v1:172.21.15.116:6789/0 conn(0x7f82d406c960 legacy=0x7f82d40b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:07.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d2ffd640 1 --2- 172.21.15.116:0/1828813176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82d41401c0 0x7f82bc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:07.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d2ffd640 1 --2- 172.21.15.116:0/1828813176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc003fd0 0x7f82bc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:07.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d2ffd640 1 --2- 172.21.15.116:0/1828813176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc004d00 0x7f82bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:07.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d2ffd640 1 -- 172.21.15.116:0/1828813176 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d415db00 con 0x7f82bc003fd0 2023-11-28T17:11:07.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d2ffd640 1 -- 172.21.15.116:0/1828813176 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc00bb40 con 0x7f82bc004d00 2023-11-28T17:11:07.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d2ffd640 1 -- 172.21.15.116:0/1828813176 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82bc00bd00 con 0x7f82d41401c0 2023-11-28T17:11:07.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.834+0000 7f82d37fe640 1 --2- 172.21.15.116:0/1828813176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc003fd0 0x7f82bc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:07.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.835+0000 7f82d3fff640 1 --2- 172.21.15.116:0/1828813176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82d41401c0 0x7f82bc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:07.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.835+0000 7f82d8aba640 1 --2- 172.21.15.116:0/1828813176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc004d00 0x7f82bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:07.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.835+0000 7f82d37fe640 1 -- 172.21.15.116:0/1828813176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82d41401c0 msgr2=0x7f82bc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.835+0000 7f82d37fe640 1 --2- 172.21.15.116:0/1828813176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82d41401c0 0x7f82bc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.835+0000 7f82d37fe640 1 -- 172.21.15.116:0/1828813176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc004d00 msgr2=0x7f82bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.835+0000 7f82d37fe640 1 --2- 172.21.15.116:0/1828813176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc004d00 0x7f82bc005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.835+0000 7f82d37fe640 1 -- 172.21.15.116:0/1828813176 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc003520 con 0x7f82bc003fd0 2023-11-28T17:11:07.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.836+0000 7f82d37fe640 1 --2- 172.21.15.116:0/1828813176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc003fd0 0x7f82bc0047c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f82c8001750 tx=0x7f82bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b8ed2932d22a0d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.836+0000 7f82d2ffd640 1 -- 172.21.15.116:0/1828813176 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c80052d0 con 0x7f82bc003fd0 2023-11-28T17:11:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.836+0000 7f82d2ffd640 1 -- 172.21.15.116:0/1828813176 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c802e050 con 0x7f82bc003fd0 2023-11-28T17:11:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.836+0000 7f82d2ffd640 1 -- 172.21.15.116:0/1828813176 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c8003830 con 0x7f82bc003fd0 2023-11-28T17:11:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.836+0000 7f82da51d640 1 -- 172.21.15.116:0/1828813176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc003fd0 msgr2=0x7f82bc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.836+0000 7f82da51d640 1 --2- 172.21.15.116:0/1828813176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc003fd0 0x7f82bc0047c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f82c8001750 tx=0x7f82bc0057f0 comp rx=0 tx=0).stop 2023-11-28T17:11:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.837+0000 7f82d8aba640 1 -- 172.21.15.116:0/1828813176 reap_dead start 2023-11-28T17:11:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.837+0000 7f82da51d640 1 -- 172.21.15.116:0/1828813176 shutdown_connections 2023-11-28T17:11:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.837+0000 7f82da51d640 1 -- 172.21.15.116:0/1828813176 >> 172.21.15.116:0/1828813176 conn(0x7f82d406a4c0 msgr2=0x7f82d406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.837+0000 7f82da51d640 1 -- 172.21.15.116:0/1828813176 shutdown_connections 2023-11-28T17:11:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.837+0000 7f82da51d640 1 -- 172.21.15.116:0/1828813176 wait complete. 2023-11-28T17:11:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.838+0000 7f82da51d640 1 Processor -- start 2023-11-28T17:11:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.839+0000 7f82da51d640 1 -- start start 2023-11-28T17:11:07.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82da51d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc003fd0 0x7f82d41f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:07.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82da51d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d41572d0 0x7f82d4158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:07.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82da51d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82d4157730 0x7f82d4158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:07.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82da51d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d41f3570 con 0x7f82d41572d0 2023-11-28T17:11:07.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82da51d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d415eb40 con 0x7f82bc003fd0 2023-11-28T17:11:07.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82da51d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d415ed90 con 0x7f82d4157730 2023-11-28T17:11:07.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82d37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d41572d0 0x7f82d4158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:07.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82d37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d41572d0 0x7f82d4158890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54772/0 (socket says 172.21.15.116:54772) 2023-11-28T17:11:07.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82d37fe640 1 -- 172.21.15.116:0/1089907005 learned_addr learned my addr 172.21.15.116:0/1089907005 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:07.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82d37fe640 1 -- 172.21.15.116:0/1089907005 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82d4157730 msgr2=0x7f82d4158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:07.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82d8aba640 1 --2- 172.21.15.116:0/1089907005 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82d4157730 0x7f82d4158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82d37fe640 1 --2- 172.21.15.116:0/1089907005 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82d4157730 0x7f82d4158dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82d3fff640 1 --2- 172.21.15.116:0/1089907005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc003fd0 0x7f82d41f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82d37fe640 1 -- 172.21.15.116:0/1089907005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc003fd0 msgr2=0x7f82d41f3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82d37fe640 1 --2- 172.21.15.116:0/1089907005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc003fd0 0x7f82d41f3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.840+0000 7f82d37fe640 1 -- 172.21.15.116:0/1089907005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d415f070 con 0x7f82d41572d0 2023-11-28T17:11:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.841+0000 7f82d37fe640 1 --2- 172.21.15.116:0/1089907005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d41572d0 0x7f82d4158890 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f82c8005040 tx=0x7f82c8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.841+0000 7f82d0ff9640 1 -- 172.21.15.116:0/1089907005 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c80076c0 con 0x7f82d41572d0 2023-11-28T17:11:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.841+0000 7f82d0ff9640 1 -- 172.21.15.116:0/1089907005 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c8007a30 con 0x7f82d41572d0 2023-11-28T17:11:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.841+0000 7f82d0ff9640 1 -- 172.21.15.116:0/1089907005 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c8006c00 con 0x7f82d41572d0 2023-11-28T17:11:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.841+0000 7f82da51d640 1 -- 172.21.15.116:0/1089907005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d415f350 con 0x7f82d41572d0 2023-11-28T17:11:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.841+0000 7f82da51d640 1 -- 172.21.15.116:0/1089907005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d415f910 con 0x7f82d41572d0 2023-11-28T17:11:07.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.842+0000 7f82da51d640 1 -- 172.21.15.116:0/1089907005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d415faa0 con 0x7f82d41572d0 2023-11-28T17:11:07.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.843+0000 7f82d0ff9640 1 -- 172.21.15.116:0/1089907005 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f82c80035f0 con 0x7f82d41572d0 2023-11-28T17:11:07.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.844+0000 7f82d0ff9640 1 --2- 172.21.15.116:0/1089907005 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f82c006b2a0 0x7f82c006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:07.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.844+0000 7f82d0ff9640 1 -- 172.21.15.116:0/1089907005 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f82c80ab5e0 con 0x7f82d41572d0 2023-11-28T17:11:07.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.844+0000 7f82d3fff640 1 --2- 172.21.15.116:0/1089907005 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f82c006b2a0 0x7f82c006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:07.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.845+0000 7f82d3fff640 1 --2- 172.21.15.116:0/1089907005 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f82c006b2a0 0x7f82c006d760 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f82c4006fd0 tx=0x7f82c4008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:07.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:07.849+0000 7f82d0ff9640 1 -- 172.21.15.116:0/1089907005 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82c80766b0 con 0x7f82d41572d0 2023-11-28T17:11:08.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.042+0000 7f82da51d640 1 -- 172.21.15.116:0/1089907005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82d40693d0 con 0x7f82d41572d0 2023-11-28T17:11:08.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.044+0000 7f82d0ff9640 1 -- 172.21.15.116:0/1089907005 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+13000 (secure 0 0 0) 0x7f82c8079cd0 con 0x7f82d41572d0 2023-11-28T17:11:08.045 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:08.046 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":39,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:11:05.567504+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6837/3177126349":"2023-11-29T17:11:02.546167+0000","172.21.15.116:6836/3177126349":"2023-11-29T17:11:02.546167+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:11:08.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.046+0000 7f82da51d640 1 -- 172.21.15.116:0/1089907005 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f82c006b2a0 msgr2=0x7f82c006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:08.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.046+0000 7f82da51d640 1 --2- 172.21.15.116:0/1089907005 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f82c006b2a0 0x7f82c006d760 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f82c4006fd0 tx=0x7f82c4008040 comp rx=0 tx=0).stop 2023-11-28T17:11:08.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.046+0000 7f82da51d640 1 -- 172.21.15.116:0/1089907005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d41572d0 msgr2=0x7f82d4158890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:08.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.046+0000 7f82da51d640 1 --2- 172.21.15.116:0/1089907005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d41572d0 0x7f82d4158890 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f82c8005040 tx=0x7f82c8001120 comp rx=0 tx=0).stop 2023-11-28T17:11:08.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.047+0000 7f82da51d640 1 -- 172.21.15.116:0/1089907005 shutdown_connections 2023-11-28T17:11:08.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.047+0000 7f82da51d640 1 --2- 172.21.15.116:0/1089907005 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f82c006b2a0 0x7f82c006d760 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.047+0000 7f82da51d640 1 --2- 172.21.15.116:0/1089907005 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82d4157730 0x7f82d4158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.047+0000 7f82da51d640 1 --2- 172.21.15.116:0/1089907005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d41572d0 0x7f82d4158890 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.047+0000 7f82da51d640 1 --2- 172.21.15.116:0/1089907005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc003fd0 0x7f82d41f3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.047+0000 7f82da51d640 1 -- 172.21.15.116:0/1089907005 >> 172.21.15.116:0/1089907005 conn(0x7f82d406a4c0 msgr2=0x7f82d413f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:08.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.047+0000 7f82da51d640 1 -- 172.21.15.116:0/1089907005 shutdown_connections 2023-11-28T17:11:08.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.047+0000 7f82da51d640 1 -- 172.21.15.116:0/1089907005 wait complete. 2023-11-28T17:11:08.067 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:11:08.068 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:11:08.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.271+0000 7f4d099d5640 1 Processor -- start 2023-11-28T17:11:08.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.271+0000 7f4d099d5640 1 -- start start 2023-11-28T17:11:08.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.273+0000 7f4d099d5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4d04146b10 0x7f4d04161a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:08.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.273+0000 7f4d099d5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0406eaf0 con 0x7f4d040d0f10 2023-11-28T17:11:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.273+0000 7f4d099d5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d0415e4e0 con 0x7f4d0415dfb0 2023-11-28T17:11:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.273+0000 7f4d099d5640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d0415e7c0 con 0x7f4d04146b10 2023-11-28T17:11:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.273+0000 7f4d02ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4d04146b10 0x7f4d04161a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.273+0000 7f4d02ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4d04146b10 0x7f4d04161a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41260/0 (socket says 172.21.15.116:41260) 2023-11-28T17:11:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.273+0000 7f4d02ffd640 1 -- 172.21.15.116:0/3609840905 learned_addr learned my addr 172.21.15.116:0/3609840905 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:08.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.274+0000 7f4d02ffd640 1 -- 172.21.15.116:0/3609840905 >> v1:172.21.15.156:6789/0 conn(0x7f4d0415dfb0 legacy=0x7f4d0406bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:08.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.274+0000 7f4d02ffd640 1 -- 172.21.15.116:0/3609840905 >> v1:172.21.15.116:6789/0 conn(0x7f4d040d0f10 legacy=0x7f4d0415d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:08.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.274+0000 7f4d02ffd640 1 -- 172.21.15.116:0/3609840905 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d04160f50 con 0x7f4d04146b10 2023-11-28T17:11:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.275+0000 7f4d02ffd640 1 --2- 172.21.15.116:0/3609840905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d04146b10 0x7f4d04161a30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4009e50 tx=0x7f4cf402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1e559f6498c38c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.275+0000 7f4d027fc640 1 -- 172.21.15.116:0/3609840905 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf402cb10 con 0x7f4d04146b10 2023-11-28T17:11:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.275+0000 7f4d027fc640 1 -- 172.21.15.116:0/3609840905 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf402cc70 con 0x7f4d04146b10 2023-11-28T17:11:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.275+0000 7f4d027fc640 1 -- 172.21.15.116:0/3609840905 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf40343f0 con 0x7f4d04146b10 2023-11-28T17:11:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.276+0000 7f4d099d5640 1 -- 172.21.15.116:0/3609840905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d04146b10 msgr2=0x7f4d04161a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.276+0000 7f4d099d5640 1 --2- 172.21.15.116:0/3609840905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d04146b10 0x7f4d04161a30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4009e50 tx=0x7f4cf402bf70 comp rx=0 tx=0).stop 2023-11-28T17:11:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.276+0000 7f4d099d5640 1 -- 172.21.15.116:0/3609840905 shutdown_connections 2023-11-28T17:11:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.276+0000 7f4d099d5640 1 --2- 172.21.15.116:0/3609840905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d04146b10 0x7f4d04161a30 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.276+0000 7f4d099d5640 1 -- 172.21.15.116:0/3609840905 >> 172.21.15.116:0/3609840905 conn(0x7f4d04070af0 msgr2=0x7f4d040662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:08.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.276+0000 7f4d099d5640 1 -- 172.21.15.116:0/3609840905 shutdown_connections 2023-11-28T17:11:08.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.276+0000 7f4d099d5640 1 -- 172.21.15.116:0/3609840905 wait complete. 2023-11-28T17:11:08.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.278+0000 7f4d099d5640 1 Processor -- start 2023-11-28T17:11:08.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.278+0000 7f4d099d5640 1 -- start start 2023-11-28T17:11:08.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4d099d5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d040d0f10 0x7f4d040d0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:08.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4d099d5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d04146b10 0x7f4d0413b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:08.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4d099d5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d0415dfb0 0x7f4d040cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:08.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4d099d5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d040a9d10 con 0x7f4d04146b10 2023-11-28T17:11:08.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4d099d5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d040ab320 con 0x7f4d040d0f10 2023-11-28T17:11:08.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4d099d5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d040ce940 con 0x7f4d0415dfb0 2023-11-28T17:11:08.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4cfa7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d04146b10 0x7f4d0413b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:08.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4cfa7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d04146b10 0x7f4d0413b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41274/0 (socket says 172.21.15.116:41274) 2023-11-28T17:11:08.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4cfa7fc640 1 -- 172.21.15.116:0/3401881867 learned_addr learned my addr 172.21.15.116:0/3401881867 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:08.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4d02ffd640 1 --2- 172.21.15.116:0/3401881867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d040d0f10 0x7f4d040d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:08.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4cfa7fc640 1 -- 172.21.15.116:0/3401881867 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d0415dfb0 msgr2=0x7f4d040cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:08.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4cfa7fc640 1 --2- 172.21.15.116:0/3401881867 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d0415dfb0 0x7f4d040cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4cfa7fc640 1 -- 172.21.15.116:0/3401881867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d040d0f10 msgr2=0x7f4d040d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:08.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4cfa7fc640 1 --2- 172.21.15.116:0/3401881867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d040d0f10 0x7f4d040d0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.279+0000 7f4cfa7fc640 1 -- 172.21.15.116:0/3401881867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf4009b30 con 0x7f4d04146b10 2023-11-28T17:11:08.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.280+0000 7f4cfa7fc640 1 --2- 172.21.15.116:0/3401881867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d04146b10 0x7f4d0413b7d0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00c450 tx=0x7f4cec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:08.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.280+0000 7f4d00ff9640 1 -- 172.21.15.116:0/3401881867 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec0052d0 con 0x7f4d04146b10 2023-11-28T17:11:08.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.280+0000 7f4d00ff9640 1 -- 172.21.15.116:0/3401881867 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cec005430 con 0x7f4d04146b10 2023-11-28T17:11:08.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.280+0000 7f4d00ff9640 1 -- 172.21.15.116:0/3401881867 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec012870 con 0x7f4d04146b10 2023-11-28T17:11:08.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.280+0000 7f4d099d5640 1 -- 172.21.15.116:0/3401881867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d0413bd10 con 0x7f4d04146b10 2023-11-28T17:11:08.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.280+0000 7f4d099d5640 1 -- 172.21.15.116:0/3401881867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d040a9a30 con 0x7f4d04146b10 2023-11-28T17:11:08.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.281+0000 7f4d099d5640 1 -- 172.21.15.116:0/3401881867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d04146670 con 0x7f4d04146b10 2023-11-28T17:11:08.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.282+0000 7f4d00ff9640 1 -- 172.21.15.116:0/3401881867 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4cec0129d0 con 0x7f4d04146b10 2023-11-28T17:11:08.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.283+0000 7f4d00ff9640 1 --2- 172.21.15.116:0/3401881867 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4ce406b2d0 0x7f4ce406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.283+0000 7f4d00ff9640 1 -- 172.21.15.116:0/3401881867 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4cec089340 con 0x7f4d04146b10 2023-11-28T17:11:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.283+0000 7f4d02ffd640 1 --2- 172.21.15.116:0/3401881867 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4ce406b2d0 0x7f4ce406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.284+0000 7f4d02ffd640 1 --2- 172.21.15.116:0/3401881867 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4ce406b2d0 0x7f4ce406d790 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4035000 tx=0x7f4cf4030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:08.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.288+0000 7f4d00ff9640 1 -- 172.21.15.116:0/3401881867 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4cec054540 con 0x7f4d04146b10 2023-11-28T17:11:08.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.478+0000 7f4d099d5640 1 -- 172.21.15.116:0/3401881867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4d04069360 con 0x7f4d04146b10 2023-11-28T17:11:08.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.480+0000 7f4d00ff9640 1 -- 172.21.15.116:0/3401881867 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4cec057b60 con 0x7f4d04146b10 2023-11-28T17:11:08.481 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:11:08.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.482+0000 7f4d099d5640 1 -- 172.21.15.116:0/3401881867 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4ce406b2d0 msgr2=0x7f4ce406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:08.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.482+0000 7f4d099d5640 1 --2- 172.21.15.116:0/3401881867 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4ce406b2d0 0x7f4ce406d790 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4035000 tx=0x7f4cf4030000 comp rx=0 tx=0).stop 2023-11-28T17:11:08.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.482+0000 7f4d099d5640 1 -- 172.21.15.116:0/3401881867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d04146b10 msgr2=0x7f4d0413b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:08.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.482+0000 7f4d099d5640 1 --2- 172.21.15.116:0/3401881867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d04146b10 0x7f4d0413b7d0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4cec00c450 tx=0x7f4cec00c910 comp rx=0 tx=0).stop 2023-11-28T17:11:08.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.482+0000 7f4d099d5640 1 -- 172.21.15.116:0/3401881867 shutdown_connections 2023-11-28T17:11:08.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.482+0000 7f4d099d5640 1 --2- 172.21.15.116:0/3401881867 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4ce406b2d0 0x7f4ce406d790 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.482+0000 7f4d099d5640 1 --2- 172.21.15.116:0/3401881867 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d0415dfb0 0x7f4d040cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.482+0000 7f4d099d5640 1 --2- 172.21.15.116:0/3401881867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d04146b10 0x7f4d0413b7d0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.482+0000 7f4d099d5640 1 --2- 172.21.15.116:0/3401881867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d040d0f10 0x7f4d040d0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.482+0000 7f4d099d5640 1 -- 172.21.15.116:0/3401881867 >> 172.21.15.116:0/3401881867 conn(0x7f4d04070af0 msgr2=0x7f4d0415fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:08.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.482+0000 7f4d099d5640 1 -- 172.21.15.116:0/3401881867 shutdown_connections 2023-11-28T17:11:08.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.483+0000 7f4d099d5640 1 -- 172.21.15.116:0/3401881867 wait complete. 2023-11-28T17:11:08.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_evict_blocklisted' 2023-11-28T17:11:08.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.717+0000 7fa38c461640 1 Processor -- start 2023-11-28T17:11:08.719 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.717+0000 7fa38c461640 1 -- start start 2023-11-28T17:11:08.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.719+0000 7fa38c461640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa384144430 0x7fa38415f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:08.721 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.719+0000 7fa38c461640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3840703f0 0x7fa384071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:08.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.719+0000 7fa38c461640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3840721b0 0x7fa38406c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:08.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.719+0000 7fa38c461640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa384144810 con 0x7fa3840703f0 2023-11-28T17:11:08.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.719+0000 7fa38c461640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa384070bd0 con 0x7fa384144430 2023-11-28T17:11:08.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.719+0000 7fa38c461640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa384070eb0 con 0x7fa3840721b0 2023-11-28T17:11:08.722 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.719+0000 7fa3899fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3840703f0 0x7fa384071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:08.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.719+0000 7fa3899fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3840703f0 0x7fa384071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41290/0 (socket says 172.21.15.116:41290) 2023-11-28T17:11:08.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.719+0000 7fa3899fc640 1 -- 172.21.15.116:0/2638755832 learned_addr learned my addr 172.21.15.116:0/2638755832 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:08.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.720+0000 7fa38a1fd640 1 --2- 172.21.15.116:0/2638755832 >> v2:172.21.15.156:3300/0 conn(0x7fa384144430 0x7fa38415f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:08.723 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.720+0000 7fa38a9fe640 1 --2- 172.21.15.116:0/2638755832 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3840721b0 0x7fa38406c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:08.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.720+0000 7fa3899fc640 1 -- 172.21.15.116:0/2638755832 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3840721b0 msgr2=0x7fa38406c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:08.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.720+0000 7fa3899fc640 1 --2- 172.21.15.116:0/2638755832 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3840721b0 0x7fa38406c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.720+0000 7fa3899fc640 1 -- 172.21.15.116:0/2638755832 >> v2:172.21.15.156:3300/0 conn(0x7fa384144430 msgr2=0x7fa38415f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:08.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.720+0000 7fa3899fc640 1 --2- 172.21.15.116:0/2638755832 >> v2:172.21.15.156:3300/0 conn(0x7fa384144430 0x7fa38415f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.720+0000 7fa3899fc640 1 -- 172.21.15.116:0/2638755832 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa384071190 con 0x7fa3840703f0 2023-11-28T17:11:08.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.721+0000 7fa3899fc640 1 --2- 172.21.15.116:0/2638755832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3840703f0 0x7fa384071c70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa38406e850 tx=0x7fa37402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9de348e4f3d839 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:08.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.721+0000 7fa3891fb640 1 -- 172.21.15.116:0/2638755832 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374031870 con 0x7fa3840703f0 2023-11-28T17:11:08.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.721+0000 7fa3891fb640 1 -- 172.21.15.116:0/2638755832 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3740319d0 con 0x7fa3840703f0 2023-11-28T17:11:08.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.721+0000 7fa3891fb640 1 -- 172.21.15.116:0/2638755832 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374031d60 con 0x7fa3840703f0 2023-11-28T17:11:08.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.722+0000 7fa38c461640 1 -- 172.21.15.116:0/2638755832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3840703f0 msgr2=0x7fa384071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:08.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.722+0000 7fa38c461640 1 --2- 172.21.15.116:0/2638755832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3840703f0 0x7fa384071c70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa38406e850 tx=0x7fa37402b910 comp rx=0 tx=0).stop 2023-11-28T17:11:08.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.722+0000 7fa38c461640 1 -- 172.21.15.116:0/2638755832 shutdown_connections 2023-11-28T17:11:08.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.722+0000 7fa38c461640 1 --2- 172.21.15.116:0/2638755832 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3840721b0 0x7fa38406c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.722+0000 7fa38c461640 1 --2- 172.21.15.116:0/2638755832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3840703f0 0x7fa384071c70 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.722+0000 7fa38c461640 1 --2- 172.21.15.116:0/2638755832 >> v2:172.21.15.156:3300/0 conn(0x7fa384144430 0x7fa38415f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.722+0000 7fa38c461640 1 -- 172.21.15.116:0/2638755832 >> 172.21.15.116:0/2638755832 conn(0x7fa38406a4c0 msgr2=0x7fa38406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:08.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.722+0000 7fa38c461640 1 -- 172.21.15.116:0/2638755832 shutdown_connections 2023-11-28T17:11:08.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.722+0000 7fa38c461640 1 -- 172.21.15.116:0/2638755832 wait complete. 2023-11-28T17:11:08.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.724+0000 7fa38c461640 1 Processor -- start 2023-11-28T17:11:08.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.724+0000 7fa38c461640 1 -- start start 2023-11-28T17:11:08.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.725+0000 7fa38c461640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3840703f0 0x7fa384142540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:08.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.725+0000 7fa38c461640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3840721b0 0x7fa3841f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:08.731 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.725+0000 7fa38c461640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa384144430 0x7fa3841f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.725+0000 7fa38c461640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3841f3ef0 con 0x7fa3840721b0 2023-11-28T17:11:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.725+0000 7fa38c461640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3841f3520 con 0x7fa384144430 2023-11-28T17:11:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.725+0000 7fa38c461640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38415a800 con 0x7fa3840703f0 2023-11-28T17:11:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.725+0000 7fa3899fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3840721b0 0x7fa3841f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:08.732 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.725+0000 7fa3899fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3840721b0 0x7fa3841f39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41306/0 (socket says 172.21.15.116:41306) 2023-11-28T17:11:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.725+0000 7fa3899fc640 1 -- 172.21.15.116:0/2407699244 learned_addr learned my addr 172.21.15.116:0/2407699244 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.726+0000 7fa3899fc640 1 -- 172.21.15.116:0/2407699244 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3840703f0 msgr2=0x7fa384142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.726+0000 7fa3899fc640 1 --2- 172.21.15.116:0/2407699244 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3840703f0 0x7fa384142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.726+0000 7fa38a9fe640 1 --2- 172.21.15.116:0/2407699244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa384144430 0x7fa3841f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:08.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.726+0000 7fa3899fc640 1 -- 172.21.15.116:0/2407699244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa384144430 msgr2=0x7fa3841f6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:08.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.726+0000 7fa3899fc640 1 --2- 172.21.15.116:0/2407699244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa384144430 0x7fa3841f6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:08.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.726+0000 7fa3899fc640 1 -- 172.21.15.116:0/2407699244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3740094d0 con 0x7fa3840721b0 2023-11-28T17:11:08.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.726+0000 7fa3899fc640 1 --2- 172.21.15.116:0/2407699244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3840721b0 0x7fa3841f39b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa37402b950 tx=0x7fa3740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:08.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.726+0000 7fa37affd640 1 -- 172.21.15.116:0/2407699244 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37403ad30 con 0x7fa3840721b0 2023-11-28T17:11:08.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.726+0000 7fa37affd640 1 -- 172.21.15.116:0/2407699244 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3740443f0 con 0x7fa3840721b0 2023-11-28T17:11:08.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.726+0000 7fa38c461640 1 -- 172.21.15.116:0/2407699244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38415aae0 con 0x7fa3840721b0 2023-11-28T17:11:08.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.726+0000 7fa37affd640 1 -- 172.21.15.116:0/2407699244 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3740447a0 con 0x7fa3840721b0 2023-11-28T17:11:08.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.726+0000 7fa38c461640 1 -- 172.21.15.116:0/2407699244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3841f2eb0 con 0x7fa3840721b0 2023-11-28T17:11:08.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.728+0000 7fa38c461640 1 -- 172.21.15.116:0/2407699244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3840693d0 con 0x7fa3840721b0 2023-11-28T17:11:08.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.729+0000 7fa37affd640 1 -- 172.21.15.116:0/2407699244 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa37404d450 con 0x7fa3840721b0 2023-11-28T17:11:08.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.730+0000 7fa37affd640 1 --2- 172.21.15.116:0/2407699244 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa36806b2a0 0x7fa36806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:08.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.730+0000 7fa37affd640 1 -- 172.21.15.116:0/2407699244 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa37402d040 con 0x7fa3840721b0 2023-11-28T17:11:08.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.730+0000 7fa38a1fd640 1 --2- 172.21.15.116:0/2407699244 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa36806b2a0 0x7fa36806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:08.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.730+0000 7fa38a1fd640 1 --2- 172.21.15.116:0/2407699244 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa36806b2a0 0x7fa36806d760 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa38000b7f0 tx=0x7fa380008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:08.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.734+0000 7fa37affd640 1 -- 172.21.15.116:0/2407699244 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa374077050 con 0x7fa3840721b0 2023-11-28T17:11:08.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:08.977+0000 7fa38c461640 1 -- 172.21.15.116:0/2407699244 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_evict_blocklisted"]} v 0) v1 -- 0x7fa3840b27e0 con 0x7fa3840721b0 2023-11-28T17:11:09.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.601+0000 7fa37affd640 1 -- 172.21.15.116:0/2407699244 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_sessionmap.TestSessionMap.test_session_evict_blocklisted"]}]=0 v141) v1 ==== 156+0+0 (secure 0 0 0) 0x7fa374035020 con 0x7fa3840721b0 2023-11-28T17:11:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.604+0000 7fa38c461640 1 -- 172.21.15.116:0/2407699244 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa36806b2a0 msgr2=0x7fa36806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.604+0000 7fa38c461640 1 --2- 172.21.15.116:0/2407699244 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa36806b2a0 0x7fa36806d760 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa38000b7f0 tx=0x7fa380008040 comp rx=0 tx=0).stop 2023-11-28T17:11:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.605+0000 7fa38c461640 1 -- 172.21.15.116:0/2407699244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3840721b0 msgr2=0x7fa3841f39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.605+0000 7fa38c461640 1 --2- 172.21.15.116:0/2407699244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3840721b0 0x7fa3841f39b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa37402b950 tx=0x7fa3740046c0 comp rx=0 tx=0).stop 2023-11-28T17:11:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.605+0000 7fa38c461640 1 -- 172.21.15.116:0/2407699244 shutdown_connections 2023-11-28T17:11:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.605+0000 7fa38c461640 1 --2- 172.21.15.116:0/2407699244 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa36806b2a0 0x7fa36806d760 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.605+0000 7fa38c461640 1 --2- 172.21.15.116:0/2407699244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa384144430 0x7fa3841f6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.605+0000 7fa38c461640 1 --2- 172.21.15.116:0/2407699244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3840721b0 0x7fa3841f39b0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.605+0000 7fa38c461640 1 --2- 172.21.15.116:0/2407699244 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa3840703f0 0x7fa384142540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.605+0000 7fa38c461640 1 -- 172.21.15.116:0/2407699244 >> 172.21.15.116:0/2407699244 conn(0x7fa38406a4c0 msgr2=0x7fa3841433b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.605+0000 7fa38c461640 1 -- 172.21.15.116:0/2407699244 shutdown_connections 2023-11-28T17:11:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.605+0000 7fa38c461640 1 -- 172.21.15.116:0/2407699244 wait complete. 2023-11-28T17:11:09.627 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-28T17:11:09.832 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.831+0000 7f2f44a9b640 1 Processor -- start 2023-11-28T17:11:09.833 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.831+0000 7f2f44a9b640 1 -- start start 2023-11-28T17:11:09.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.833+0000 7f2f44a9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f40146a60 0x7f2f400d1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:09.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.833+0000 7f2f44a9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2f400d16d0 0x7f2f40160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:09.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.833+0000 7f2f44a9b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f400d1b30 con 0x7f2f40161530 2023-11-28T17:11:09.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.833+0000 7f2f44a9b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f4015da00 con 0x7f2f400d16d0 2023-11-28T17:11:09.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.833+0000 7f2f44a9b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40161d60 con 0x7f2f40146a60 2023-11-28T17:11:09.835 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.834+0000 7f2f3e59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f40146a60 0x7f2f400d1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:09.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.834+0000 7f2f3e59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f40146a60 0x7f2f400d1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:33648/0 (socket says 172.21.15.116:33648) 2023-11-28T17:11:09.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.834+0000 7f2f3e59c640 1 -- 172.21.15.116:0/896466253 learned_addr learned my addr 172.21.15.116:0/896466253 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:09.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.834+0000 7f2f35d9b640 1 --2- 172.21.15.116:0/896466253 >> v2:172.21.15.156:3300/0 conn(0x7f2f400d16d0 0x7f2f40160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:09.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.834+0000 7f2f3e59c640 1 -- 172.21.15.116:0/896466253 >> v2:172.21.15.156:3300/0 conn(0x7f2f400d16d0 msgr2=0x7f2f40160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:09.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.834+0000 7f2f3e59c640 1 --2- 172.21.15.116:0/896466253 >> v2:172.21.15.156:3300/0 conn(0x7f2f400d16d0 0x7f2f40160ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:09.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.834+0000 7f2f3e59c640 1 -- 172.21.15.116:0/896466253 >> v1:172.21.15.156:6789/0 conn(0x7f2f40161530 legacy=0x7f2f400b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:09.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.834+0000 7f2f3e59c640 1 -- 172.21.15.116:0/896466253 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4015dcc0 con 0x7f2f40146a60 2023-11-28T17:11:09.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.836+0000 7f2f3e59c640 1 --2- 172.21.15.116:0/896466253 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f40146a60 0x7f2f400d1190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009870 tx=0x7f2f3002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15baded16809f500 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:09.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.836+0000 7f2f3dd9b640 1 -- 172.21.15.116:0/896466253 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f30031640 con 0x7f2f40146a60 2023-11-28T17:11:09.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.836+0000 7f2f3dd9b640 1 -- 172.21.15.116:0/896466253 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f300317a0 con 0x7f2f40146a60 2023-11-28T17:11:09.838 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.836+0000 7f2f3dd9b640 1 -- 172.21.15.116:0/896466253 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f30031b30 con 0x7f2f40146a60 2023-11-28T17:11:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.836+0000 7f2f44a9b640 1 -- 172.21.15.116:0/896466253 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f40146a60 msgr2=0x7f2f400d1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.836+0000 7f2f44a9b640 1 --2- 172.21.15.116:0/896466253 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f40146a60 0x7f2f400d1190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2f30009870 tx=0x7f2f3002b8f0 comp rx=0 tx=0).stop 2023-11-28T17:11:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.837+0000 7f2f44a9b640 1 -- 172.21.15.116:0/896466253 shutdown_connections 2023-11-28T17:11:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.837+0000 7f2f44a9b640 1 --2- 172.21.15.116:0/896466253 >> v2:172.21.15.156:3300/0 conn(0x7f2f400d16d0 0x7f2f40160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.837+0000 7f2f44a9b640 1 --2- 172.21.15.116:0/896466253 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f40146a60 0x7f2f400d1190 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.837+0000 7f2f44a9b640 1 -- 172.21.15.116:0/896466253 >> 172.21.15.116:0/896466253 conn(0x7f2f4006a6a0 msgr2=0x7f2f4006aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.837+0000 7f2f44a9b640 1 -- 172.21.15.116:0/896466253 shutdown_connections 2023-11-28T17:11:09.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.837+0000 7f2f44a9b640 1 -- 172.21.15.116:0/896466253 wait complete. 2023-11-28T17:11:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.838+0000 7f2f44a9b640 1 Processor -- start 2023-11-28T17:11:09.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.839+0000 7f2f44a9b640 1 -- start start 2023-11-28T17:11:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.839+0000 7f2f44a9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f400d16d0 0x7f2f401431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.839+0000 7f2f44a9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f40146a60 0x7f2f401412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.840+0000 7f2f44a9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f40161530 0x7f2f4015c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.840+0000 7f2f44a9b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4013cd20 con 0x7f2f40146a60 2023-11-28T17:11:09.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.840+0000 7f2f44a9b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40141830 con 0x7f2f40161530 2023-11-28T17:11:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.840+0000 7f2f44a9b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40141ab0 con 0x7f2f400d16d0 2023-11-28T17:11:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.840+0000 7f2f35d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f40146a60 0x7f2f401412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.840+0000 7f2f35d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f40146a60 0x7f2f401412f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41318/0 (socket says 172.21.15.116:41318) 2023-11-28T17:11:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.840+0000 7f2f35d9b640 1 -- 172.21.15.116:0/1338699716 learned_addr learned my addr 172.21.15.116:0/1338699716 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.840+0000 7f2f3e59c640 1 --2- 172.21.15.116:0/1338699716 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f400d16d0 0x7f2f401431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.840+0000 7f2f35d9b640 1 -- 172.21.15.116:0/1338699716 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f400d16d0 msgr2=0x7f2f401431f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:09.843 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.840+0000 7f2f35d9b640 1 --2- 172.21.15.116:0/1338699716 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f400d16d0 0x7f2f401431f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.840+0000 7f2f35d9b640 1 -- 172.21.15.116:0/1338699716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f40161530 msgr2=0x7f2f4015c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.840+0000 7f2f35d9b640 1 --2- 172.21.15.116:0/1338699716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f40161530 0x7f2f4015c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.840+0000 7f2f35d9b640 1 -- 172.21.15.116:0/1338699716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f300094d0 con 0x7f2f40146a60 2023-11-28T17:11:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.840+0000 7f2f35d9b640 1 --2- 172.21.15.116:0/1338699716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f40146a60 0x7f2f401412f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800efc0 tx=0x7f2f2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.841+0000 7f2f377fe640 1 -- 172.21.15.116:0/1338699716 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28014480 con 0x7f2f40146a60 2023-11-28T17:11:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.841+0000 7f2f377fe640 1 -- 172.21.15.116:0/1338699716 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2800f040 con 0x7f2f40146a60 2023-11-28T17:11:09.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.841+0000 7f2f377fe640 1 -- 172.21.15.116:0/1338699716 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28014800 con 0x7f2f40146a60 2023-11-28T17:11:09.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.841+0000 7f2f44a9b640 1 -- 172.21.15.116:0/1338699716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f40141d90 con 0x7f2f40146a60 2023-11-28T17:11:09.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.841+0000 7f2f44a9b640 1 -- 172.21.15.116:0/1338699716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f40142350 con 0x7f2f40146a60 2023-11-28T17:11:09.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.842+0000 7f2f44a9b640 1 -- 172.21.15.116:0/1338699716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f400693d0 con 0x7f2f40146a60 2023-11-28T17:11:09.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.843+0000 7f2f377fe640 1 -- 172.21.15.116:0/1338699716 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2f28014960 con 0x7f2f40146a60 2023-11-28T17:11:09.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.844+0000 7f2f377fe640 1 --2- 172.21.15.116:0/1338699716 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2f2006b2a0 0x7f2f2006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:09.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.844+0000 7f2f377fe640 1 -- 172.21.15.116:0/1338699716 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2f2808a5f0 con 0x7f2f40146a60 2023-11-28T17:11:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.844+0000 7f2f3e59c640 1 --2- 172.21.15.116:0/1338699716 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2f2006b2a0 0x7f2f2006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:09.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.845+0000 7f2f3e59c640 1 --2- 172.21.15.116:0/1338699716 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2f2006b2a0 0x7f2f2006d760 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2f30036000 tx=0x7f2f3003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:09.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:09.850+0000 7f2f377fe640 1 -- 172.21.15.116:0/1338699716 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f280556c0 con 0x7f2f40146a60 2023-11-28T17:11:10.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.040+0000 7f2f44a9b640 1 -- 172.21.15.116:0/1338699716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2f40070cf0 con 0x7f2f40146a60 2023-11-28T17:11:10.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.046+0000 7f2f377fe640 1 -- 172.21.15.116:0/1338699716 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2f28058ce0 con 0x7f2f40146a60 2023-11-28T17:11:10.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.049+0000 7f2f44a9b640 1 -- 172.21.15.116:0/1338699716 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2f2006b2a0 msgr2=0x7f2f2006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:10.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.049+0000 7f2f44a9b640 1 --2- 172.21.15.116:0/1338699716 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2f2006b2a0 0x7f2f2006d760 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2f30036000 tx=0x7f2f3003c040 comp rx=0 tx=0).stop 2023-11-28T17:11:10.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.049+0000 7f2f44a9b640 1 -- 172.21.15.116:0/1338699716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f40146a60 msgr2=0x7f2f401412f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:10.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.049+0000 7f2f44a9b640 1 --2- 172.21.15.116:0/1338699716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f40146a60 0x7f2f401412f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2f2800efc0 tx=0x7f2f2800c490 comp rx=0 tx=0).stop 2023-11-28T17:11:10.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.049+0000 7f2f44a9b640 1 -- 172.21.15.116:0/1338699716 shutdown_connections 2023-11-28T17:11:10.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.050+0000 7f2f44a9b640 1 --2- 172.21.15.116:0/1338699716 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2f2006b2a0 0x7f2f2006d760 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:10.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.050+0000 7f2f44a9b640 1 --2- 172.21.15.116:0/1338699716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f40161530 0x7f2f4015c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:10.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.050+0000 7f2f44a9b640 1 --2- 172.21.15.116:0/1338699716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f40146a60 0x7f2f401412f0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:10.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.050+0000 7f2f44a9b640 1 --2- 172.21.15.116:0/1338699716 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2f400d16d0 0x7f2f401431f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:10.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.050+0000 7f2f44a9b640 1 -- 172.21.15.116:0/1338699716 >> 172.21.15.116:0/1338699716 conn(0x7f2f4006a6a0 msgr2=0x7f2f4006aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:10.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.050+0000 7f2f44a9b640 1 -- 172.21.15.116:0/1338699716 shutdown_connections 2023-11-28T17:11:10.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.050+0000 7f2f44a9b640 1 -- 172.21.15.116:0/1338699716 wait complete. 2023-11-28T17:11:10.070 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T17:11:10.070 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T17:11:10.178 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T17:11:10.295 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T17:11:10.295 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:11:10.296 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:11:10.355 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054299,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T17:11:10.356 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:11:10.356 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:11:10.417 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T17:11:10.417 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:11:10.417 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:11:10.476 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054299,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T17:11:10.476 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:11:10.476 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:11:10.538 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T17:11:10.538 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-28T17:11:10.539 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:11:10.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.759+0000 7f78167ec640 1 Processor -- start 2023-11-28T17:11:10.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.760+0000 7f78167ec640 1 -- start start 2023-11-28T17:11:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.761+0000 7f78167ec640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f781015aba0 0x7f7810164e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:10.763 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.761+0000 7f78167ec640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78100aea20 con 0x7f781013cee0 2023-11-28T17:11:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.761+0000 7f78167ec640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781013d2c0 con 0x7f781006c960 2023-11-28T17:11:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.761+0000 7f78167ec640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f781015b3a0 con 0x7f781015aba0 2023-11-28T17:11:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.761+0000 7f780ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f781013cee0 0x7f781006c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47200/0 (socket says 172.21.15.116:47200) 2023-11-28T17:11:10.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.761+0000 7f780ffff640 1 -- 172.21.15.116:0/3479626905 learned_addr learned my addr 172.21.15.116:0/3479626905 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:10.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.762+0000 7f7814d89640 1 --2- 172.21.15.116:0/3479626905 >> v2:172.21.15.116:3300/0 conn(0x7f781015aba0 0x7f7810164e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:10.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.762+0000 7f780f7fe640 1 -- 172.21.15.116:0/3479626905 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469083707 0 0) 0x7f78100aea20 con 0x7f781013cee0 2023-11-28T17:11:10.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.762+0000 7f780f7fe640 1 -- 172.21.15.116:0/3479626905 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77fc003520 con 0x7f781013cee0 2023-11-28T17:11:10.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.762+0000 7f7814d89640 1 -- 172.21.15.116:0/3479626905 >> v1:172.21.15.156:6789/0 conn(0x7f781006c960 legacy=0x7f781015a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:10.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.762+0000 7f7814d89640 1 -- 172.21.15.116:0/3479626905 >> v1:172.21.15.116:6789/0 conn(0x7f781013cee0 legacy=0x7f781006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:10.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.762+0000 7f7814d89640 1 -- 172.21.15.116:0/3479626905 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781015b0c0 con 0x7f781015aba0 2023-11-28T17:11:10.767 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.763+0000 7f7814d89640 1 --2- 172.21.15.116:0/3479626905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781015aba0 0x7f7810164e10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7808009d90 tx=0x7f780802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd45118d8df9e87 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:10.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.763+0000 7f780f7fe640 1 -- 172.21.15.116:0/3479626905 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780802cbd0 con 0x7f781015aba0 2023-11-28T17:11:10.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.763+0000 7f780f7fe640 1 -- 172.21.15.116:0/3479626905 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f780802cd30 con 0x7f781015aba0 2023-11-28T17:11:10.768 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.763+0000 7f780f7fe640 1 -- 172.21.15.116:0/3479626905 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780802cea0 con 0x7f781015aba0 2023-11-28T17:11:10.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.764+0000 7f78167ec640 1 -- 172.21.15.116:0/3479626905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781015aba0 msgr2=0x7f7810164e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:10.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.764+0000 7f78167ec640 1 --2- 172.21.15.116:0/3479626905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781015aba0 0x7f7810164e10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7808009d90 tx=0x7f780802be20 comp rx=0 tx=0).stop 2023-11-28T17:11:10.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.764+0000 7f78167ec640 1 -- 172.21.15.116:0/3479626905 shutdown_connections 2023-11-28T17:11:10.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.764+0000 7f78167ec640 1 --2- 172.21.15.116:0/3479626905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781015aba0 0x7f7810164e10 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:10.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.764+0000 7f78167ec640 1 -- 172.21.15.116:0/3479626905 >> 172.21.15.116:0/3479626905 conn(0x7f781006a4c0 msgr2=0x7f781006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:10.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.764+0000 7f78167ec640 1 -- 172.21.15.116:0/3479626905 shutdown_connections 2023-11-28T17:11:10.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.764+0000 7f78167ec640 1 -- 172.21.15.116:0/3479626905 wait complete. 2023-11-28T17:11:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.766+0000 7f78167ec640 1 Processor -- start 2023-11-28T17:11:10.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.766+0000 7f78167ec640 1 -- start start 2023-11-28T17:11:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.767+0000 7f78167ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781006c960 0x7f7810071f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.767+0000 7f78167ec640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781013cee0 0x7f781013a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.767+0000 7f78167ec640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f781015aba0 0x7f7810135c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.767+0000 7f78167ec640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7810070a70 con 0x7f781006c960 2023-11-28T17:11:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.767+0000 7f78167ec640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781013abf0 con 0x7f781015aba0 2023-11-28T17:11:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.767+0000 7f78167ec640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7810070490 con 0x7f781013cee0 2023-11-28T17:11:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.767+0000 7f780ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781006c960 0x7f7810071f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.767+0000 7f780ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781006c960 0x7f7810071f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41328/0 (socket says 172.21.15.116:41328) 2023-11-28T17:11:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.767+0000 7f780ffff640 1 -- 172.21.15.116:0/3540362988 learned_addr learned my addr 172.21.15.116:0/3540362988 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.768+0000 7f780ffff640 1 -- 172.21.15.116:0/3540362988 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781013cee0 msgr2=0x7f781013a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.768+0000 7f780ffff640 1 --2- 172.21.15.116:0/3540362988 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781013cee0 0x7f781013a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.768+0000 7f780ffff640 1 -- 172.21.15.116:0/3540362988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f781015aba0 msgr2=0x7f7810135c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:10.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.768+0000 7f780ffff640 1 --2- 172.21.15.116:0/3540362988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f781015aba0 0x7f7810135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.768+0000 7f780ffff640 1 -- 172.21.15.116:0/3540362988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7808009a70 con 0x7f781006c960 2023-11-28T17:11:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.768+0000 7f780ffff640 1 --2- 172.21.15.116:0/3540362988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781006c960 0x7f7810071f80 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f780000c500 tx=0x7f780000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.768+0000 7f780d7fa640 1 -- 172.21.15.116:0/3540362988 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7800005ce0 con 0x7f781006c960 2023-11-28T17:11:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.768+0000 7f780d7fa640 1 -- 172.21.15.116:0/3540362988 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7800005e40 con 0x7f781006c960 2023-11-28T17:11:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.768+0000 7f78167ec640 1 -- 172.21.15.116:0/3540362988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7810148c70 con 0x7f781006c960 2023-11-28T17:11:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.768+0000 7f780d7fa640 1 -- 172.21.15.116:0/3540362988 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7800007330 con 0x7f781006c960 2023-11-28T17:11:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.768+0000 7f78167ec640 1 -- 172.21.15.116:0/3540362988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7810070700 con 0x7f781006c960 2023-11-28T17:11:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.769+0000 7f78167ec640 1 -- 172.21.15.116:0/3540362988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f781006b270 con 0x7f781006c960 2023-11-28T17:11:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.770+0000 7f780d7fa640 1 -- 172.21.15.116:0/3540362988 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f7800007550 con 0x7f781006c960 2023-11-28T17:11:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.771+0000 7f780d7fa640 1 --2- 172.21.15.116:0/3540362988 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f77f006b2a0 0x7f77f006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.771+0000 7f780d7fa640 1 -- 172.21.15.116:0/3540362988 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7800088ee0 con 0x7f781006c960 2023-11-28T17:11:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.771+0000 7f7807fff640 1 --2- 172.21.15.116:0/3540362988 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f77f006b2a0 0x7f77f006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.772+0000 7f7807fff640 1 --2- 172.21.15.116:0/3540362988 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f77f006b2a0 0x7f77f006d760 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7810157d30 tx=0x7f77f80079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:10.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:10.776+0000 7f780d7fa640 1 -- 172.21.15.116:0/3540362988 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78000541a0 con 0x7f781006c960 2023-11-28T17:11:11.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:11.005+0000 7f78167ec640 1 -- 172.21.15.116:0/3540362988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78100693d0 con 0x7f781006c960 2023-11-28T17:11:11.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:11.006+0000 7f780d7fa640 1 -- 172.21.15.116:0/3540362988 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f7800017090 con 0x7f781006c960 2023-11-28T17:11:11.008 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:11.008 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4619,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6834/491158514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":491158514},{"type":"v1","addr":"172.21.15.156:6834","nonce":491158514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3927645596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3927645596},{"type":"v1","addr":"172.21.15.116:6835","nonce":3927645596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4738,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/141796020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6833","nonce":141796020},{"type":"v1","addr":"172.21.15.156:6835","nonce":141796020}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5736,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/2326053922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":2326053922},{"type":"v1","addr":"172.21.15.116:6837","nonce":2326053922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[]} 2023-11-28T17:11:11.008 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 17 2023-11-28T17:11:11.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:11.009+0000 7f78167ec640 1 -- 172.21.15.116:0/3540362988 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f77f006b2a0 msgr2=0x7f77f006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:11.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:11.009+0000 7f78167ec640 1 --2- 172.21.15.116:0/3540362988 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f77f006b2a0 0x7f77f006d760 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7810157d30 tx=0x7f77f80079b0 comp rx=0 tx=0).stop 2023-11-28T17:11:11.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:11.009+0000 7f78167ec640 1 -- 172.21.15.116:0/3540362988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781006c960 msgr2=0x7f7810071f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:11.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:11.009+0000 7f78167ec640 1 --2- 172.21.15.116:0/3540362988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781006c960 0x7f7810071f80 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f780000c500 tx=0x7f780000c9c0 comp rx=0 tx=0).stop 2023-11-28T17:11:11.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:11.009+0000 7f78167ec640 1 -- 172.21.15.116:0/3540362988 shutdown_connections 2023-11-28T17:11:11.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:11.009+0000 7f78167ec640 1 --2- 172.21.15.116:0/3540362988 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f77f006b2a0 0x7f77f006d760 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:11.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:11.009+0000 7f78167ec640 1 --2- 172.21.15.116:0/3540362988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f781015aba0 0x7f7810135c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:11.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:11.009+0000 7f78167ec640 1 --2- 172.21.15.116:0/3540362988 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781013cee0 0x7f781013a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:11.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:11.009+0000 7f78167ec640 1 --2- 172.21.15.116:0/3540362988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f781006c960 0x7f7810071f80 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:11.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:11.009+0000 7f78167ec640 1 -- 172.21.15.116:0/3540362988 >> 172.21.15.116:0/3540362988 conn(0x7f781006a4c0 msgr2=0x7f7810150720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:11.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:11.009+0000 7f78167ec640 1 -- 172.21.15.116:0/3540362988 shutdown_connections 2023-11-28T17:11:11.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:11.009+0000 7f78167ec640 1 -- 172.21.15.116:0/3540362988 wait complete. 2023-11-28T17:11:11.031 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-28T17:11:11.031 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-28T17:11:11.032 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T17:11:11.032 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:11:11.032 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-28T17:11:11.032 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-28T17:11:11.032 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T17:11:11.032 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:11:11.033 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-28T17:11:11.033 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-28T17:11:11.033 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T17:11:11.033 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:11:11.033 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-28T17:11:11.033 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-28T17:11:11.034 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T17:11:11.034 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:11:11.055 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T17:11:11.055 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-28T17:11:11.056 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T17:11:11.056 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-28T17:11:11.058 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T17:11:11.058 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-28T17:11:11.104 INFO:tasks.ceph.mds.c:Started 2023-11-28T17:11:11.105 DEBUG:teuthology.parallel:result is None 2023-11-28T17:11:11.106 INFO:tasks.ceph.mds.b:Started 2023-11-28T17:11:11.106 DEBUG:teuthology.parallel:result is None 2023-11-28T17:11:11.111 INFO:tasks.ceph.mds.a:Started 2023-11-28T17:11:11.111 DEBUG:teuthology.parallel:result is None 2023-11-28T17:11:11.266 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2023-11-28T17:11:11.271 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2023-11-28T17:11:11.291 INFO:tasks.ceph.mds.b.smithi156.stdout:starting mds.b at 2023-11-28T17:11:15.394 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-11-28T17:11:17.057 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T17:11:17.058 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-28T17:11:17.067 INFO:tasks.ceph.mds.d:Started 2023-11-28T17:11:17.068 DEBUG:teuthology.parallel:result is None 2023-11-28T17:11:17.068 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-28T17:11:17.257 INFO:tasks.ceph.mds.d.smithi156.stdout:starting mds.d at 2023-11-28T17:11:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.288+0000 7ff7a753c640 1 Processor -- start 2023-11-28T17:11:17.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.288+0000 7ff7a753c640 1 -- start start 2023-11-28T17:11:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.290+0000 7ff7a753c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff7a013cee0 0x7ff7a006c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.290+0000 7ff7a753c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a00aea20 con 0x7ff7a0140200 2023-11-28T17:11:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.290+0000 7ff7a753c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7a013d2c0 con 0x7ff7a015aba0 2023-11-28T17:11:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.290+0000 7ff7a753c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7a015b3a0 con 0x7ff7a013cee0 2023-11-28T17:11:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.290+0000 7ff7a4ad7640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff7a0140200 0x7ff7a015a490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47224/0 (socket says 172.21.15.116:47224) 2023-11-28T17:11:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.290+0000 7ff7a4ad7640 1 -- 172.21.15.116:0/3723348162 learned_addr learned my addr 172.21.15.116:0/3723348162 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.290+0000 7ff7a52d8640 1 --2- 172.21.15.116:0/3723348162 >> v2:172.21.15.156:3300/0 conn(0x7ff7a013cee0 0x7ff7a006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.291+0000 7ff797fff640 1 -- 172.21.15.116:0/3723348162 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539823071 0 0) 0x7ff7a00aea20 con 0x7ff7a0140200 2023-11-28T17:11:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.291+0000 7ff797fff640 1 -- 172.21.15.116:0/3723348162 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff78c003520 con 0x7ff7a0140200 2023-11-28T17:11:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.291+0000 7ff7a52d8640 1 -- 172.21.15.116:0/3723348162 >> v1:172.21.15.156:6789/0 conn(0x7ff7a015aba0 legacy=0x7ff7a00cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.291+0000 7ff797fff640 1 -- 172.21.15.116:0/3723348162 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218379446 0 0) 0x7ff798001490 con 0x7ff7a015aba0 2023-11-28T17:11:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.291+0000 7ff7a52d8640 1 -- 172.21.15.116:0/3723348162 >> v1:172.21.15.116:6789/0 conn(0x7ff7a0140200 legacy=0x7ff7a015a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.291+0000 7ff7a52d8640 1 -- 172.21.15.116:0/3723348162 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a015b0c0 con 0x7ff7a013cee0 2023-11-28T17:11:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.292+0000 7ff7a52d8640 1 --2- 172.21.15.116:0/3723348162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a013cee0 0x7ff7a006c250 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0097a0 tx=0x7ff79c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85c6e3e8f220711e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.292+0000 7ff797fff640 1 -- 172.21.15.116:0/3723348162 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c031830 con 0x7ff7a013cee0 2023-11-28T17:11:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.293+0000 7ff797fff640 1 -- 172.21.15.116:0/3723348162 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79c031990 con 0x7ff7a013cee0 2023-11-28T17:11:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.293+0000 7ff797fff640 1 -- 172.21.15.116:0/3723348162 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79c031d20 con 0x7ff7a013cee0 2023-11-28T17:11:17.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.293+0000 7ff7a753c640 1 -- 172.21.15.116:0/3723348162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a013cee0 msgr2=0x7ff7a006c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:17.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.293+0000 7ff7a753c640 1 --2- 172.21.15.116:0/3723348162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a013cee0 0x7ff7a006c250 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0097a0 tx=0x7ff79c02b8d0 comp rx=0 tx=0).stop 2023-11-28T17:11:17.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.293+0000 7ff7a753c640 1 -- 172.21.15.116:0/3723348162 shutdown_connections 2023-11-28T17:11:17.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.293+0000 7ff7a753c640 1 --2- 172.21.15.116:0/3723348162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a013cee0 0x7ff7a006c250 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:17.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.293+0000 7ff7a753c640 1 -- 172.21.15.116:0/3723348162 >> 172.21.15.116:0/3723348162 conn(0x7ff7a006a4c0 msgr2=0x7ff7a006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:17.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.293+0000 7ff7a753c640 1 -- 172.21.15.116:0/3723348162 shutdown_connections 2023-11-28T17:11:17.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.293+0000 7ff7a753c640 1 -- 172.21.15.116:0/3723348162 wait complete. 2023-11-28T17:11:17.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.295+0000 7ff7a753c640 1 Processor -- start 2023-11-28T17:11:17.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.295+0000 7ff7a753c640 1 -- start start 2023-11-28T17:11:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.296+0000 7ff7a753c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff7a013cee0 0x7ff7a0138710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.296+0000 7ff7a753c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a0140200 0x7ff7a00b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.296+0000 7ff7a753c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7a015aba0 0x7ff7a015b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.296+0000 7ff7a753c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a00b31b0 con 0x7ff7a015aba0 2023-11-28T17:11:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.296+0000 7ff7a753c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a00b2bf0 con 0x7ff7a0140200 2023-11-28T17:11:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.296+0000 7ff7a753c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a00b2ed0 con 0x7ff7a013cee0 2023-11-28T17:11:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.296+0000 7ff7a5ad9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7a015aba0 0x7ff7a015b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.296+0000 7ff7a5ad9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7a015aba0 0x7ff7a015b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41374/0 (socket says 172.21.15.116:41374) 2023-11-28T17:11:17.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.297+0000 7ff7a5ad9640 1 -- 172.21.15.116:0/2177162435 learned_addr learned my addr 172.21.15.116:0/2177162435 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.297+0000 7ff7a4ad7640 1 --2- 172.21.15.116:0/2177162435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a0140200 0x7ff7a00b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.297+0000 7ff7a52d8640 1 --2- 172.21.15.116:0/2177162435 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff7a013cee0 0x7ff7a0138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.297+0000 7ff7a5ad9640 1 -- 172.21.15.116:0/2177162435 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff7a013cee0 msgr2=0x7ff7a0138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.297+0000 7ff7a5ad9640 1 --2- 172.21.15.116:0/2177162435 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff7a013cee0 0x7ff7a0138710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.297+0000 7ff7a5ad9640 1 -- 172.21.15.116:0/2177162435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a0140200 msgr2=0x7ff7a00b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.297+0000 7ff7a5ad9640 1 --2- 172.21.15.116:0/2177162435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a0140200 0x7ff7a00b2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.297+0000 7ff7a5ad9640 1 -- 172.21.15.116:0/2177162435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c009400 con 0x7ff7a015aba0 2023-11-28T17:11:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.297+0000 7ff7a5ad9640 1 --2- 172.21.15.116:0/2177162435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7a015aba0 0x7ff7a015b5c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff79800c780 tx=0x7ff79800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.297+0000 7ff795ffb640 1 -- 172.21.15.116:0/2177162435 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff798003da0 con 0x7ff7a015aba0 2023-11-28T17:11:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.297+0000 7ff795ffb640 1 -- 172.21.15.116:0/2177162435 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79800ce30 con 0x7ff7a015aba0 2023-11-28T17:11:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.297+0000 7ff795ffb640 1 -- 172.21.15.116:0/2177162435 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff798005290 con 0x7ff7a015aba0 2023-11-28T17:11:17.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.297+0000 7ff7a753c640 1 -- 172.21.15.116:0/2177162435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a0155fb0 con 0x7ff7a015aba0 2023-11-28T17:11:17.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.298+0000 7ff7a753c640 1 -- 172.21.15.116:0/2177162435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a0138c50 con 0x7ff7a015aba0 2023-11-28T17:11:17.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.299+0000 7ff7a753c640 1 -- 172.21.15.116:0/2177162435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a01564d0 con 0x7ff7a015aba0 2023-11-28T17:11:17.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.300+0000 7ff795ffb640 1 -- 172.21.15.116:0/2177162435 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff7980053f0 con 0x7ff7a015aba0 2023-11-28T17:11:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.301+0000 7ff795ffb640 1 --2- 172.21.15.116:0/2177162435 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff78406b2a0 0x7ff78406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:17.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.301+0000 7ff795ffb640 1 -- 172.21.15.116:0/2177162435 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff798088e50 con 0x7ff7a015aba0 2023-11-28T17:11:17.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.301+0000 7ff7a52d8640 1 --2- 172.21.15.116:0/2177162435 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff78406b2a0 0x7ff78406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:17.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.302+0000 7ff7a52d8640 1 --2- 172.21.15.116:0/2177162435 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff78406b2a0 0x7ff78406d760 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff79c036000 tx=0x7ff79c03d040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:17.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.306+0000 7ff795ffb640 1 -- 172.21.15.116:0/2177162435 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff7a01564d0 con 0x7ff7a015aba0 2023-11-28T17:11:17.511 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:17.509+0000 7ff7a753c640 1 -- 172.21.15.116:0/2177162435 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ff7a013d2c0 con 0x7ff7a015aba0 2023-11-28T17:11:19.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.147+0000 7ff795ffb640 1 -- 172.21.15.116:0/2177162435 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v44) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff7a013d2c0 con 0x7ff7a015aba0 2023-11-28T17:11:19.149 INFO:teuthology.orchestra.run.smithi116.stderr: removed all blocklist entries 2023-11-28T17:11:19.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.151+0000 7ff7a753c640 1 -- 172.21.15.116:0/2177162435 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff78406b2a0 msgr2=0x7ff78406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:19.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.151+0000 7ff7a753c640 1 --2- 172.21.15.116:0/2177162435 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff78406b2a0 0x7ff78406d760 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff79c036000 tx=0x7ff79c03d040 comp rx=0 tx=0).stop 2023-11-28T17:11:19.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.151+0000 7ff7a753c640 1 -- 172.21.15.116:0/2177162435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7a015aba0 msgr2=0x7ff7a015b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:19.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.151+0000 7ff7a753c640 1 --2- 172.21.15.116:0/2177162435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7a015aba0 0x7ff7a015b5c0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff79800c780 tx=0x7ff79800cc40 comp rx=0 tx=0).stop 2023-11-28T17:11:19.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.151+0000 7ff7a753c640 1 -- 172.21.15.116:0/2177162435 shutdown_connections 2023-11-28T17:11:19.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.151+0000 7ff7a753c640 1 --2- 172.21.15.116:0/2177162435 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff78406b2a0 0x7ff78406d760 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.151+0000 7ff7a753c640 1 --2- 172.21.15.116:0/2177162435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7a015aba0 0x7ff7a015b5c0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.151+0000 7ff7a753c640 1 --2- 172.21.15.116:0/2177162435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff7a0140200 0x7ff7a00b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.152+0000 7ff7a753c640 1 --2- 172.21.15.116:0/2177162435 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff7a013cee0 0x7ff7a0138710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.152+0000 7ff7a753c640 1 -- 172.21.15.116:0/2177162435 >> 172.21.15.116:0/2177162435 conn(0x7ff7a006a4c0 msgr2=0x7ff7a006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:19.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.152+0000 7ff7a753c640 1 -- 172.21.15.116:0/2177162435 shutdown_connections 2023-11-28T17:11:19.154 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.152+0000 7ff7a753c640 1 -- 172.21.15.116:0/2177162435 wait complete. 2023-11-28T17:11:19.177 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-28T17:11:19.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.392+0000 7f0adb5e5640 1 Processor -- start 2023-11-28T17:11:19.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.393+0000 7f0adb5e5640 1 -- start start 2023-11-28T17:11:19.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.394+0000 7f0adb5e5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0ad406c960 0x7f0ad415a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:19.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.394+0000 7f0adb5e5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ad4140200 0x7f0ad415a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:19.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.394+0000 7f0adb5e5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad40aea20 con 0x7f0ad413cee0 2023-11-28T17:11:19.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.394+0000 7f0adb5e5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad413d2c0 con 0x7f0ad406c960 2023-11-28T17:11:19.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.394+0000 7f0adb5e5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad415b3a0 con 0x7f0ad4140200 2023-11-28T17:11:19.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.395+0000 7f0ad9b82640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ad4140200 0x7f0ad415a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:19.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.395+0000 7f0ad8b80640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0ad406c960 0x7f0ad415a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:19.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.395+0000 7f0ad8b80640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0ad406c960 0x7f0ad415a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:57470/0 (socket says 172.21.15.116:57470) 2023-11-28T17:11:19.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.395+0000 7f0ad8b80640 1 -- 172.21.15.116:0/3857319311 learned_addr learned my addr 172.21.15.116:0/3857319311 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:19.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.396+0000 7f0ad9b82640 1 -- 172.21.15.116:0/3857319311 >> v2:172.21.15.156:3300/0 conn(0x7f0ad406c960 msgr2=0x7f0ad415a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:19.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.396+0000 7f0ad9b82640 1 --2- 172.21.15.116:0/3857319311 >> v2:172.21.15.156:3300/0 conn(0x7f0ad406c960 0x7f0ad415a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.396+0000 7f0ad9b82640 1 -- 172.21.15.116:0/3857319311 >> v1:172.21.15.156:6789/0 conn(0x7f0ad413cee0 legacy=0x7f0ad406c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:19.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.396+0000 7f0ad9b82640 1 -- 172.21.15.116:0/3857319311 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad415b0c0 con 0x7f0ad4140200 2023-11-28T17:11:19.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.396+0000 7f0ad8b80640 1 --2- 172.21.15.116:0/3857319311 >> v2:172.21.15.156:3300/0 conn(0x7f0ad406c960 0x7f0ad415a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:11:19.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.397+0000 7f0ad9b82640 1 --2- 172.21.15.116:0/3857319311 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ad4140200 0x7f0ad415a9d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0097a0 tx=0x7f0acc02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1de860668e80f31b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.398+0000 7f0ad3fff640 1 -- 172.21.15.116:0/3857319311 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc031780 con 0x7f0ad4140200 2023-11-28T17:11:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.398+0000 7f0ad3fff640 1 -- 172.21.15.116:0/3857319311 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0acc0318e0 con 0x7f0ad4140200 2023-11-28T17:11:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.398+0000 7f0ad3fff640 1 -- 172.21.15.116:0/3857319311 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc031c70 con 0x7f0ad4140200 2023-11-28T17:11:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.399+0000 7f0adb5e5640 1 -- 172.21.15.116:0/3857319311 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ad4140200 msgr2=0x7f0ad415a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.399+0000 7f0adb5e5640 1 --2- 172.21.15.116:0/3857319311 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ad4140200 0x7f0ad415a9d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0097a0 tx=0x7f0acc02b820 comp rx=0 tx=0).stop 2023-11-28T17:11:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.399+0000 7f0adb5e5640 1 -- 172.21.15.116:0/3857319311 shutdown_connections 2023-11-28T17:11:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.399+0000 7f0adb5e5640 1 --2- 172.21.15.116:0/3857319311 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ad4140200 0x7f0ad415a9d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.399+0000 7f0adb5e5640 1 --2- 172.21.15.116:0/3857319311 >> v2:172.21.15.156:3300/0 conn(0x7f0ad406c960 0x7f0ad415a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.399+0000 7f0adb5e5640 1 -- 172.21.15.116:0/3857319311 >> 172.21.15.116:0/3857319311 conn(0x7f0ad406a4c0 msgr2=0x7f0ad406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:19.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.399+0000 7f0adb5e5640 1 -- 172.21.15.116:0/3857319311 shutdown_connections 2023-11-28T17:11:19.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.399+0000 7f0adb5e5640 1 -- 172.21.15.116:0/3857319311 wait complete. 2023-11-28T17:11:19.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.400+0000 7f0adb5e5640 1 Processor -- start 2023-11-28T17:11:19.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.401+0000 7f0adb5e5640 1 -- start start 2023-11-28T17:11:19.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0adb5e5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ad406c960 0x7f0ad41367a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:19.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0adb5e5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ad413cee0 0x7f0ad4136f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:19.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0adb5e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad4140200 0x7f0ad40712e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:19.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0adb5e5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad413f410 con 0x7f0ad4140200 2023-11-28T17:11:19.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0adb5e5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4070520 con 0x7f0ad413cee0 2023-11-28T17:11:19.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0adb5e5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4070800 con 0x7f0ad406c960 2023-11-28T17:11:19.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0ad9b82640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad4140200 0x7f0ad40712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:19.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0ad9b82640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad4140200 0x7f0ad40712e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33808/0 (socket says 172.21.15.116:33808) 2023-11-28T17:11:19.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0ad9b82640 1 -- 172.21.15.116:0/1881075641 learned_addr learned my addr 172.21.15.116:0/1881075641 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:19.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0ad9381640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ad406c960 0x7f0ad41367a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:19.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0ad8b80640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ad413cee0 0x7f0ad4136f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:19.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0ad9b82640 1 -- 172.21.15.116:0/1881075641 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ad406c960 msgr2=0x7f0ad41367a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:19.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0ad9b82640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ad406c960 0x7f0ad41367a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0ad9b82640 1 -- 172.21.15.116:0/1881075641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ad413cee0 msgr2=0x7f0ad4136f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:19.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0ad9b82640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ad413cee0 0x7f0ad4136f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.402+0000 7f0ad9b82640 1 -- 172.21.15.116:0/1881075641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac00094d0 con 0x7f0ad4140200 2023-11-28T17:11:19.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.403+0000 7f0ad8b80640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ad413cee0 0x7f0ad4136f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:19.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.403+0000 7f0ad9b82640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad4140200 0x7f0ad40712e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0097a0 tx=0x7f0acc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:19.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.403+0000 7f0ad1ffb640 1 -- 172.21.15.116:0/1881075641 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc032820 con 0x7f0ad4140200 2023-11-28T17:11:19.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.403+0000 7f0ad1ffb640 1 -- 172.21.15.116:0/1881075641 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0acc032980 con 0x7f0ad4140200 2023-11-28T17:11:19.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.403+0000 7f0adb5e5640 1 -- 172.21.15.116:0/1881075641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0acc009400 con 0x7f0ad4140200 2023-11-28T17:11:19.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.403+0000 7f0ad1ffb640 1 -- 172.21.15.116:0/1881075641 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc032d10 con 0x7f0ad4140200 2023-11-28T17:11:19.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.403+0000 7f0adb5e5640 1 -- 172.21.15.116:0/1881075641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad4070dc0 con 0x7f0ad4140200 2023-11-28T17:11:19.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.404+0000 7f0adb5e5640 1 -- 172.21.15.116:0/1881075641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad40693d0 con 0x7f0ad4140200 2023-11-28T17:11:19.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.405+0000 7f0ad1ffb640 1 -- 172.21.15.116:0/1881075641 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0acc04d3f0 con 0x7f0ad4140200 2023-11-28T17:11:19.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.407+0000 7f0ad1ffb640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ab806b2a0 0x7f0ab806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:19.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.407+0000 7f0ad1ffb640 1 -- 172.21.15.116:0/1881075641 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0acc0ab410 con 0x7f0ad4140200 2023-11-28T17:11:19.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.407+0000 7f0ad9381640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ab806b2a0 0x7f0ab806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:19.409 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.408+0000 7f0ad9381640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ab806b2a0 0x7f0ab806d760 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8001d40 tx=0x7f0ac8009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:19.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.411+0000 7f0ad1ffb640 1 -- 172.21.15.116:0/1881075641 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0acc0765c0 con 0x7f0ad4140200 2023-11-28T17:11:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.669+0000 7f0adb5e5640 1 -- 172.21.15.116:0/1881075641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0ad406ced0 con 0x7f0ad4140200 2023-11-28T17:11:19.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.670+0000 7f0ad1ffb640 1 -- 172.21.15.116:0/1881075641 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0acc079be0 con 0x7f0ad4140200 2023-11-28T17:11:19.671 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:19.671 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-28T17:11:19.671 INFO:teuthology.orchestra.run.smithi116.stdout: "auth_dump": [ 2023-11-28T17:11:19.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.672 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.a", 2023-11-28T17:11:19.672 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBsHmZlDk7pERAAzfB2wxucZ6BlKfc/eBL5fA==", 2023-11-28T17:11:19.672 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.672 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:11:19.672 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-28T17:11:19.672 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-28T17:11:19.672 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.672 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.672 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.672 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.b", 2023-11-28T17:11:19.673 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBsHmZln5I1NRAAK30dyAqRuGLeJ4mGw90Rpg==", 2023-11-28T17:11:19.673 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.673 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:11:19.673 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-28T17:11:19.673 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-28T17:11:19.673 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.673 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.673 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.673 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.c", 2023-11-28T17:11:19.673 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBsHmZlvVteIxAAqG8H+6/g2owr7mI87oYHgA==", 2023-11-28T17:11:19.674 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.674 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:11:19.674 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-28T17:11:19.674 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-28T17:11:19.674 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.674 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.674 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.674 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mds.d", 2023-11-28T17:11:19.674 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBtHmZl6ieLCRAAVEyizVZ3mGTtPGRQeee11A==", 2023-11-28T17:11:19.674 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.674 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:11:19.675 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-28T17:11:19.675 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-28T17:11:19.675 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.675 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.675 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.675 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.0", 2023-11-28T17:11:19.675 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBxHmZlUqmdNhAAc7C+EUtBtgCaLUkr0PPrHg==", 2023-11-28T17:11:19.675 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.675 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:11:19.675 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:11:19.675 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.675 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.676 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.676 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.1", 2023-11-28T17:11:19.676 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQB1HmZlKQ07FRAAKncDe3f+D1TfLU8lhQVaGA==", 2023-11-28T17:11:19.676 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.676 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:11:19.676 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:11:19.676 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.676 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.676 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.676 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.2", 2023-11-28T17:11:19.676 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQB4HmZlvpDnLRAAG2kEjnvjJmln7TCBM9hUTw==", 2023-11-28T17:11:19.676 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.677 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:11:19.677 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:11:19.677 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.677 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.677 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.677 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.3", 2023-11-28T17:11:19.677 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQB8HmZlGD45CxAA2LESF1EBeNjPX6fjvSyEFA==", 2023-11-28T17:11:19.677 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.677 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:11:19.678 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:11:19.678 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.678 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.678 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.678 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.4", 2023-11-28T17:11:19.678 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQCBHmZlJOgqAxAAc23riVIAOTXw28t8OrTjQg==", 2023-11-28T17:11:19.678 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.678 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:11:19.678 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:11:19.678 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.678 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.679 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.679 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.5", 2023-11-28T17:11:19.679 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQCEHmZlvTMIHBAAuVkPBcYAAuTQUVqdkZ5liw==", 2023-11-28T17:11:19.679 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.679 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:11:19.679 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:11:19.679 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.679 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.679 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.679 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.6", 2023-11-28T17:11:19.680 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQCHHmZls5zfJBAAmH97PMKvbQXpeznCSkt6IQ==", 2023-11-28T17:11:19.680 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.680 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:11:19.680 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:11:19.680 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.680 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.680 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.680 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.680 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "osd.7", 2023-11-28T17:11:19.680 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQCKHmZlahyvLRAADZqskufX275VqtUDlUHU7Q==", 2023-11-28T17:11:19.680 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.680 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow profile osd", 2023-11-28T17:11:19.681 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile osd", 2023-11-28T17:11:19.681 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.681 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.681 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.681 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.681 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.0", 2023-11-28T17:11:19.681 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBtHmZlPRewFxAAfLJPZrTljlPIoghT5ZzoBA==", 2023-11-28T17:11:19.681 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.681 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:11:19.681 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow r", 2023-11-28T17:11:19.681 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-11-28T17:11:19.681 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.682 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.682 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.682 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.1", 2023-11-28T17:11:19.682 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBtHmZlWG0nLhAA4xwi4Ut4dfJWwY275L/mHQ==", 2023-11-28T17:11:19.682 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.682 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:11:19.682 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-28T17:11:19.682 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-28T17:11:19.682 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-28T17:11:19.682 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.682 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.682 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.682 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.2", 2023-11-28T17:11:19.683 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBtHmZleq8ZIxAAOZsBs+OlL6TXLGqwtX11pw==", 2023-11-28T17:11:19.683 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.683 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:11:19.683 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-28T17:11:19.683 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-28T17:11:19.683 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-28T17:11:19.683 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.683 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.683 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.683 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.3", 2023-11-28T17:11:19.683 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBtHmZlkoZmORAAPWGkPoFfqsaCfAjnWmfqtw==", 2023-11-28T17:11:19.683 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.684 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow", 2023-11-28T17:11:19.684 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow r", 2023-11-28T17:11:19.684 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow rw", 2023-11-28T17:11:19.684 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow rwx" 2023-11-28T17:11:19.684 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.684 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.684 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.684 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "client.admin", 2023-11-28T17:11:19.684 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBrHmZlYzCkBRAAuc8Y4PZZyyT7/RQ+2mQD6Q==", 2023-11-28T17:11:19.684 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.684 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-28T17:11:19.685 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr": "allow *", 2023-11-28T17:11:19.685 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow *", 2023-11-28T17:11:19.685 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.685 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.685 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.685 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.685 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mgr.x", 2023-11-28T17:11:19.685 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBsHmZlRWsIBhAAmDx975LSXMvBGPQf/P3etQ==", 2023-11-28T17:11:19.685 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.685 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-28T17:11:19.685 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile mgr", 2023-11-28T17:11:19.685 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.686 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.686 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:19.686 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:19.686 INFO:teuthology.orchestra.run.smithi116.stdout: "entity": "mgr.y", 2023-11-28T17:11:19.686 INFO:teuthology.orchestra.run.smithi116.stdout: "key": "AQBrHmZl159eNRAAqk03SlXduu04+ZNQe8YLUg==", 2023-11-28T17:11:19.686 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": { 2023-11-28T17:11:19.686 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": "allow *", 2023-11-28T17:11:19.686 INFO:teuthology.orchestra.run.smithi116.stdout: "mon": "allow profile mgr", 2023-11-28T17:11:19.686 INFO:teuthology.orchestra.run.smithi116.stdout: "osd": "allow *" 2023-11-28T17:11:19.686 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.686 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:19.687 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2023-11-28T17:11:19.687 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-28T17:11:19.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.672+0000 7f0adb5e5640 1 -- 172.21.15.116:0/1881075641 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ab806b2a0 msgr2=0x7f0ab806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:19.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.672+0000 7f0adb5e5640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ab806b2a0 0x7f0ab806d760 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8001d40 tx=0x7f0ac8009040 comp rx=0 tx=0).stop 2023-11-28T17:11:19.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.672+0000 7f0adb5e5640 1 -- 172.21.15.116:0/1881075641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad4140200 msgr2=0x7f0ad40712e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:19.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.672+0000 7f0adb5e5640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad4140200 0x7f0ad40712e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0097a0 tx=0x7f0acc0046c0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.672+0000 7f0adb5e5640 1 -- 172.21.15.116:0/1881075641 shutdown_connections 2023-11-28T17:11:19.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.673+0000 7f0adb5e5640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0ab806b2a0 0x7f0ab806d760 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.673+0000 7f0adb5e5640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ad4140200 0x7f0ad40712e0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.673+0000 7f0adb5e5640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ad413cee0 0x7f0ad4136f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.673+0000 7f0adb5e5640 1 --2- 172.21.15.116:0/1881075641 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ad406c960 0x7f0ad41367a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.673+0000 7f0adb5e5640 1 -- 172.21.15.116:0/1881075641 >> 172.21.15.116:0/1881075641 conn(0x7f0ad406a4c0 msgr2=0x7f0ad40aefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:19.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.673+0000 7f0adb5e5640 1 -- 172.21.15.116:0/1881075641 shutdown_connections 2023-11-28T17:11:19.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.673+0000 7f0adb5e5640 1 -- 172.21.15.116:0/1881075641 wait complete. 2023-11-28T17:11:19.694 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:11:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.906+0000 7ff34e02d640 1 Processor -- start 2023-11-28T17:11:19.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.906+0000 7ff34e02d640 1 -- start start 2023-11-28T17:11:19.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.907+0000 7ff34e02d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff34815dab0 0x7ff3480703f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:19.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.907+0000 7ff34e02d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff348154220 con 0x7ff3480b3310 2023-11-28T17:11:19.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.907+0000 7ff34e02d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34815d2f0 con 0x7ff34806cae0 2023-11-28T17:11:19.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.907+0000 7ff34e02d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34815a3c0 con 0x7ff34815dab0 2023-11-28T17:11:19.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.908+0000 7ff346ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff3480b3310 0x7ff348159cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53098/0 (socket says 172.21.15.116:53098) 2023-11-28T17:11:19.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.908+0000 7ff346ffd640 1 -- 172.21.15.116:0/2779724747 learned_addr learned my addr 172.21.15.116:0/2779724747 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:19.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.908+0000 7ff347fff640 1 --2- 172.21.15.116:0/2779724747 >> v2:172.21.15.156:3300/0 conn(0x7ff34815dab0 0x7ff3480703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:19.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.908+0000 7ff3467fc640 1 -- 172.21.15.116:0/2779724747 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120027054 0 0) 0x7ff33c001110 con 0x7ff3480b3310 2023-11-28T17:11:19.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.908+0000 7ff3467fc640 1 -- 172.21.15.116:0/2779724747 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff330005730 con 0x7ff3480b3310 2023-11-28T17:11:19.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.908+0000 7ff3467fc640 1 -- 172.21.15.116:0/2779724747 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1213758316 0 0) 0x7ff330005730 con 0x7ff3480b3310 2023-11-28T17:11:19.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.908+0000 7ff3467fc640 1 -- 172.21.15.116:0/2779724747 >> v2:172.21.15.156:3300/0 conn(0x7ff34815dab0 msgr2=0x7ff3480703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:19.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.908+0000 7ff3467fc640 1 --2- 172.21.15.116:0/2779724747 >> v2:172.21.15.156:3300/0 conn(0x7ff34815dab0 0x7ff3480703f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.908+0000 7ff3467fc640 1 -- 172.21.15.116:0/2779724747 >> v1:172.21.15.156:6789/0 conn(0x7ff34806cae0 legacy=0x7ff34806cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:19.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.908+0000 7ff3467fc640 1 -- 172.21.15.116:0/2779724747 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34815a5c0 con 0x7ff3480b3310 2023-11-28T17:11:19.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.908+0000 7ff3467fc640 1 -- 172.21.15.116:0/2779724747 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7ff33c001800 con 0x7ff3480b3310 2023-11-28T17:11:19.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff3467fc640 1 -- 172.21.15.116:0/2779724747 >> v1:172.21.15.116:6789/0 conn(0x7ff3480b3310 legacy=0x7ff348159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:19.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff3467fc640 1 --2- 172.21.15.116:0/2779724747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330005b80 0x7ff330005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:19.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff3467fc640 1 --2- 172.21.15.116:0/2779724747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3300069d0 0x7ff330006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:19.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff3467fc640 1 --2- 172.21.15.116:0/2779724747 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff330007390 0x7ff33000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:19.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff3467fc640 1 -- 172.21.15.116:0/2779724747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34815a3c0 con 0x7ff330005b80 2023-11-28T17:11:19.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff3467fc640 1 -- 172.21.15.116:0/2779724747 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33000e2c0 con 0x7ff3300069d0 2023-11-28T17:11:19.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff3467fc640 1 -- 172.21.15.116:0/2779724747 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33000e480 con 0x7ff330007390 2023-11-28T17:11:19.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff347fff640 1 --2- 172.21.15.116:0/2779724747 >> v2:172.21.15.156:3300/0 conn(0x7ff34815dab0 0x7ff3480703f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:11:19.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff3477fe640 1 --2- 172.21.15.116:0/2779724747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330005b80 0x7ff330005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:19.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff346ffd640 1 --2- 172.21.15.116:0/2779724747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3300069d0 0x7ff330006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:19.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff3477fe640 1 -- 172.21.15.116:0/2779724747 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff330007390 msgr2=0x7ff33000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:19.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff3477fe640 1 --2- 172.21.15.116:0/2779724747 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff330007390 0x7ff33000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff3477fe640 1 -- 172.21.15.116:0/2779724747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3300069d0 msgr2=0x7ff330006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:19.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff3477fe640 1 --2- 172.21.15.116:0/2779724747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3300069d0 0x7ff330006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff3477fe640 1 -- 172.21.15.116:0/2779724747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff330005730 con 0x7ff330005b80 2023-11-28T17:11:19.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.909+0000 7ff347fff640 1 -- 172.21.15.116:0/2779724747 reap_dead start 2023-11-28T17:11:19.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.910+0000 7ff3477fe640 1 --2- 172.21.15.116:0/2779724747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330005b80 0x7ff330005f80 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff338003d80 tx=0x7ff330007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec859596ba30199 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:19.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.911+0000 7ff3467fc640 1 -- 172.21.15.116:0/2779724747 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338032600 con 0x7ff330005b80 2023-11-28T17:11:19.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.911+0000 7ff3467fc640 1 -- 172.21.15.116:0/2779724747 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33802e050 con 0x7ff330005b80 2023-11-28T17:11:19.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.911+0000 7ff3467fc640 1 -- 172.21.15.116:0/2779724747 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3380329e0 con 0x7ff330005b80 2023-11-28T17:11:19.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.911+0000 7ff34e02d640 1 -- 172.21.15.116:0/2779724747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330005b80 msgr2=0x7ff330005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:19.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.911+0000 7ff34e02d640 1 --2- 172.21.15.116:0/2779724747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330005b80 0x7ff330005f80 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff338003d80 tx=0x7ff330007a00 comp rx=0 tx=0).stop 2023-11-28T17:11:19.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.911+0000 7ff34e02d640 1 -- 172.21.15.116:0/2779724747 shutdown_connections 2023-11-28T17:11:19.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.911+0000 7ff34e02d640 1 --2- 172.21.15.116:0/2779724747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff330005b80 0x7ff330005f80 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.911+0000 7ff34e02d640 1 -- 172.21.15.116:0/2779724747 >> 172.21.15.116:0/2779724747 conn(0x7ff34806a4c0 msgr2=0x7ff34806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:19.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.911+0000 7ff34e02d640 1 -- 172.21.15.116:0/2779724747 shutdown_connections 2023-11-28T17:11:19.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.911+0000 7ff34e02d640 1 -- 172.21.15.116:0/2779724747 wait complete. 2023-11-28T17:11:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.913+0000 7ff34e02d640 1 Processor -- start 2023-11-28T17:11:19.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.913+0000 7ff34e02d640 1 -- start start 2023-11-28T17:11:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.914+0000 7ff34e02d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff330005b80 0x7ff3480aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.914+0000 7ff34e02d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3480af1a0 0x7ff34813b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.914+0000 7ff34e02d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff34813b9b0 0x7ff34813be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:19.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.914+0000 7ff34e02d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480af600 con 0x7ff34813b9b0 2023-11-28T17:11:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.914+0000 7ff34e02d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34813b1f0 con 0x7ff330005b80 2023-11-28T17:11:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.914+0000 7ff34e02d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34813af10 con 0x7ff3480af1a0 2023-11-28T17:11:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.914+0000 7ff347fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff34813b9b0 0x7ff34813be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.914+0000 7ff347fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff34813b9b0 0x7ff34813be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33828/0 (socket says 172.21.15.116:33828) 2023-11-28T17:11:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.915+0000 7ff347fff640 1 -- 172.21.15.116:0/6825476 learned_addr learned my addr 172.21.15.116:0/6825476 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.915+0000 7ff3477fe640 1 --2- 172.21.15.116:0/6825476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff330005b80 0x7ff3480aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.915+0000 7ff346ffd640 1 --2- 172.21.15.116:0/6825476 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3480af1a0 0x7ff34813b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.915+0000 7ff347fff640 1 -- 172.21.15.116:0/6825476 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3480af1a0 msgr2=0x7ff34813b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.915+0000 7ff347fff640 1 --2- 172.21.15.116:0/6825476 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3480af1a0 0x7ff34813b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.915+0000 7ff347fff640 1 -- 172.21.15.116:0/6825476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff330005b80 msgr2=0x7ff3480aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.915+0000 7ff347fff640 1 --2- 172.21.15.116:0/6825476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff330005b80 0x7ff3480aec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.915+0000 7ff347fff640 1 -- 172.21.15.116:0/6825476 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3481e2fb0 con 0x7ff34813b9b0 2023-11-28T17:11:19.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.915+0000 7ff347fff640 1 --2- 172.21.15.116:0/6825476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff34813b9b0 0x7ff34813be30 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff348070240 tx=0x7ff34000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.915+0000 7ff32ffff640 1 -- 172.21.15.116:0/6825476 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340007e90 con 0x7ff34813b9b0 2023-11-28T17:11:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.916+0000 7ff32ffff640 1 -- 172.21.15.116:0/6825476 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34000f040 con 0x7ff34813b9b0 2023-11-28T17:11:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.916+0000 7ff34e02d640 1 -- 172.21.15.116:0/6825476 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3481e3290 con 0x7ff34813b9b0 2023-11-28T17:11:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.916+0000 7ff32ffff640 1 -- 172.21.15.116:0/6825476 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340003bb0 con 0x7ff34813b9b0 2023-11-28T17:11:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.916+0000 7ff34e02d640 1 -- 172.21.15.116:0/6825476 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3481399d0 con 0x7ff34813b9b0 2023-11-28T17:11:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.917+0000 7ff34e02d640 1 -- 172.21.15.116:0/6825476 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3481e37b0 con 0x7ff34813b9b0 2023-11-28T17:11:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.918+0000 7ff32ffff640 1 -- 172.21.15.116:0/6825476 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff340003d10 con 0x7ff34813b9b0 2023-11-28T17:11:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.919+0000 7ff32ffff640 1 --2- 172.21.15.116:0/6825476 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff33406b2a0 0x7ff33406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.919+0000 7ff32ffff640 1 -- 172.21.15.116:0/6825476 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff3400898b0 con 0x7ff34813b9b0 2023-11-28T17:11:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.919+0000 7ff3477fe640 1 --2- 172.21.15.116:0/6825476 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff33406b2a0 0x7ff33406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:19.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.920+0000 7ff3477fe640 1 --2- 172.21.15.116:0/6825476 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff33406b2a0 0x7ff33406d760 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff338034000 tx=0x7ff338009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:19.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:19.923+0000 7ff32ffff640 1 -- 172.21.15.116:0/6825476 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3481e37b0 con 0x7ff34813b9b0 2023-11-28T17:11:20.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.117+0000 7ff34e02d640 1 -- 172.21.15.116:0/6825476 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3481e37b0 con 0x7ff34813b9b0 2023-11-28T17:11:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.119+0000 7ff32ffff640 1 -- 172.21.15.116:0/6825476 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+12869 (secure 0 0 0) 0x7ff3481e37b0 con 0x7ff34813b9b0 2023-11-28T17:11:20.120 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:20.121 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":44,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:11:19.145214+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:11:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.122+0000 7ff34e02d640 1 -- 172.21.15.116:0/6825476 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff33406b2a0 msgr2=0x7ff33406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.122+0000 7ff34e02d640 1 --2- 172.21.15.116:0/6825476 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff33406b2a0 0x7ff33406d760 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff338034000 tx=0x7ff338009030 comp rx=0 tx=0).stop 2023-11-28T17:11:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.122+0000 7ff34e02d640 1 -- 172.21.15.116:0/6825476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff34813b9b0 msgr2=0x7ff34813be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.122+0000 7ff34e02d640 1 --2- 172.21.15.116:0/6825476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff34813b9b0 0x7ff34813be30 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff348070240 tx=0x7ff34000d4a0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.123+0000 7ff34e02d640 1 -- 172.21.15.116:0/6825476 shutdown_connections 2023-11-28T17:11:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.123+0000 7ff34e02d640 1 --2- 172.21.15.116:0/6825476 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff33406b2a0 0x7ff33406d760 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.123+0000 7ff34e02d640 1 --2- 172.21.15.116:0/6825476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff34813b9b0 0x7ff34813be30 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.123+0000 7ff34e02d640 1 --2- 172.21.15.116:0/6825476 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3480af1a0 0x7ff34813b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.123+0000 7ff34e02d640 1 --2- 172.21.15.116:0/6825476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff330005b80 0x7ff3480aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.123+0000 7ff34e02d640 1 -- 172.21.15.116:0/6825476 >> 172.21.15.116:0/6825476 conn(0x7ff34806a4c0 msgr2=0x7ff34814cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.123+0000 7ff34e02d640 1 -- 172.21.15.116:0/6825476 shutdown_connections 2023-11-28T17:11:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.123+0000 7ff34e02d640 1 -- 172.21.15.116:0/6825476 wait complete. 2023-11-28T17:11:20.144 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:11:20.144 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:11:20.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.360+0000 7f6fea101640 1 Processor -- start 2023-11-28T17:11:20.361 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.360+0000 7f6fea101640 1 -- start start 2023-11-28T17:11:20.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.362+0000 7f6fea101640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fe413cee0 0x7f6fe413d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:20.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.362+0000 7f6fea101640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe413d2c0 con 0x7f6fe4072430 2023-11-28T17:11:20.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.362+0000 7f6fea101640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe41400e0 con 0x7f6fe413f640 2023-11-28T17:11:20.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.362+0000 7f6fea101640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe41403c0 con 0x7f6fe413cee0 2023-11-28T17:11:20.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.362+0000 7f6fe3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6fe4072430 0x7f6fe40ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53102/0 (socket says 172.21.15.116:53102) 2023-11-28T17:11:20.363 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.362+0000 7f6fe3fff640 1 -- 172.21.15.116:0/3593189698 learned_addr learned my addr 172.21.15.116:0/3593189698 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:20.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.362+0000 7f6fe2ffd640 1 -- 172.21.15.116:0/3593189698 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455080509 0 0) 0x7f6fe413d2c0 con 0x7f6fe4072430 2023-11-28T17:11:20.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.362+0000 7f6fe2ffd640 1 -- 172.21.15.116:0/3593189698 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd0003520 con 0x7f6fe4072430 2023-11-28T17:11:20.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.362+0000 7f6fe37fe640 1 --2- 172.21.15.116:0/3593189698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fe413cee0 0x7f6fe413d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:20.365 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.363+0000 7f6fe2ffd640 1 -- 172.21.15.116:0/3593189698 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3795789170 0 0) 0x7f6fd0003520 con 0x7f6fe4072430 2023-11-28T17:11:20.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.363+0000 7f6fe2ffd640 1 -- 172.21.15.116:0/3593189698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fe413cee0 msgr2=0x7f6fe413d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:20.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.363+0000 7f6fe2ffd640 1 --2- 172.21.15.116:0/3593189698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fe413cee0 0x7f6fe413d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.363+0000 7f6fe2ffd640 1 -- 172.21.15.116:0/3593189698 >> v1:172.21.15.156:6789/0 conn(0x7f6fe413f640 legacy=0x7f6fe4071d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:20.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.363+0000 7f6fe2ffd640 1 -- 172.21.15.116:0/3593189698 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe40cdb50 con 0x7f6fe4072430 2023-11-28T17:11:20.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.363+0000 7f6fe2ffd640 1 -- 172.21.15.116:0/3593189698 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f6fdc0019f0 con 0x7f6fe4072430 2023-11-28T17:11:20.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.363+0000 7f6fe2ffd640 1 -- 172.21.15.116:0/3593189698 >> v1:172.21.15.116:6789/0 conn(0x7f6fe4072430 legacy=0x7f6fe40ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:20.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.363+0000 7f6fe2ffd640 1 --2- 172.21.15.116:0/3593189698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd00039a0 0x7f6fd0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:20.366 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.363+0000 7f6fe2ffd640 1 --2- 172.21.15.116:0/3593189698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fd00047c0 0x7f6fd0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:20.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.363+0000 7f6fe2ffd640 1 --2- 172.21.15.116:0/3593189698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fe413cee0 0x7f6fd000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:20.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.363+0000 7f6fe2ffd640 1 -- 172.21.15.116:0/3593189698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe41403c0 con 0x7f6fd00039a0 2023-11-28T17:11:20.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.363+0000 7f6fe2ffd640 1 -- 172.21.15.116:0/3593189698 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00055c0 con 0x7f6fd00047c0 2023-11-28T17:11:20.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.363+0000 7f6fe2ffd640 1 -- 172.21.15.116:0/3593189698 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd000e0d0 con 0x7f6fe413cee0 2023-11-28T17:11:20.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.363+0000 7f6fe37fe640 1 --2- 172.21.15.116:0/3593189698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd00039a0 0x7f6fd0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:20.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.364+0000 7f6fe37fe640 1 -- 172.21.15.116:0/3593189698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fe413cee0 msgr2=0x7f6fd000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:20.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.364+0000 7f6fe37fe640 1 --2- 172.21.15.116:0/3593189698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fe413cee0 0x7f6fd000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.364+0000 7f6fe37fe640 1 -- 172.21.15.116:0/3593189698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fd00047c0 msgr2=0x7f6fd0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:20.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.364+0000 7f6fdbfff640 1 --2- 172.21.15.116:0/3593189698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fd00047c0 0x7f6fd0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:20.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.364+0000 7f6fe37fe640 1 --2- 172.21.15.116:0/3593189698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fd00047c0 0x7f6fd0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.364+0000 7f6fe37fe640 1 -- 172.21.15.116:0/3593189698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd0003520 con 0x7f6fd00039a0 2023-11-28T17:11:20.367 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.364+0000 7f6fdbfff640 1 --2- 172.21.15.116:0/3593189698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fd00047c0 0x7f6fd0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:20.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.365+0000 7f6fe37fe640 1 --2- 172.21.15.116:0/3593189698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd00039a0 0x7f6fd0003dc0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400ba30 tx=0x7f6fd00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0794cc8f5857e7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:20.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.365+0000 7f6fe2ffd640 1 -- 172.21.15.116:0/3593189698 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4032680 con 0x7f6fd00039a0 2023-11-28T17:11:20.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.365+0000 7f6fe2ffd640 1 -- 172.21.15.116:0/3593189698 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd40327e0 con 0x7f6fd00039a0 2023-11-28T17:11:20.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.365+0000 7f6fe2ffd640 1 -- 172.21.15.116:0/3593189698 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4032b70 con 0x7f6fd00039a0 2023-11-28T17:11:20.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.365+0000 7f6fea101640 1 -- 172.21.15.116:0/3593189698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd00039a0 msgr2=0x7f6fd0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:20.369 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.365+0000 7f6fea101640 1 --2- 172.21.15.116:0/3593189698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd00039a0 0x7f6fd0003dc0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400ba30 tx=0x7f6fd00057f0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.365+0000 7f6fe3fff640 1 -- 172.21.15.116:0/3593189698 reap_dead start 2023-11-28T17:11:20.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.365+0000 7f6fea101640 1 -- 172.21.15.116:0/3593189698 shutdown_connections 2023-11-28T17:11:20.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.366+0000 7f6fea101640 1 -- 172.21.15.116:0/3593189698 >> 172.21.15.116:0/3593189698 conn(0x7f6fe40b0cf0 msgr2=0x7f6fe40b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:20.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.366+0000 7f6fea101640 1 -- 172.21.15.116:0/3593189698 shutdown_connections 2023-11-28T17:11:20.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.366+0000 7f6fea101640 1 -- 172.21.15.116:0/3593189698 wait complete. 2023-11-28T17:11:20.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.367+0000 7f6fea101640 1 Processor -- start 2023-11-28T17:11:20.370 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.367+0000 7f6fea101640 1 -- start start 2023-11-28T17:11:20.371 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.368+0000 7f6fea101640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd00039a0 0x7f6fe40b27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:20.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.368+0000 7f6fea101640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fe412b860 0x7f6fe413b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:20.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.368+0000 7f6fea101640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fe40b48c0 0x7f6fe413b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:20.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.368+0000 7f6fea101640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe41417f0 con 0x7f6fd00039a0 2023-11-28T17:11:20.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.368+0000 7f6fea101640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4141450 con 0x7f6fe40b48c0 2023-11-28T17:11:20.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.368+0000 7f6fea101640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe41f2ca0 con 0x7f6fe412b860 2023-11-28T17:11:20.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.369+0000 7f6fe37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd00039a0 0x7f6fe40b27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:20.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.369+0000 7f6fe37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd00039a0 0x7f6fe40b27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33846/0 (socket says 172.21.15.116:33846) 2023-11-28T17:11:20.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.369+0000 7f6fe37fe640 1 -- 172.21.15.116:0/1091178916 learned_addr learned my addr 172.21.15.116:0/1091178916 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:20.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.369+0000 7f6fe37fe640 1 -- 172.21.15.116:0/1091178916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fe412b860 msgr2=0x7f6fe413b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:20.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.369+0000 7f6fe37fe640 1 --2- 172.21.15.116:0/1091178916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fe412b860 0x7f6fe413b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.369+0000 7f6fe37fe640 1 -- 172.21.15.116:0/1091178916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fe40b48c0 msgr2=0x7f6fe413b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:20.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.369+0000 7f6fe37fe640 1 --2- 172.21.15.116:0/1091178916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fe40b48c0 0x7f6fe413b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.369+0000 7f6fe37fe640 1 -- 172.21.15.116:0/1091178916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe40b2fd0 con 0x7f6fd00039a0 2023-11-28T17:11:20.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.369+0000 7f6fe37fe640 1 --2- 172.21.15.116:0/1091178916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd00039a0 0x7f6fe40b27a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6fd402bf00 tx=0x7f6fd4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:20.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.369+0000 7f6fe0ff9640 1 -- 172.21.15.116:0/1091178916 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4004220 con 0x7f6fd00039a0 2023-11-28T17:11:20.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.369+0000 7f6fe0ff9640 1 -- 172.21.15.116:0/1091178916 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd40043a0 con 0x7f6fd00039a0 2023-11-28T17:11:20.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.369+0000 7f6fe0ff9640 1 -- 172.21.15.116:0/1091178916 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4030ae0 con 0x7f6fd00039a0 2023-11-28T17:11:20.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.369+0000 7f6fea101640 1 -- 172.21.15.116:0/1091178916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe40b32b0 con 0x7f6fd00039a0 2023-11-28T17:11:20.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.370+0000 7f6fea101640 1 -- 172.21.15.116:0/1091178916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe40b2ce0 con 0x7f6fd00039a0 2023-11-28T17:11:20.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.371+0000 7f6fea101640 1 -- 172.21.15.116:0/1091178916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe40b37d0 con 0x7f6fd00039a0 2023-11-28T17:11:20.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.371+0000 7f6fe0ff9640 1 -- 172.21.15.116:0/1091178916 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f6fd4030c40 con 0x7f6fd00039a0 2023-11-28T17:11:20.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.373+0000 7f6fe0ff9640 1 --2- 172.21.15.116:0/1091178916 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6fc406b2a0 0x7f6fc406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:20.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.373+0000 7f6fe0ff9640 1 -- 172.21.15.116:0/1091178916 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6fd40ac2b0 con 0x7f6fd00039a0 2023-11-28T17:11:20.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.373+0000 7f6fdbfff640 1 --2- 172.21.15.116:0/1091178916 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6fc406b2a0 0x7f6fc406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:20.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.374+0000 7f6fdbfff640 1 --2- 172.21.15.116:0/1091178916 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6fc406b2a0 0x7f6fc406d760 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4143c10 tx=0x7f6fcc009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:20.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.377+0000 7f6fe0ff9640 1 -- 172.21.15.116:0/1091178916 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6fe40b37d0 con 0x7f6fd00039a0 2023-11-28T17:11:20.566 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.565+0000 7f6fea101640 1 -- 172.21.15.116:0/1091178916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6fe40b37d0 con 0x7f6fd00039a0 2023-11-28T17:11:20.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.566+0000 7f6fe0ff9640 1 -- 172.21.15.116:0/1091178916 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6fe40b37d0 con 0x7f6fd00039a0 2023-11-28T17:11:20.568 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:11:20.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.568+0000 7f6fea101640 1 -- 172.21.15.116:0/1091178916 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6fc406b2a0 msgr2=0x7f6fc406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:20.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.568+0000 7f6fea101640 1 --2- 172.21.15.116:0/1091178916 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6fc406b2a0 0x7f6fc406d760 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4143c10 tx=0x7f6fcc009040 comp rx=0 tx=0).stop 2023-11-28T17:11:20.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.569+0000 7f6fea101640 1 -- 172.21.15.116:0/1091178916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd00039a0 msgr2=0x7f6fe40b27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:20.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.569+0000 7f6fea101640 1 --2- 172.21.15.116:0/1091178916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd00039a0 0x7f6fe40b27a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6fd402bf00 tx=0x7f6fd4004730 comp rx=0 tx=0).stop 2023-11-28T17:11:20.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.569+0000 7f6fea101640 1 -- 172.21.15.116:0/1091178916 shutdown_connections 2023-11-28T17:11:20.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.569+0000 7f6fea101640 1 --2- 172.21.15.116:0/1091178916 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f6fc406b2a0 0x7f6fc406d760 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.569+0000 7f6fea101640 1 --2- 172.21.15.116:0/1091178916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fe40b48c0 0x7f6fe413b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.569+0000 7f6fea101640 1 --2- 172.21.15.116:0/1091178916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fe412b860 0x7f6fe413b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.569+0000 7f6fea101640 1 --2- 172.21.15.116:0/1091178916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fd00039a0 0x7f6fe40b27a0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.569+0000 7f6fea101640 1 -- 172.21.15.116:0/1091178916 >> 172.21.15.116:0/1091178916 conn(0x7f6fe40b0cf0 msgr2=0x7f6fe40ce250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:20.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.569+0000 7f6fea101640 1 -- 172.21.15.116:0/1091178916 shutdown_connections 2023-11-28T17:11:20.572 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.569+0000 7f6fea101640 1 -- 172.21.15.116:0/1091178916 wait complete. 2023-11-28T17:11:20.590 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-28T17:11:20.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.800+0000 7f5f9e461640 1 Processor -- start 2023-11-28T17:11:20.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.800+0000 7f5f9e461640 1 -- start start 2023-11-28T17:11:20.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.802+0000 7f5f9e461640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5f98144430 0x7f5f9815f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:20.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.802+0000 7f5f9e461640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f980703f0 0x7f5f98071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:20.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.802+0000 7f5f9e461640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f98070bd0 con 0x7f5f980721b0 2023-11-28T17:11:20.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.802+0000 7f5f9e461640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f98144810 con 0x7f5f980703f0 2023-11-28T17:11:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.802+0000 7f5f9e461640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f98070eb0 con 0x7f5f98144430 2023-11-28T17:11:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.802+0000 7f5f977fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f980703f0 0x7f5f98071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.802+0000 7f5f977fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f980703f0 0x7f5f98071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33856/0 (socket says 172.21.15.116:33856) 2023-11-28T17:11:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.802+0000 7f5f977fe640 1 -- 172.21.15.116:0/1625355673 learned_addr learned my addr 172.21.15.116:0/1625355673 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.802+0000 7f5f97fff640 1 --2- 172.21.15.116:0/1625355673 >> v2:172.21.15.156:3300/0 conn(0x7f5f98144430 0x7f5f9815f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.803+0000 7f5f977fe640 1 -- 172.21.15.116:0/1625355673 >> v2:172.21.15.156:3300/0 conn(0x7f5f98144430 msgr2=0x7f5f9815f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.803+0000 7f5f977fe640 1 --2- 172.21.15.116:0/1625355673 >> v2:172.21.15.156:3300/0 conn(0x7f5f98144430 0x7f5f9815f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.803+0000 7f5f977fe640 1 -- 172.21.15.116:0/1625355673 >> v1:172.21.15.156:6789/0 conn(0x7f5f980721b0 legacy=0x7f5f9806c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.803+0000 7f5f977fe640 1 -- 172.21.15.116:0/1625355673 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f98071190 con 0x7f5f980703f0 2023-11-28T17:11:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.803+0000 7f5f97fff640 1 --2- 172.21.15.116:0/1625355673 >> v2:172.21.15.156:3300/0 conn(0x7f5f98144430 0x7f5f9815f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:11:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.804+0000 7f5f977fe640 1 --2- 172.21.15.116:0/1625355673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f980703f0 0x7f5f98071c70 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f5f9806e850 tx=0x7f5f8c02b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83a6cfc06ab2e98 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:20.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.805+0000 7f5f96ffd640 1 -- 172.21.15.116:0/1625355673 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c030880 con 0x7f5f980703f0 2023-11-28T17:11:20.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.805+0000 7f5f96ffd640 1 -- 172.21.15.116:0/1625355673 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8c0309e0 con 0x7f5f980703f0 2023-11-28T17:11:20.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.805+0000 7f5f96ffd640 1 -- 172.21.15.116:0/1625355673 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c030d70 con 0x7f5f980703f0 2023-11-28T17:11:20.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.805+0000 7f5f9e461640 1 -- 172.21.15.116:0/1625355673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f980703f0 msgr2=0x7f5f98071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.805+0000 7f5f9e461640 1 --2- 172.21.15.116:0/1625355673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f980703f0 0x7f5f98071c70 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f5f9806e850 tx=0x7f5f8c02b9c0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.805+0000 7f5f9e461640 1 -- 172.21.15.116:0/1625355673 shutdown_connections 2023-11-28T17:11:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.805+0000 7f5f9e461640 1 --2- 172.21.15.116:0/1625355673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f980703f0 0x7f5f98071c70 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.805+0000 7f5f9e461640 1 --2- 172.21.15.116:0/1625355673 >> v2:172.21.15.156:3300/0 conn(0x7f5f98144430 0x7f5f9815f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.805+0000 7f5f9e461640 1 -- 172.21.15.116:0/1625355673 >> 172.21.15.116:0/1625355673 conn(0x7f5f9806a4c0 msgr2=0x7f5f9806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:20.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.805+0000 7f5f9e461640 1 -- 172.21.15.116:0/1625355673 shutdown_connections 2023-11-28T17:11:20.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.805+0000 7f5f9e461640 1 -- 172.21.15.116:0/1625355673 wait complete. 2023-11-28T17:11:20.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.807+0000 7f5f9e461640 1 Processor -- start 2023-11-28T17:11:20.812 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.807+0000 7f5f9e461640 1 -- start start 2023-11-28T17:11:20.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.808+0000 7f5f9e461640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5f980703f0 0x7f5f980c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:20.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.808+0000 7f5f9e461640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f980721b0 0x7f5f980ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:20.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.809+0000 7f5f9e461640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f98144430 0x7f5f98140d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:20.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.809+0000 7f5f9e461640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9813ca80 con 0x7f5f98144430 2023-11-28T17:11:20.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.809+0000 7f5f9e461640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9813c7a0 con 0x7f5f980721b0 2023-11-28T17:11:20.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.809+0000 7f5f9e461640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9813bc40 con 0x7f5f980703f0 2023-11-28T17:11:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.809+0000 7f5f9c9fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f98144430 0x7f5f98140d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.809+0000 7f5f9c9fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f98144430 0x7f5f98140d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33870/0 (socket says 172.21.15.116:33870) 2023-11-28T17:11:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.809+0000 7f5f9c9fe640 1 -- 172.21.15.116:0/1432439304 learned_addr learned my addr 172.21.15.116:0/1432439304 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.809+0000 7f5f9c9fe640 1 -- 172.21.15.116:0/1432439304 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5f980703f0 msgr2=0x7f5f980c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.809+0000 7f5f9c9fe640 1 --2- 172.21.15.116:0/1432439304 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5f980703f0 0x7f5f980c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.809+0000 7f5f9c9fe640 1 -- 172.21.15.116:0/1432439304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f980721b0 msgr2=0x7f5f980ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.809+0000 7f5f977fe640 1 --2- 172.21.15.116:0/1432439304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f980721b0 0x7f5f980ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.809+0000 7f5f9c9fe640 1 --2- 172.21.15.116:0/1432439304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f980721b0 0x7f5f980ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:20.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.809+0000 7f5f9c9fe640 1 -- 172.21.15.116:0/1432439304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c0094d0 con 0x7f5f98144430 2023-11-28T17:11:20.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.810+0000 7f5f9c9fe640 1 --2- 172.21.15.116:0/1432439304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f98144430 0x7f5f98140d40 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000c730 tx=0x7f5f9000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:20.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.810+0000 7f5f94ff9640 1 -- 172.21.15.116:0/1432439304 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90003b70 con 0x7f5f98144430 2023-11-28T17:11:20.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.810+0000 7f5f94ff9640 1 -- 172.21.15.116:0/1432439304 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f90003cd0 con 0x7f5f98144430 2023-11-28T17:11:20.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.810+0000 7f5f94ff9640 1 -- 172.21.15.116:0/1432439304 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9000aa90 con 0x7f5f98144430 2023-11-28T17:11:20.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.810+0000 7f5f9e461640 1 -- 172.21.15.116:0/1432439304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9813bf20 con 0x7f5f98144430 2023-11-28T17:11:20.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.810+0000 7f5f9e461640 1 -- 172.21.15.116:0/1432439304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9815a960 con 0x7f5f98144430 2023-11-28T17:11:20.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.811+0000 7f5f9e461640 1 -- 172.21.15.116:0/1432439304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9813baa0 con 0x7f5f98144430 2023-11-28T17:11:20.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.812+0000 7f5f94ff9640 1 -- 172.21.15.116:0/1432439304 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f5f9000abf0 con 0x7f5f98144430 2023-11-28T17:11:20.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.813+0000 7f5f94ff9640 1 --2- 172.21.15.116:0/1432439304 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5f8406b2d0 0x7f5f8406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:20.817 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.813+0000 7f5f94ff9640 1 -- 172.21.15.116:0/1432439304 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5f900896b0 con 0x7f5f98144430 2023-11-28T17:11:20.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.813+0000 7f5f97fff640 1 --2- 172.21.15.116:0/1432439304 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5f8406b2d0 0x7f5f8406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:20.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.814+0000 7f5f97fff640 1 --2- 172.21.15.116:0/1432439304 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5f8406b2d0 0x7f5f8406d790 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5f88006e80 tx=0x7f5f88008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:20.818 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:20.817+0000 7f5f94ff9640 1 -- 172.21.15.116:0/1432439304 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f9813baa0 con 0x7f5f98144430 2023-11-28T17:11:21.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.009+0000 7f5f9e461640 1 -- 172.21.15.116:0/1432439304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5f9813baa0 con 0x7f5f98144430 2023-11-28T17:11:21.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.010+0000 7f5f94ff9640 1 -- 172.21.15.116:0/1432439304 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v44) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5f9813baa0 con 0x7f5f98144430 2023-11-28T17:11:21.012 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:21.012 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2023-11-28T17:11:21.012 INFO:teuthology.orchestra.run.smithi116.stdout: { 2023-11-28T17:11:21.013 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 1, 2023-11-28T17:11:21.013 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": ".mgr" 2023-11-28T17:11:21.013 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:21.014 INFO:teuthology.orchestra.run.smithi116.stdout:] 2023-11-28T17:11:21.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.013+0000 7f5f9e461640 1 -- 172.21.15.116:0/1432439304 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5f8406b2d0 msgr2=0x7f5f8406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:21.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.013+0000 7f5f9e461640 1 --2- 172.21.15.116:0/1432439304 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5f8406b2d0 0x7f5f8406d790 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5f88006e80 tx=0x7f5f88008040 comp rx=0 tx=0).stop 2023-11-28T17:11:21.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.013+0000 7f5f9e461640 1 -- 172.21.15.116:0/1432439304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f98144430 msgr2=0x7f5f98140d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:21.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.013+0000 7f5f9e461640 1 --2- 172.21.15.116:0/1432439304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f98144430 0x7f5f98140d40 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000c730 tx=0x7f5f9000cbf0 comp rx=0 tx=0).stop 2023-11-28T17:11:21.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.013+0000 7f5f9e461640 1 -- 172.21.15.116:0/1432439304 shutdown_connections 2023-11-28T17:11:21.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.013+0000 7f5f9e461640 1 --2- 172.21.15.116:0/1432439304 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5f8406b2d0 0x7f5f8406d790 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:21.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.013+0000 7f5f9e461640 1 --2- 172.21.15.116:0/1432439304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f98144430 0x7f5f98140d40 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:21.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.013+0000 7f5f9e461640 1 --2- 172.21.15.116:0/1432439304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5f980721b0 0x7f5f980ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:21.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.013+0000 7f5f9e461640 1 --2- 172.21.15.116:0/1432439304 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5f980703f0 0x7f5f980c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:21.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.013+0000 7f5f9e461640 1 -- 172.21.15.116:0/1432439304 >> 172.21.15.116:0/1432439304 conn(0x7f5f9806a4c0 msgr2=0x7f5f9815b470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.013+0000 7f5f9e461640 1 -- 172.21.15.116:0/1432439304 shutdown_connections 2023-11-28T17:11:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.013+0000 7f5f9e461640 1 -- 172.21.15.116:0/1432439304 wait complete. 2023-11-28T17:11:21.034 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-28T17:11:21.034 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-28T17:11:21.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.250+0000 7f44fe0a6640 1 Processor -- start 2023-11-28T17:11:21.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.251+0000 7f44fe0a6640 1 -- start start 2023-11-28T17:11:21.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.252+0000 7f44fe0a6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44f814a740 0x7f44f80cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:21.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.252+0000 7f44fe0a6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f8070420 con 0x7f44f814ac30 2023-11-28T17:11:21.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.253+0000 7f44fe0a6640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f80b2fc0 con 0x7f44f80b4600 2023-11-28T17:11:21.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.253+0000 7f44fe0a6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f8070aa0 con 0x7f44f814a740 2023-11-28T17:11:21.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.253+0000 7f44eeffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f44f814ac30 0x7f44f80b3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53108/0 (socket says 172.21.15.116:53108) 2023-11-28T17:11:21.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.253+0000 7f44eeffd640 1 -- 172.21.15.116:0/3535889150 learned_addr learned my addr 172.21.15.116:0/3535889150 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:21.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.253+0000 7f44f6ffd640 1 -- 172.21.15.116:0/3535889150 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197527296 0 0) 0x7f44f8070420 con 0x7f44f814ac30 2023-11-28T17:11:21.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.253+0000 7f44f6ffd640 1 -- 172.21.15.116:0/3535889150 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44e4003520 con 0x7f44f814ac30 2023-11-28T17:11:21.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.253+0000 7f44f77fe640 1 --2- 172.21.15.116:0/3535889150 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44f814a740 0x7f44f80cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:21.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f6ffd640 1 -- 172.21.15.116:0/3535889150 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2931629015 0 0) 0x7f44e4003520 con 0x7f44f814ac30 2023-11-28T17:11:21.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f6ffd640 1 -- 172.21.15.116:0/3535889150 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44f814a740 msgr2=0x7f44f80cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:21.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f6ffd640 1 --2- 172.21.15.116:0/3535889150 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44f814a740 0x7f44f80cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:21.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f6ffd640 1 -- 172.21.15.116:0/3535889150 >> v1:172.21.15.156:6789/0 conn(0x7f44f80b4600 legacy=0x7f44f8070ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:21.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f6ffd640 1 -- 172.21.15.116:0/3535889150 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f80706c0 con 0x7f44f814ac30 2023-11-28T17:11:21.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f6ffd640 1 -- 172.21.15.116:0/3535889150 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f44e0001670 con 0x7f44f814ac30 2023-11-28T17:11:21.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f6ffd640 1 -- 172.21.15.116:0/3535889150 >> v1:172.21.15.116:6789/0 conn(0x7f44f814ac30 legacy=0x7f44f80b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:21.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f6ffd640 1 --2- 172.21.15.116:0/3535889150 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44f814a740 0x7f44e40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:21.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f6ffd640 1 --2- 172.21.15.116:0/3535889150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44e4004020 0x7f44e40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:21.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f6ffd640 1 --2- 172.21.15.116:0/3535889150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e4004d00 0x7f44e4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:21.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f6ffd640 1 -- 172.21.15.116:0/3535889150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f8070aa0 con 0x7f44e4004d00 2023-11-28T17:11:21.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f6ffd640 1 -- 172.21.15.116:0/3535889150 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e400bbc0 con 0x7f44e4004020 2023-11-28T17:11:21.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f6ffd640 1 -- 172.21.15.116:0/3535889150 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e400bd80 con 0x7f44f814a740 2023-11-28T17:11:21.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f7fff640 1 --2- 172.21.15.116:0/3535889150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e4004d00 0x7f44e4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:21.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44f77fe640 1 --2- 172.21.15.116:0/3535889150 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44f814a740 0x7f44e40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:21.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.254+0000 7f44eeffd640 1 --2- 172.21.15.116:0/3535889150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44e4004020 0x7f44e40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:21.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.255+0000 7f44f7fff640 1 -- 172.21.15.116:0/3535889150 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44f814a740 msgr2=0x7f44e40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:21.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.255+0000 7f44f7fff640 1 --2- 172.21.15.116:0/3535889150 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44f814a740 0x7f44e40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:21.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.255+0000 7f44f7fff640 1 -- 172.21.15.116:0/3535889150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44e4004020 msgr2=0x7f44e40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:21.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.255+0000 7f44f7fff640 1 --2- 172.21.15.116:0/3535889150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44e4004020 0x7f44e40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:21.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.255+0000 7f44f7fff640 1 -- 172.21.15.116:0/3535889150 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e4003520 con 0x7f44e4004d00 2023-11-28T17:11:21.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.255+0000 7f44f77fe640 1 --2- 172.21.15.116:0/3535889150 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44f814a740 0x7f44e40039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:21.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.256+0000 7f44f7fff640 1 --2- 172.21.15.116:0/3535889150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e4004d00 0x7f44e4005180 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f44f0001e50 tx=0x7f44e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40989d4cc4c4a60 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:21.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.256+0000 7f44f6ffd640 1 -- 172.21.15.116:0/3535889150 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f0003b70 con 0x7f44e4004d00 2023-11-28T17:11:21.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.256+0000 7f44f6ffd640 1 -- 172.21.15.116:0/3535889150 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f0003cd0 con 0x7f44e4004d00 2023-11-28T17:11:21.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.256+0000 7f44f6ffd640 1 -- 172.21.15.116:0/3535889150 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f0005310 con 0x7f44e4004d00 2023-11-28T17:11:21.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.256+0000 7f44fe0a6640 1 -- 172.21.15.116:0/3535889150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e4004d00 msgr2=0x7f44e4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:21.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.256+0000 7f44fe0a6640 1 --2- 172.21.15.116:0/3535889150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e4004d00 0x7f44e4005180 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f44f0001e50 tx=0x7f44e40057f0 comp rx=0 tx=0).stop 2023-11-28T17:11:21.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.256+0000 7f44f7fff640 1 -- 172.21.15.116:0/3535889150 reap_dead start 2023-11-28T17:11:21.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.257+0000 7f44fe0a6640 1 -- 172.21.15.116:0/3535889150 shutdown_connections 2023-11-28T17:11:21.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.257+0000 7f44fe0a6640 1 -- 172.21.15.116:0/3535889150 >> 172.21.15.116:0/3535889150 conn(0x7f44f80b0cf0 msgr2=0x7f44f80b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:21.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.257+0000 7f44fe0a6640 1 -- 172.21.15.116:0/3535889150 shutdown_connections 2023-11-28T17:11:21.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.257+0000 7f44fe0a6640 1 -- 172.21.15.116:0/3535889150 wait complete. 2023-11-28T17:11:21.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.258+0000 7f44fe0a6640 1 Processor -- start 2023-11-28T17:11:21.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.259+0000 7f44fe0a6640 1 -- start start 2023-11-28T17:11:21.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.259+0000 7f44fe0a6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e4004d00 0x7f44f80ae410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:21.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.259+0000 7f44fe0a6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44f8164b10 0x7f44f8164f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:21.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.259+0000 7f44fe0a6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44f8143c10 0x7f44f8144090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:21.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.259+0000 7f44fe0a6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f80ae950 con 0x7f44e4004d00 2023-11-28T17:11:21.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.259+0000 7f44fe0a6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f8145590 con 0x7f44f8143c10 2023-11-28T17:11:21.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.259+0000 7f44fe0a6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f81654d0 con 0x7f44f8164b10 2023-11-28T17:11:21.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.260+0000 7f44f77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e4004d00 0x7f44f80ae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:21.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.260+0000 7f44f77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e4004d00 0x7f44f80ae410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33900/0 (socket says 172.21.15.116:33900) 2023-11-28T17:11:21.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.260+0000 7f44f77fe640 1 -- 172.21.15.116:0/878859586 learned_addr learned my addr 172.21.15.116:0/878859586 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:21.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.260+0000 7f44f7fff640 1 --2- 172.21.15.116:0/878859586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44f8143c10 0x7f44f8144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:21.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.260+0000 7f44f77fe640 1 -- 172.21.15.116:0/878859586 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44f8164b10 msgr2=0x7f44f8164f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:21.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.260+0000 7f44f77fe640 1 --2- 172.21.15.116:0/878859586 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44f8164b10 0x7f44f8164f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:21.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.260+0000 7f44f77fe640 1 -- 172.21.15.116:0/878859586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44f8143c10 msgr2=0x7f44f8144090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:21.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.260+0000 7f44f77fe640 1 --2- 172.21.15.116:0/878859586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44f8143c10 0x7f44f8144090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:21.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.260+0000 7f44f77fe640 1 -- 172.21.15.116:0/878859586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f8144980 con 0x7f44e4004d00 2023-11-28T17:11:21.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.260+0000 7f44f77fe640 1 --2- 172.21.15.116:0/878859586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e4004d00 0x7f44f80ae410 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f44e800e810 tx=0x7f44e800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:21.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.261+0000 7f44f4ff9640 1 -- 172.21.15.116:0/878859586 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e800c670 con 0x7f44e4004d00 2023-11-28T17:11:21.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.261+0000 7f44f4ff9640 1 -- 172.21.15.116:0/878859586 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e800c7d0 con 0x7f44e4004d00 2023-11-28T17:11:21.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.261+0000 7f44f4ff9640 1 -- 172.21.15.116:0/878859586 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e800cb40 con 0x7f44e4004d00 2023-11-28T17:11:21.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.261+0000 7f44fe0a6640 1 -- 172.21.15.116:0/878859586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f8144c60 con 0x7f44e4004d00 2023-11-28T17:11:21.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.261+0000 7f44fe0a6640 1 -- 172.21.15.116:0/878859586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f8145220 con 0x7f44e4004d00 2023-11-28T17:11:21.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.262+0000 7f44fe0a6640 1 -- 172.21.15.116:0/878859586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f80cbe70 con 0x7f44e4004d00 2023-11-28T17:11:21.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.263+0000 7f44f4ff9640 1 -- 172.21.15.116:0/878859586 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f44e800cca0 con 0x7f44e4004d00 2023-11-28T17:11:21.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.264+0000 7f44f4ff9640 1 --2- 172.21.15.116:0/878859586 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f44d806b2a0 0x7f44d806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:21.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.264+0000 7f44f4ff9640 1 -- 172.21.15.116:0/878859586 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f44e8089640 con 0x7f44e4004d00 2023-11-28T17:11:21.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.265+0000 7f44eeffd640 1 --2- 172.21.15.116:0/878859586 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f44d806b2a0 0x7f44d806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:21.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.265+0000 7f44eeffd640 1 --2- 172.21.15.116:0/878859586 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f44d806b2a0 0x7f44d806d760 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f44e0001eb0 tx=0x7f44e0000f90 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:21.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.269+0000 7f44f4ff9640 1 -- 172.21.15.116:0/878859586 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44e8054760 con 0x7f44e4004d00 2023-11-28T17:11:21.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:21.462+0000 7f44fe0a6640 1 -- 172.21.15.116:0/878859586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f44f8069360 con 0x7f44e4004d00 2023-11-28T17:11:22.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2023-11-28T17:11:22.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.172+0000 7f44f4ff9640 1 -- 172.21.15.116:0/878859586 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v45) v1 ==== 138+0+0 (secure 0 0 0) 0x7f44e8057d80 con 0x7f44e4004d00 2023-11-28T17:11:22.175 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' created 2023-11-28T17:11:22.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.175+0000 7f44fe0a6640 1 -- 172.21.15.116:0/878859586 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f44d806b2a0 msgr2=0x7f44d806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:22.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.175+0000 7f44fe0a6640 1 --2- 172.21.15.116:0/878859586 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f44d806b2a0 0x7f44d806d760 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f44e0001eb0 tx=0x7f44e0000f90 comp rx=0 tx=0).stop 2023-11-28T17:11:22.178 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.175+0000 7f44fe0a6640 1 -- 172.21.15.116:0/878859586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e4004d00 msgr2=0x7f44f80ae410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:22.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.175+0000 7f44fe0a6640 1 --2- 172.21.15.116:0/878859586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e4004d00 0x7f44f80ae410 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f44e800e810 tx=0x7f44e800ecd0 comp rx=0 tx=0).stop 2023-11-28T17:11:22.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.175+0000 7f44fe0a6640 1 -- 172.21.15.116:0/878859586 shutdown_connections 2023-11-28T17:11:22.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.175+0000 7f44fe0a6640 1 --2- 172.21.15.116:0/878859586 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f44d806b2a0 0x7f44d806d760 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:22.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.175+0000 7f44fe0a6640 1 --2- 172.21.15.116:0/878859586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44f8143c10 0x7f44f8144090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:22.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.175+0000 7f44fe0a6640 1 --2- 172.21.15.116:0/878859586 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44f8164b10 0x7f44f8164f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:22.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.175+0000 7f44fe0a6640 1 --2- 172.21.15.116:0/878859586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44e4004d00 0x7f44f80ae410 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:22.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.175+0000 7f44fe0a6640 1 -- 172.21.15.116:0/878859586 >> 172.21.15.116:0/878859586 conn(0x7f44f80b0cf0 msgr2=0x7f44f80cc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:22.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.176+0000 7f44fe0a6640 1 -- 172.21.15.116:0/878859586 shutdown_connections 2023-11-28T17:11:22.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.176+0000 7f44fe0a6640 1 -- 172.21.15.116:0/878859586 wait complete. 2023-11-28T17:11:22.205 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-28T17:11:22.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.411+0000 7f23eea85640 1 Processor -- start 2023-11-28T17:11:22.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.411+0000 7f23eea85640 1 -- start start 2023-11-28T17:11:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.413+0000 7f23eea85640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f23e8143cc0 0x7f23e8144380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.413+0000 7f23eea85640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23e80b48a0 0x7f23e815b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.413+0000 7f23eea85640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e8071390 con 0x7f23e81448c0 2023-11-28T17:11:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.413+0000 7f23eea85640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e81440a0 con 0x7f23e8143cc0 2023-11-28T17:11:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.413+0000 7f23eea85640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e8071630 con 0x7f23e80b48a0 2023-11-28T17:11:22.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.413+0000 7f23ed282640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f23e81448c0 0x7f23e80b4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53110/0 (socket says 172.21.15.116:53110) 2023-11-28T17:11:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.413+0000 7f23ed282640 1 -- 172.21.15.116:0/292992033 learned_addr learned my addr 172.21.15.116:0/292992033 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.413+0000 7f23eda83640 1 --2- 172.21.15.116:0/292992033 >> v2:172.21.15.116:3300/0 conn(0x7f23e8143cc0 0x7f23e8144380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.413+0000 7f23ee284640 1 --2- 172.21.15.116:0/292992033 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23e80b48a0 0x7f23e815b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.414+0000 7f23eda83640 1 -- 172.21.15.116:0/292992033 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23e80b48a0 msgr2=0x7f23e815b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:22.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.414+0000 7f23eda83640 1 --2- 172.21.15.116:0/292992033 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23e80b48a0 0x7f23e815b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:22.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.414+0000 7f23eda83640 1 -- 172.21.15.116:0/292992033 >> v1:172.21.15.116:6789/0 conn(0x7f23e81448c0 legacy=0x7f23e80b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:22.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.414+0000 7f23eca81640 1 -- 172.21.15.116:0/292992033 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856620773 0 0) 0x7f23e8071390 con 0x7f23e81448c0 2023-11-28T17:11:22.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.414+0000 7f23eda83640 1 -- 172.21.15.116:0/292992033 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e8071910 con 0x7f23e8143cc0 2023-11-28T17:11:22.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.415+0000 7f23eda83640 1 --2- 172.21.15.116:0/292992033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23e8143cc0 0x7f23e8144380 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f23e40097a0 tx=0x7f23e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c443c0416710c0b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:22.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.415+0000 7f23eca81640 1 -- 172.21.15.116:0/292992033 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4031830 con 0x7f23e8143cc0 2023-11-28T17:11:22.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.415+0000 7f23eca81640 1 -- 172.21.15.116:0/292992033 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e4031990 con 0x7f23e8143cc0 2023-11-28T17:11:22.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.415+0000 7f23eca81640 1 -- 172.21.15.116:0/292992033 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4031d20 con 0x7f23e8143cc0 2023-11-28T17:11:22.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.416+0000 7f23eea85640 1 -- 172.21.15.116:0/292992033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23e8143cc0 msgr2=0x7f23e8144380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:22.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.416+0000 7f23eea85640 1 --2- 172.21.15.116:0/292992033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23e8143cc0 0x7f23e8144380 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f23e40097a0 tx=0x7f23e402b8d0 comp rx=0 tx=0).stop 2023-11-28T17:11:22.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.416+0000 7f23eea85640 1 -- 172.21.15.116:0/292992033 shutdown_connections 2023-11-28T17:11:22.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.416+0000 7f23eea85640 1 --2- 172.21.15.116:0/292992033 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23e80b48a0 0x7f23e815b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:22.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.416+0000 7f23eea85640 1 --2- 172.21.15.116:0/292992033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23e8143cc0 0x7f23e8144380 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:22.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.416+0000 7f23eea85640 1 -- 172.21.15.116:0/292992033 >> 172.21.15.116:0/292992033 conn(0x7f23e80b0b80 msgr2=0x7f23e80b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:22.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.416+0000 7f23eea85640 1 -- 172.21.15.116:0/292992033 shutdown_connections 2023-11-28T17:11:22.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.416+0000 7f23eea85640 1 -- 172.21.15.116:0/292992033 wait complete. 2023-11-28T17:11:22.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.418+0000 7f23eea85640 1 Processor -- start 2023-11-28T17:11:22.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.418+0000 7f23eea85640 1 -- start start 2023-11-28T17:11:22.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.419+0000 7f23eea85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23e80b48a0 0x7f23e8156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:22.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.419+0000 7f23eea85640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23e8143cc0 0x7f23e8157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:22.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.419+0000 7f23eea85640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23e81448c0 0x7f23e81f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:22.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.419+0000 7f23eea85640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e81e6240 con 0x7f23e80b48a0 2023-11-28T17:11:22.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.419+0000 7f23eea85640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e81e5630 con 0x7f23e8143cc0 2023-11-28T17:11:22.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.419+0000 7f23eea85640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e81e5910 con 0x7f23e81448c0 2023-11-28T17:11:22.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.419+0000 7f23eda83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23e80b48a0 0x7f23e8156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:22.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.419+0000 7f23eda83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23e80b48a0 0x7f23e8156e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33910/0 (socket says 172.21.15.116:33910) 2023-11-28T17:11:22.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.419+0000 7f23eda83640 1 -- 172.21.15.116:0/170104399 learned_addr learned my addr 172.21.15.116:0/170104399 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:22.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.419+0000 7f23eda83640 1 -- 172.21.15.116:0/170104399 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23e81448c0 msgr2=0x7f23e81f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:22.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.419+0000 7f23eda83640 1 --2- 172.21.15.116:0/170104399 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23e81448c0 0x7f23e81f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:22.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.419+0000 7f23eda83640 1 -- 172.21.15.116:0/170104399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23e8143cc0 msgr2=0x7f23e8157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:22.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.419+0000 7f23eda83640 1 --2- 172.21.15.116:0/170104399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23e8143cc0 0x7f23e8157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:22.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.419+0000 7f23eda83640 1 -- 172.21.15.116:0/170104399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e4009400 con 0x7f23e80b48a0 2023-11-28T17:11:22.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.420+0000 7f23eda83640 1 --2- 172.21.15.116:0/170104399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23e80b48a0 0x7f23e8156e30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f23e40098c0 tx=0x7f23e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:22.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.420+0000 7f23de7fc640 1 -- 172.21.15.116:0/170104399 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e403ba70 con 0x7f23e80b48a0 2023-11-28T17:11:22.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.420+0000 7f23de7fc640 1 -- 172.21.15.116:0/170104399 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e403bbf0 con 0x7f23e80b48a0 2023-11-28T17:11:22.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.420+0000 7f23de7fc640 1 -- 172.21.15.116:0/170104399 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e40453f0 con 0x7f23e80b48a0 2023-11-28T17:11:22.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.420+0000 7f23eea85640 1 -- 172.21.15.116:0/170104399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e81e5bf0 con 0x7f23e80b48a0 2023-11-28T17:11:22.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.420+0000 7f23eea85640 1 -- 172.21.15.116:0/170104399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e81e51c0 con 0x7f23e80b48a0 2023-11-28T17:11:22.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.421+0000 7f23eea85640 1 -- 172.21.15.116:0/170104399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e81f0ae0 con 0x7f23e80b48a0 2023-11-28T17:11:22.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.423+0000 7f23de7fc640 1 -- 172.21.15.116:0/170104399 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f23e404e450 con 0x7f23e80b48a0 2023-11-28T17:11:22.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.423+0000 7f23de7fc640 1 --2- 172.21.15.116:0/170104399 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f23cc06b2a0 0x7f23cc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:22.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.424+0000 7f23de7fc640 1 -- 172.21.15.116:0/170104399 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f23e40abb50 con 0x7f23e80b48a0 2023-11-28T17:11:22.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.424+0000 7f23ed282640 1 --2- 172.21.15.116:0/170104399 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f23cc06b2a0 0x7f23cc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:22.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.424+0000 7f23ed282640 1 --2- 172.21.15.116:0/170104399 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f23cc06b2a0 0x7f23cc06d760 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f23d8005880 tx=0x7f23d8007a00 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:22.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.428+0000 7f23de7fc640 1 -- 172.21.15.116:0/170104399 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f23e4076ba0 con 0x7f23e80b48a0 2023-11-28T17:11:22.623 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:22.621+0000 7f23eea85640 1 -- 172.21.15.116:0/170104399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f23e8069360 con 0x7f23e80b48a0 2023-11-28T17:11:22.960 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:11:22.970 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:11:23.189 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.188+0000 7f23de7fc640 1 -- 172.21.15.116:0/170104399 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v46) v1 ==== 170+0+0 (secure 0 0 0) 0x7f23e4035020 con 0x7f23e80b48a0 2023-11-28T17:11:23.190 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' created 2023-11-28T17:11:23.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.192+0000 7f23eea85640 1 -- 172.21.15.116:0/170104399 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f23cc06b2a0 msgr2=0x7f23cc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:23.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.192+0000 7f23eea85640 1 --2- 172.21.15.116:0/170104399 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f23cc06b2a0 0x7f23cc06d760 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f23d8005880 tx=0x7f23d8007a00 comp rx=0 tx=0).stop 2023-11-28T17:11:23.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.192+0000 7f23eea85640 1 -- 172.21.15.116:0/170104399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23e80b48a0 msgr2=0x7f23e8156e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:23.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.192+0000 7f23eea85640 1 --2- 172.21.15.116:0/170104399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23e80b48a0 0x7f23e8156e30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f23e40098c0 tx=0x7f23e40046c0 comp rx=0 tx=0).stop 2023-11-28T17:11:23.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.192+0000 7f23eea85640 1 -- 172.21.15.116:0/170104399 shutdown_connections 2023-11-28T17:11:23.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.192+0000 7f23eea85640 1 --2- 172.21.15.116:0/170104399 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f23cc06b2a0 0x7f23cc06d760 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:23.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.192+0000 7f23eea85640 1 --2- 172.21.15.116:0/170104399 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f23e81448c0 0x7f23e81f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:23.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.192+0000 7f23eea85640 1 --2- 172.21.15.116:0/170104399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23e8143cc0 0x7f23e8157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:23.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.193+0000 7f23eea85640 1 --2- 172.21.15.116:0/170104399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23e80b48a0 0x7f23e8156e30 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:23.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.193+0000 7f23eea85640 1 -- 172.21.15.116:0/170104399 >> 172.21.15.116:0/170104399 conn(0x7f23e80b0b80 msgr2=0x7f23e815a800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:23.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.193+0000 7f23eea85640 1 -- 172.21.15.116:0/170104399 shutdown_connections 2023-11-28T17:11:23.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.193+0000 7f23eea85640 1 -- 172.21.15.116:0/170104399 wait complete. 2023-11-28T17:11:23.231 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-28T17:11:23.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.418+0000 7fe11e1b0640 1 Processor -- start 2023-11-28T17:11:23.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.418+0000 7fe11e1b0640 1 -- start start 2023-11-28T17:11:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.420+0000 7fe11e1b0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe118146540 0x7fe118160e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:23.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.420+0000 7fe11e1b0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1180b5110 con 0x7fe118146a90 2023-11-28T17:11:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.420+0000 7fe11e1b0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1180b2190 con 0x7fe118145ee0 2023-11-28T17:11:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.420+0000 7fe11e1b0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe118161dc0 con 0x7fe118146540 2023-11-28T17:11:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.420+0000 7fe117fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe118146a90 0x7fe118071030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53114/0 (socket says 172.21.15.116:53114) 2023-11-28T17:11:23.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.420+0000 7fe117fff640 1 -- 172.21.15.116:0/2636797144 learned_addr learned my addr 172.21.15.116:0/2636797144 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.421+0000 7fe10ffff640 1 --2- 172.21.15.116:0/2636797144 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe118146540 0x7fe118160e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.421+0000 7fe116ffd640 1 -- 172.21.15.116:0/2636797144 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496442920 0 0) 0x7fe1180b5110 con 0x7fe118146a90 2023-11-28T17:11:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.421+0000 7fe116ffd640 1 -- 172.21.15.116:0/2636797144 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe104003520 con 0x7fe118146a90 2023-11-28T17:11:23.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.421+0000 7fe116ffd640 1 -- 172.21.15.116:0/2636797144 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2652238918 0 0) 0x7fe104003520 con 0x7fe118146a90 2023-11-28T17:11:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.421+0000 7fe116ffd640 1 -- 172.21.15.116:0/2636797144 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe118146540 msgr2=0x7fe118160e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.421+0000 7fe116ffd640 1 --2- 172.21.15.116:0/2636797144 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe118146540 0x7fe118160e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.421+0000 7fe116ffd640 1 -- 172.21.15.116:0/2636797144 >> v1:172.21.15.156:6789/0 conn(0x7fe118145ee0 legacy=0x7fe1180ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.421+0000 7fe116ffd640 1 -- 172.21.15.116:0/2636797144 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe118161ae0 con 0x7fe118146a90 2023-11-28T17:11:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.421+0000 7fe116ffd640 1 -- 172.21.15.116:0/2636797144 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fe1100019f0 con 0x7fe118146a90 2023-11-28T17:11:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.421+0000 7fe116ffd640 1 -- 172.21.15.116:0/2636797144 >> v1:172.21.15.116:6789/0 conn(0x7fe118146a90 legacy=0x7fe118071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:23.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.421+0000 7fe116ffd640 1 --2- 172.21.15.116:0/2636797144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1040039a0 0x7fe104003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.421+0000 7fe116ffd640 1 --2- 172.21.15.116:0/2636797144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1040047c0 0x7fe104004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.421+0000 7fe116ffd640 1 --2- 172.21.15.116:0/2636797144 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe118146540 0x7fe104005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.422+0000 7fe116ffd640 1 -- 172.21.15.116:0/2636797144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe118161dc0 con 0x7fe1040039a0 2023-11-28T17:11:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.422+0000 7fe116ffd640 1 -- 172.21.15.116:0/2636797144 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10400ba30 con 0x7fe1040047c0 2023-11-28T17:11:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.422+0000 7fe116ffd640 1 -- 172.21.15.116:0/2636797144 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10400bbf0 con 0x7fe118146540 2023-11-28T17:11:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.422+0000 7fe1177fe640 1 --2- 172.21.15.116:0/2636797144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1040039a0 0x7fe104003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.422+0000 7fe117fff640 1 --2- 172.21.15.116:0/2636797144 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe118146540 0x7fe104005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:23.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.422+0000 7fe10ffff640 1 --2- 172.21.15.116:0/2636797144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1040047c0 0x7fe104004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:23.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.422+0000 7fe1177fe640 1 -- 172.21.15.116:0/2636797144 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe118146540 msgr2=0x7fe104005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:23.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.422+0000 7fe1177fe640 1 --2- 172.21.15.116:0/2636797144 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe118146540 0x7fe104005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:23.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.422+0000 7fe1177fe640 1 -- 172.21.15.116:0/2636797144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1040047c0 msgr2=0x7fe104004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:23.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.422+0000 7fe1177fe640 1 --2- 172.21.15.116:0/2636797144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1040047c0 0x7fe104004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:23.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.422+0000 7fe1177fe640 1 -- 172.21.15.116:0/2636797144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe104003520 con 0x7fe1040039a0 2023-11-28T17:11:23.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.423+0000 7fe10ffff640 1 --2- 172.21.15.116:0/2636797144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1040047c0 0x7fe104004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:23.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.424+0000 7fe1177fe640 1 --2- 172.21.15.116:0/2636797144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1040039a0 0x7fe104003dc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe108001720 tx=0x7fe1040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4301ea215d6021 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:23.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.424+0000 7fe116ffd640 1 -- 172.21.15.116:0/2636797144 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1080052d0 con 0x7fe1040039a0 2023-11-28T17:11:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.425+0000 7fe116ffd640 1 -- 172.21.15.116:0/2636797144 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10802e050 con 0x7fe1040039a0 2023-11-28T17:11:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.425+0000 7fe116ffd640 1 -- 172.21.15.116:0/2636797144 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1080035f0 con 0x7fe1040039a0 2023-11-28T17:11:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.425+0000 7fe11e1b0640 1 -- 172.21.15.116:0/2636797144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1040039a0 msgr2=0x7fe104003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:23.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.425+0000 7fe11e1b0640 1 --2- 172.21.15.116:0/2636797144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1040039a0 0x7fe104003dc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe108001720 tx=0x7fe1040057f0 comp rx=0 tx=0).stop 2023-11-28T17:11:23.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.425+0000 7fe117fff640 1 -- 172.21.15.116:0/2636797144 reap_dead start 2023-11-28T17:11:23.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.425+0000 7fe11e1b0640 1 -- 172.21.15.116:0/2636797144 shutdown_connections 2023-11-28T17:11:23.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.425+0000 7fe11e1b0640 1 -- 172.21.15.116:0/2636797144 >> 172.21.15.116:0/2636797144 conn(0x7fe1180b0cf0 msgr2=0x7fe1180b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:23.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.425+0000 7fe11e1b0640 1 -- 172.21.15.116:0/2636797144 shutdown_connections 2023-11-28T17:11:23.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.425+0000 7fe11e1b0640 1 -- 172.21.15.116:0/2636797144 wait complete. 2023-11-28T17:11:23.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.427+0000 7fe11e1b0640 1 Processor -- start 2023-11-28T17:11:23.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.427+0000 7fe11e1b0640 1 -- start start 2023-11-28T17:11:23.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.428+0000 7fe11e1b0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1040039a0 0x7fe1181600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:23.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.428+0000 7fe11e1b0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe11813abb0 0x7fe118144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:23.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.428+0000 7fe11e1b0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1181447d0 0x7fe1180acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:23.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.428+0000 7fe11e1b0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11813d2c0 con 0x7fe1181447d0 2023-11-28T17:11:23.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.428+0000 7fe11e1b0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11813df90 con 0x7fe1040039a0 2023-11-28T17:11:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.428+0000 7fe11e1b0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11813b390 con 0x7fe11813abb0 2023-11-28T17:11:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.429+0000 7fe117fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1181447d0 0x7fe1180acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.429+0000 7fe117fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1181447d0 0x7fe1180acfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33930/0 (socket says 172.21.15.116:33930) 2023-11-28T17:11:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.429+0000 7fe117fff640 1 -- 172.21.15.116:0/522708003 learned_addr learned my addr 172.21.15.116:0/522708003 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.429+0000 7fe1177fe640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1040039a0 0x7fe1181600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.429+0000 7fe10ffff640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe11813abb0 0x7fe118144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.429+0000 7fe117fff640 1 -- 172.21.15.116:0/522708003 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe11813abb0 msgr2=0x7fe118144290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.429+0000 7fe117fff640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe11813abb0 0x7fe118144290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:23.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.429+0000 7fe117fff640 1 -- 172.21.15.116:0/522708003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1040039a0 msgr2=0x7fe1181600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:23.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.429+0000 7fe117fff640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1040039a0 0x7fe1181600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:23.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.429+0000 7fe117fff640 1 -- 172.21.15.116:0/522708003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11813b0b0 con 0x7fe1181447d0 2023-11-28T17:11:23.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.429+0000 7fe1177fe640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1040039a0 0x7fe1181600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:23.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.429+0000 7fe10ffff640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe11813abb0 0x7fe118144290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:23.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.430+0000 7fe117fff640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1181447d0 0x7fe1180acfe0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe118160d70 tx=0x7fe110007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:23.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.430+0000 7fe114ff9640 1 -- 172.21.15.116:0/522708003 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe110017070 con 0x7fe1181447d0 2023-11-28T17:11:23.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.430+0000 7fe114ff9640 1 -- 172.21.15.116:0/522708003 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11000eb00 con 0x7fe1181447d0 2023-11-28T17:11:23.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.430+0000 7fe114ff9640 1 -- 172.21.15.116:0/522708003 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11000ecb0 con 0x7fe1181447d0 2023-11-28T17:11:23.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.430+0000 7fe11e1b0640 1 -- 172.21.15.116:0/522708003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11813d4b0 con 0x7fe1181447d0 2023-11-28T17:11:23.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.430+0000 7fe11e1b0640 1 -- 172.21.15.116:0/522708003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11813da70 con 0x7fe1181447d0 2023-11-28T17:11:23.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.431+0000 7fe11e1b0640 1 -- 172.21.15.116:0/522708003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe118069360 con 0x7fe1181447d0 2023-11-28T17:11:23.436 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.432+0000 7fe114ff9640 1 -- 172.21.15.116:0/522708003 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe110013420 con 0x7fe1181447d0 2023-11-28T17:11:23.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.433+0000 7fe114ff9640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe0f806b2a0 0x7fe0f806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:23.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.433+0000 7fe114ff9640 1 -- 172.21.15.116:0/522708003 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fe1100896c0 con 0x7fe1181447d0 2023-11-28T17:11:23.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.434+0000 7fe1177fe640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe0f806b2a0 0x7fe0f806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:23.437 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.434+0000 7fe1177fe640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe0f806b2a0 0x7fe0f806d760 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe108033000 tx=0x7fe108009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:23.439 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.438+0000 7fe114ff9640 1 -- 172.21.15.116:0/522708003 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe110054640 con 0x7fe1181447d0 2023-11-28T17:11:23.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:23.666+0000 7fe11e1b0640 1 -- 172.21.15.116:0/522708003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe118073c80 con 0x7fe1181447d0 2023-11-28T17:11:24.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.192+0000 7fe114ff9640 1 -- 172.21.15.116:0/522708003 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v20) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe118073c80 con 0x7fe1181447d0 2023-11-28T17:11:24.194 INFO:teuthology.orchestra.run.smithi116.stderr:new fs with metadata pool 6 and data pool 7 2023-11-28T17:11:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.196+0000 7fe11e1b0640 1 -- 172.21.15.116:0/522708003 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe0f806b2a0 msgr2=0x7fe0f806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.196+0000 7fe11e1b0640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe0f806b2a0 0x7fe0f806d760 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe108033000 tx=0x7fe108009030 comp rx=0 tx=0).stop 2023-11-28T17:11:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.196+0000 7fe11e1b0640 1 -- 172.21.15.116:0/522708003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1181447d0 msgr2=0x7fe1180acfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:24.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.196+0000 7fe11e1b0640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1181447d0 0x7fe1180acfe0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe118160d70 tx=0x7fe110007bc0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.196+0000 7fe11e1b0640 1 -- 172.21.15.116:0/522708003 shutdown_connections 2023-11-28T17:11:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.196+0000 7fe11e1b0640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe0f806b2a0 0x7fe0f806d760 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.196+0000 7fe11e1b0640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1181447d0 0x7fe1180acfe0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.196+0000 7fe11e1b0640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe11813abb0 0x7fe118144290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.196+0000 7fe11e1b0640 1 --2- 172.21.15.116:0/522708003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1040039a0 0x7fe1181600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.196+0000 7fe11e1b0640 1 -- 172.21.15.116:0/522708003 >> 172.21.15.116:0/522708003 conn(0x7fe1180b0cf0 msgr2=0x7fe1180aec60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:24.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.198+0000 7fe11e1b0640 1 -- 172.21.15.116:0/522708003 shutdown_connections 2023-11-28T17:11:24.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.198+0000 7fe11e1b0640 1 -- 172.21.15.116:0/522708003 wait complete. 2023-11-28T17:11:24.234 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:11:24.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.417+0000 7f2edc81c640 1 Processor -- start 2023-11-28T17:11:24.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.418+0000 7f2edc81c640 1 -- start start 2023-11-28T17:11:24.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.419+0000 7f2edc81c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ed8144430 0x7f2ed815f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:24.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.419+0000 7f2edc81c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2ed806c250 0x7f2ed806c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:24.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.419+0000 7f2edc81c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed8070bd0 con 0x7f2ed80703f0 2023-11-28T17:11:24.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.419+0000 7f2edc81c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed8144810 con 0x7f2ed806c250 2023-11-28T17:11:24.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.419+0000 7f2edc81c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed806cc20 con 0x7f2ed8144430 2023-11-28T17:11:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.419+0000 7f2ecdd9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2ed80703f0 0x7f2ed8071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53116/0 (socket says 172.21.15.116:53116) 2023-11-28T17:11:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.420+0000 7f2ecdd9b640 1 -- 172.21.15.116:0/397248809 learned_addr learned my addr 172.21.15.116:0/397248809 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.420+0000 7f2ed6d9d640 1 --2- 172.21.15.116:0/397248809 >> v2:172.21.15.116:3300/0 conn(0x7f2ed806c250 0x7f2ed806c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.420+0000 7f2ed659c640 1 --2- 172.21.15.116:0/397248809 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ed8144430 0x7f2ed815f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.420+0000 7f2ed5d9b640 1 -- 172.21.15.116:0/397248809 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450858563 0 0) 0x7f2ed8070bd0 con 0x7f2ed80703f0 2023-11-28T17:11:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.420+0000 7f2ed5d9b640 1 -- 172.21.15.116:0/397248809 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb8003520 con 0x7f2ed80703f0 2023-11-28T17:11:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.420+0000 7f2ed6d9d640 1 -- 172.21.15.116:0/397248809 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ed8144430 msgr2=0x7f2ed815f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.420+0000 7f2ed6d9d640 1 --2- 172.21.15.116:0/397248809 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ed8144430 0x7f2ed815f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.420+0000 7f2ed6d9d640 1 -- 172.21.15.116:0/397248809 >> v1:172.21.15.116:6789/0 conn(0x7f2ed80703f0 legacy=0x7f2ed8071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.420+0000 7f2ed6d9d640 1 -- 172.21.15.116:0/397248809 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed8070eb0 con 0x7f2ed806c250 2023-11-28T17:11:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.421+0000 7f2ed6d9d640 1 --2- 172.21.15.116:0/397248809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ed806c250 0x7f2ed806c6e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009d90 tx=0x7f2ed002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ffad6a27a8257d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.422+0000 7f2ed5d9b640 1 -- 172.21.15.116:0/397248809 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed002cc50 con 0x7f2ed806c250 2023-11-28T17:11:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.422+0000 7f2ed5d9b640 1 -- 172.21.15.116:0/397248809 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed002cdb0 con 0x7f2ed806c250 2023-11-28T17:11:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.422+0000 7f2ed5d9b640 1 -- 172.21.15.116:0/397248809 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0034660 con 0x7f2ed806c250 2023-11-28T17:11:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.422+0000 7f2edc81c640 1 -- 172.21.15.116:0/397248809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ed806c250 msgr2=0x7f2ed806c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.422+0000 7f2edc81c640 1 --2- 172.21.15.116:0/397248809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ed806c250 0x7f2ed806c6e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009d90 tx=0x7f2ed002bea0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.422+0000 7f2edc81c640 1 -- 172.21.15.116:0/397248809 shutdown_connections 2023-11-28T17:11:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.422+0000 7f2edc81c640 1 --2- 172.21.15.116:0/397248809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ed806c250 0x7f2ed806c6e0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.422+0000 7f2edc81c640 1 --2- 172.21.15.116:0/397248809 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ed8144430 0x7f2ed815f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.422+0000 7f2edc81c640 1 -- 172.21.15.116:0/397248809 >> 172.21.15.116:0/397248809 conn(0x7f2ed806a4c0 msgr2=0x7f2ed806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.422+0000 7f2edc81c640 1 -- 172.21.15.116:0/397248809 shutdown_connections 2023-11-28T17:11:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.423+0000 7f2edc81c640 1 -- 172.21.15.116:0/397248809 wait complete. 2023-11-28T17:11:24.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.424+0000 7f2edc81c640 1 Processor -- start 2023-11-28T17:11:24.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.424+0000 7f2edc81c640 1 -- start start 2023-11-28T17:11:24.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.425+0000 7f2edc81c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ed806c250 0x7f2ed813a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:24.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.425+0000 7f2edc81c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ed80703f0 0x7f2ed815be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:24.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.425+0000 7f2edc81c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ed8144430 0x7f2ed80c5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:24.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.425+0000 7f2edc81c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed815c3d0 con 0x7f2ed806c250 2023-11-28T17:11:24.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.425+0000 7f2edc81c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed813b440 con 0x7f2ed8144430 2023-11-28T17:11:24.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.425+0000 7f2edc81c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed813a830 con 0x7f2ed80703f0 2023-11-28T17:11:24.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.426+0000 7f2ed659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ed806c250 0x7f2ed813a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:24.428 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.426+0000 7f2ed659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ed806c250 0x7f2ed813a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33946/0 (socket says 172.21.15.116:33946) 2023-11-28T17:11:24.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.426+0000 7f2ed659c640 1 -- 172.21.15.116:0/2872614063 learned_addr learned my addr 172.21.15.116:0/2872614063 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:24.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.426+0000 7f2ed659c640 1 -- 172.21.15.116:0/2872614063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ed80703f0 msgr2=0x7f2ed815be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:24.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.426+0000 7f2ed659c640 1 --2- 172.21.15.116:0/2872614063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ed80703f0 0x7f2ed815be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.426+0000 7f2ed659c640 1 -- 172.21.15.116:0/2872614063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ed8144430 msgr2=0x7f2ed80c5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:24.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.426+0000 7f2ed659c640 1 --2- 172.21.15.116:0/2872614063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ed8144430 0x7f2ed80c5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.426+0000 7f2ed659c640 1 -- 172.21.15.116:0/2872614063 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed0009a70 con 0x7f2ed806c250 2023-11-28T17:11:24.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.426+0000 7f2ed659c640 1 --2- 172.21.15.116:0/2872614063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ed806c250 0x7f2ed813a1e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2ec800d800 tx=0x7f2ec800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:24.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.427+0000 7f2ecf7fe640 1 -- 172.21.15.116:0/2872614063 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec800b6c0 con 0x7f2ed806c250 2023-11-28T17:11:24.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.427+0000 7f2ecf7fe640 1 -- 172.21.15.116:0/2872614063 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec800b820 con 0x7f2ed806c250 2023-11-28T17:11:24.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.427+0000 7f2ecf7fe640 1 -- 172.21.15.116:0/2872614063 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec800bbb0 con 0x7f2ed806c250 2023-11-28T17:11:24.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.427+0000 7f2edc81c640 1 -- 172.21.15.116:0/2872614063 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed813ab10 con 0x7f2ed806c250 2023-11-28T17:11:24.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.427+0000 7f2edc81c640 1 -- 172.21.15.116:0/2872614063 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed813b0d0 con 0x7f2ed806c250 2023-11-28T17:11:24.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.428+0000 7f2edc81c640 1 -- 172.21.15.116:0/2872614063 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed80693d0 con 0x7f2ed806c250 2023-11-28T17:11:24.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.429+0000 7f2ecf7fe640 1 -- 172.21.15.116:0/2872614063 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2ec800bd10 con 0x7f2ed806c250 2023-11-28T17:11:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.430+0000 7f2ecf7fe640 1 --2- 172.21.15.116:0/2872614063 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ebc06b2f0 0x7f2ebc06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.430+0000 7f2ecf7fe640 1 -- 172.21.15.116:0/2872614063 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ec808a090 con 0x7f2ed806c250 2023-11-28T17:11:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.430+0000 7f2ecdd9b640 1 --2- 172.21.15.116:0/2872614063 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ebc06b2f0 0x7f2ebc06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:24.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.431+0000 7f2ecdd9b640 1 --2- 172.21.15.116:0/2872614063 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ebc06b2f0 0x7f2ebc06d7b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2ed80cb110 tx=0x7f2ec0007930 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:24.435 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.434+0000 7f2ecf7fe640 1 -- 172.21.15.116:0/2872614063 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ec8054fe0 con 0x7f2ed806c250 2023-11-28T17:11:24.625 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.624+0000 7f2edc81c640 1 -- 172.21.15.116:0/2872614063 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ed806d100 con 0x7f2ed806c250 2023-11-28T17:11:24.627 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.625+0000 7f2ecf7fe640 1 -- 172.21.15.116:0/2872614063 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16534 (secure 0 0 0) 0x7f2ec8058600 con 0x7f2ed806c250 2023-11-28T17:11:24.627 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:24.627 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":47,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:11:23.668051+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:11:21.518078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T17:11:22.672451+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:11:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.628+0000 7f2edc81c640 1 -- 172.21.15.116:0/2872614063 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ebc06b2f0 msgr2=0x7f2ebc06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.628+0000 7f2edc81c640 1 --2- 172.21.15.116:0/2872614063 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ebc06b2f0 0x7f2ebc06d7b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2ed80cb110 tx=0x7f2ec0007930 comp rx=0 tx=0).stop 2023-11-28T17:11:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.628+0000 7f2edc81c640 1 -- 172.21.15.116:0/2872614063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ed806c250 msgr2=0x7f2ed813a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.628+0000 7f2edc81c640 1 --2- 172.21.15.116:0/2872614063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ed806c250 0x7f2ed813a1e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f2ec800d800 tx=0x7f2ec800dcc0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.628+0000 7f2edc81c640 1 -- 172.21.15.116:0/2872614063 shutdown_connections 2023-11-28T17:11:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.628+0000 7f2edc81c640 1 --2- 172.21.15.116:0/2872614063 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ebc06b2f0 0x7f2ebc06d7b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.628+0000 7f2edc81c640 1 --2- 172.21.15.116:0/2872614063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ed8144430 0x7f2ed80c5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.628+0000 7f2edc81c640 1 --2- 172.21.15.116:0/2872614063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ed80703f0 0x7f2ed815be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.628+0000 7f2edc81c640 1 --2- 172.21.15.116:0/2872614063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ed806c250 0x7f2ed813a1e0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.628+0000 7f2edc81c640 1 -- 172.21.15.116:0/2872614063 >> 172.21.15.116:0/2872614063 conn(0x7f2ed806a4c0 msgr2=0x7f2ed8071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.629+0000 7f2edc81c640 1 -- 172.21.15.116:0/2872614063 shutdown_connections 2023-11-28T17:11:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.629+0000 7f2edc81c640 1 -- 172.21.15.116:0/2872614063 wait complete. 2023-11-28T17:11:24.649 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:11:24.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.846+0000 7f4f246cf640 1 Processor -- start 2023-11-28T17:11:24.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.846+0000 7f4f246cf640 1 -- start start 2023-11-28T17:11:24.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.847+0000 7f4f246cf640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4f1c15f7b0 0x7f4f1c071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:24.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.847+0000 7f4f246cf640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c15fc10 con 0x7f4f1c0721b0 2023-11-28T17:11:24.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.847+0000 7f4f246cf640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c070bd0 con 0x7f4f1c144430 2023-11-28T17:11:24.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.847+0000 7f4f246cf640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1c144810 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.848+0000 7f4f21c6a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4f1c15f7b0 0x7f4f1c071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:24.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.848+0000 7f4f22c6c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4f1c0721b0 0x7f4f1c06c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53122/0 (socket says 172.21.15.116:53122) 2023-11-28T17:11:24.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.848+0000 7f4f22c6c640 1 -- 172.21.15.116:0/3614224770 learned_addr learned my addr 172.21.15.116:0/3614224770 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:24.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.848+0000 7f4f21c6a640 1 -- 172.21.15.116:0/3614224770 >> v1:172.21.15.156:6789/0 conn(0x7f4f1c144430 legacy=0x7f4f1c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:24.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.848+0000 7f4f21c6a640 1 -- 172.21.15.116:0/3614224770 >> v1:172.21.15.116:6789/0 conn(0x7f4f1c0721b0 legacy=0x7f4f1c06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:24.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.848+0000 7f4f21c6a640 1 -- 172.21.15.116:0/3614224770 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c070eb0 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.848+0000 7f4f21469640 1 -- 172.21.15.116:0/3614224770 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364552602 0 0) 0x7f4f1c15fc10 con 0x7f4f1c0721b0 2023-11-28T17:11:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.849+0000 7f4f21c6a640 1 --2- 172.21.15.116:0/3614224770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f1c15f7b0 0x7f4f1c071c70 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4f080097a0 tx=0x7f4f0802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d668a67388d039 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.850+0000 7f4f21469640 1 -- 172.21.15.116:0/3614224770 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08031830 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.850+0000 7f4f21469640 1 -- 172.21.15.116:0/3614224770 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f08031990 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.850+0000 7f4f21469640 1 -- 172.21.15.116:0/3614224770 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08031d20 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.850+0000 7f4f246cf640 1 -- 172.21.15.116:0/3614224770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f1c15f7b0 msgr2=0x7f4f1c071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.850+0000 7f4f246cf640 1 --2- 172.21.15.116:0/3614224770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f1c15f7b0 0x7f4f1c071c70 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4f080097a0 tx=0x7f4f0802b8d0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.850+0000 7f4f246cf640 1 -- 172.21.15.116:0/3614224770 shutdown_connections 2023-11-28T17:11:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.850+0000 7f4f246cf640 1 --2- 172.21.15.116:0/3614224770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f1c15f7b0 0x7f4f1c071c70 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.850+0000 7f4f246cf640 1 -- 172.21.15.116:0/3614224770 >> 172.21.15.116:0/3614224770 conn(0x7f4f1c06a4c0 msgr2=0x7f4f1c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.850+0000 7f4f246cf640 1 -- 172.21.15.116:0/3614224770 shutdown_connections 2023-11-28T17:11:24.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.850+0000 7f4f246cf640 1 -- 172.21.15.116:0/3614224770 wait complete. 2023-11-28T17:11:24.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.852+0000 7f4f246cf640 1 Processor -- start 2023-11-28T17:11:24.853 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.852+0000 7f4f246cf640 1 -- start start 2023-11-28T17:11:24.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.853+0000 7f4f246cf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f1c0721b0 0x7f4f1c13b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:24.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.853+0000 7f4f246cf640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4f1c144430 0x7f4f1c0c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:24.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.853+0000 7f4f246cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f1c15f7b0 0x7f4f1c13e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:24.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.853+0000 7f4f246cf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c0c9a20 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.853+0000 7f4f246cf640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c0c8e10 con 0x7f4f1c0721b0 2023-11-28T17:11:24.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.853+0000 7f4f246cf640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c0c90f0 con 0x7f4f1c144430 2023-11-28T17:11:24.855 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.853+0000 7f4f22c6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f1c15f7b0 0x7f4f1c13e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:24.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.853+0000 7f4f22c6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f1c15f7b0 0x7f4f1c13e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33962/0 (socket says 172.21.15.116:33962) 2023-11-28T17:11:24.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.853+0000 7f4f22c6c640 1 -- 172.21.15.116:0/3241543499 learned_addr learned my addr 172.21.15.116:0/3241543499 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:24.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.854+0000 7f4f22c6c640 1 -- 172.21.15.116:0/3241543499 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4f1c144430 msgr2=0x7f4f1c0c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:24.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.854+0000 7f4f22c6c640 1 --2- 172.21.15.116:0/3241543499 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4f1c144430 0x7f4f1c0c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.854+0000 7f4f22c6c640 1 -- 172.21.15.116:0/3241543499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f1c0721b0 msgr2=0x7f4f1c13b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:24.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.854+0000 7f4f22c6c640 1 --2- 172.21.15.116:0/3241543499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f1c0721b0 0x7f4f1c13b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:24.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.854+0000 7f4f22c6c640 1 -- 172.21.15.116:0/3241543499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f08009400 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.854+0000 7f4f22c6c640 1 --2- 172.21.15.116:0/3241543499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f1c15f7b0 0x7f4f1c13e800 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4f1400c450 tx=0x7f4f1400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.854+0000 7f4f1affd640 1 -- 172.21.15.116:0/3241543499 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f140074e0 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.854+0000 7f4f1affd640 1 -- 172.21.15.116:0/3241543499 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f14007660 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.854+0000 7f4f1affd640 1 -- 172.21.15.116:0/3241543499 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f14004a20 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.854+0000 7f4f246cf640 1 -- 172.21.15.116:0/3241543499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1c0c93d0 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.854+0000 7f4f246cf640 1 -- 172.21.15.116:0/3241543499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1c13bda0 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.856+0000 7f4f246cf640 1 -- 172.21.15.116:0/3241543499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1c0693d0 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.857+0000 7f4f1affd640 1 -- 172.21.15.116:0/3241543499 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f4f14004b80 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.858 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.857+0000 7f4f1affd640 1 --2- 172.21.15.116:0/3241543499 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4f0006b2d0 0x7f4f0006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:24.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.858+0000 7f4f1affd640 1 -- 172.21.15.116:0/3241543499 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f140899a0 con 0x7f4f1c15f7b0 2023-11-28T17:11:24.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.858+0000 7f4f2246b640 1 --2- 172.21.15.116:0/3241543499 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4f0006b2d0 0x7f4f0006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:24.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.858+0000 7f4f2246b640 1 --2- 172.21.15.116:0/3241543499 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4f0006b2d0 0x7f4f0006d790 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4f100043c0 tx=0x7f4f10007d30 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:24.863 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:24.862+0000 7f4f1affd640 1 -- 172.21.15.116:0/3241543499 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f14054810 con 0x7f4f1c15f7b0 2023-11-28T17:11:25.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.055+0000 7f4f246cf640 1 -- 172.21.15.116:0/3241543499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f1c06d100 con 0x7f4f1c15f7b0 2023-11-28T17:11:25.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.057+0000 7f4f1affd640 1 -- 172.21.15.116:0/3241543499 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16534 (secure 0 0 0) 0x7f4f14057e30 con 0x7f4f1c15f7b0 2023-11-28T17:11:25.059 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:25.059 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":47,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:11:23.668051+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:11:21.518078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T17:11:22.672451+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:11:25.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.060+0000 7f4f246cf640 1 -- 172.21.15.116:0/3241543499 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4f0006b2d0 msgr2=0x7f4f0006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:25.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.060+0000 7f4f246cf640 1 --2- 172.21.15.116:0/3241543499 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4f0006b2d0 0x7f4f0006d790 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4f100043c0 tx=0x7f4f10007d30 comp rx=0 tx=0).stop 2023-11-28T17:11:25.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.060+0000 7f4f246cf640 1 -- 172.21.15.116:0/3241543499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f1c15f7b0 msgr2=0x7f4f1c13e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:25.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.060+0000 7f4f246cf640 1 --2- 172.21.15.116:0/3241543499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f1c15f7b0 0x7f4f1c13e800 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4f1400c450 tx=0x7f4f1400c910 comp rx=0 tx=0).stop 2023-11-28T17:11:25.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.060+0000 7f4f246cf640 1 -- 172.21.15.116:0/3241543499 shutdown_connections 2023-11-28T17:11:25.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.060+0000 7f4f246cf640 1 --2- 172.21.15.116:0/3241543499 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f4f0006b2d0 0x7f4f0006d790 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:25.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.060+0000 7f4f246cf640 1 --2- 172.21.15.116:0/3241543499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4f1c15f7b0 0x7f4f1c13e800 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:25.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.060+0000 7f4f246cf640 1 --2- 172.21.15.116:0/3241543499 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4f1c144430 0x7f4f1c0c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:25.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.060+0000 7f4f246cf640 1 --2- 172.21.15.116:0/3241543499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4f1c0721b0 0x7f4f1c13b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:25.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.060+0000 7f4f246cf640 1 -- 172.21.15.116:0/3241543499 >> 172.21.15.116:0/3241543499 conn(0x7f4f1c06a4c0 msgr2=0x7f4f1c15df60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:25.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.060+0000 7f4f246cf640 1 -- 172.21.15.116:0/3241543499 shutdown_connections 2023-11-28T17:11:25.062 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.060+0000 7f4f246cf640 1 -- 172.21.15.116:0/3241543499 wait complete. 2023-11-28T17:11:25.080 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-28T17:11:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.302+0000 7f901df8a640 1 Processor -- start 2023-11-28T17:11:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.302+0000 7f901df8a640 1 -- start start 2023-11-28T17:11:25.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.303+0000 7f901df8a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f901814a740 0x7f90180cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:25.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.304+0000 7f901df8a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f90180b4600 0x7f9018070ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:25.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.304+0000 7f901df8a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9018070420 con 0x7f901814ac30 2023-11-28T17:11:25.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.304+0000 7f901df8a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9018070aa0 con 0x7f90180b4600 2023-11-28T17:11:25.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.304+0000 7f901df8a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90180b2fc0 con 0x7f901814a740 2023-11-28T17:11:25.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.304+0000 7f900ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f901814ac30 0x7f90180b3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53136/0 (socket says 172.21.15.116:53136) 2023-11-28T17:11:25.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.304+0000 7f900ffff640 1 -- 172.21.15.116:0/507518976 learned_addr learned my addr 172.21.15.116:0/507518976 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.304+0000 7f9016ffd640 1 -- 172.21.15.116:0/507518976 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675813420 0 0) 0x7f9018070420 con 0x7f901814ac30 2023-11-28T17:11:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.304+0000 7f9017fff640 1 --2- 172.21.15.116:0/507518976 >> v2:172.21.15.156:3300/0 conn(0x7f90180b4600 0x7f9018070ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.304+0000 7f90177fe640 1 --2- 172.21.15.116:0/507518976 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f901814a740 0x7f90180cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.304+0000 7f9016ffd640 1 -- 172.21.15.116:0/507518976 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9004003520 con 0x7f901814ac30 2023-11-28T17:11:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9016ffd640 1 -- 172.21.15.116:0/507518976 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1219218267 0 0) 0x7f9004003520 con 0x7f901814ac30 2023-11-28T17:11:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9016ffd640 1 -- 172.21.15.116:0/507518976 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f901814a740 msgr2=0x7f90180cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9016ffd640 1 --2- 172.21.15.116:0/507518976 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f901814a740 0x7f90180cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9016ffd640 1 -- 172.21.15.116:0/507518976 >> v2:172.21.15.156:3300/0 conn(0x7f90180b4600 msgr2=0x7f9018070ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9016ffd640 1 --2- 172.21.15.116:0/507518976 >> v2:172.21.15.156:3300/0 conn(0x7f90180b4600 0x7f9018070ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9016ffd640 1 -- 172.21.15.116:0/507518976 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90180706c0 con 0x7f901814ac30 2023-11-28T17:11:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9016ffd640 1 -- 172.21.15.116:0/507518976 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f9000001670 con 0x7f901814ac30 2023-11-28T17:11:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9016ffd640 1 -- 172.21.15.116:0/507518976 >> v1:172.21.15.116:6789/0 conn(0x7f901814ac30 legacy=0x7f90180b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9016ffd640 1 --2- 172.21.15.116:0/507518976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9004003970 0x7f9004003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9016ffd640 1 --2- 172.21.15.116:0/507518976 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f901814a740 0x7f90040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9016ffd640 1 --2- 172.21.15.116:0/507518976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9004004d00 0x7f9004005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9016ffd640 1 -- 172.21.15.116:0/507518976 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9018070aa0 con 0x7f9004004d00 2023-11-28T17:11:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9016ffd640 1 -- 172.21.15.116:0/507518976 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90180b2fc0 con 0x7f9004003970 2023-11-28T17:11:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9016ffd640 1 -- 172.21.15.116:0/507518976 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f900400bab0 con 0x7f901814a740 2023-11-28T17:11:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9017fff640 1 --2- 172.21.15.116:0/507518976 >> v2:172.21.15.156:3300/0 conn(0x7f90180b4600 0x7f9018070ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:11:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.305+0000 7f9017fff640 1 --2- 172.21.15.116:0/507518976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9004004d00 0x7f9004005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.306+0000 7f90177fe640 1 --2- 172.21.15.116:0/507518976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9004003970 0x7f9004003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.306+0000 7f900ffff640 1 --2- 172.21.15.116:0/507518976 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f901814a740 0x7f90040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.306+0000 7f9017fff640 1 -- 172.21.15.116:0/507518976 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f901814a740 msgr2=0x7f90040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.306+0000 7f9017fff640 1 --2- 172.21.15.116:0/507518976 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f901814a740 0x7f90040047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.306+0000 7f9017fff640 1 -- 172.21.15.116:0/507518976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9004003970 msgr2=0x7f9004003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.306+0000 7f9017fff640 1 --2- 172.21.15.116:0/507518976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9004003970 0x7f9004003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.306+0000 7f9017fff640 1 -- 172.21.15.116:0/507518976 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9004003520 con 0x7f9004004d00 2023-11-28T17:11:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.307+0000 7f9017fff640 1 --2- 172.21.15.116:0/507518976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9004004d00 0x7f9004005180 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f90100073e0 tx=0x7f90040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8583dde912b0807 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.307+0000 7f9016ffd640 1 -- 172.21.15.116:0/507518976 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901002cdc0 con 0x7f9004004d00 2023-11-28T17:11:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.307+0000 7f9016ffd640 1 -- 172.21.15.116:0/507518976 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9010034440 con 0x7f9004004d00 2023-11-28T17:11:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.307+0000 7f9016ffd640 1 -- 172.21.15.116:0/507518976 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90100347b0 con 0x7f9004004d00 2023-11-28T17:11:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.308+0000 7f901df8a640 1 -- 172.21.15.116:0/507518976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9004004d00 msgr2=0x7f9004005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.308+0000 7f901df8a640 1 --2- 172.21.15.116:0/507518976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9004004d00 0x7f9004005180 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f90100073e0 tx=0x7f90040057f0 comp rx=0 tx=0).stop 2023-11-28T17:11:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.308+0000 7f9017fff640 1 -- 172.21.15.116:0/507518976 reap_dead start 2023-11-28T17:11:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.308+0000 7f901df8a640 1 -- 172.21.15.116:0/507518976 shutdown_connections 2023-11-28T17:11:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.308+0000 7f901df8a640 1 -- 172.21.15.116:0/507518976 >> 172.21.15.116:0/507518976 conn(0x7f90180b0cf0 msgr2=0x7f90180b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.308+0000 7f901df8a640 1 -- 172.21.15.116:0/507518976 shutdown_connections 2023-11-28T17:11:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.308+0000 7f901df8a640 1 -- 172.21.15.116:0/507518976 wait complete. 2023-11-28T17:11:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.310+0000 7f901df8a640 1 Processor -- start 2023-11-28T17:11:25.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.310+0000 7f901df8a640 1 -- start start 2023-11-28T17:11:25.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.311+0000 7f901df8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9004004d00 0x7f90181f4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.311+0000 7f901df8a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90181f8230 0x7f901806c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.311+0000 7f901df8a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9018161a40 0x7f901815fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.311+0000 7f901df8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9018163100 con 0x7f9004004d00 2023-11-28T17:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.311+0000 7f901df8a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90181f7da0 con 0x7f9018161a40 2023-11-28T17:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.311+0000 7f901df8a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90181f74b0 con 0x7f90181f8230 2023-11-28T17:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.311+0000 7f90177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9004004d00 0x7f90181f4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.311+0000 7f90177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9004004d00 0x7f90181f4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33986/0 (socket says 172.21.15.116:33986) 2023-11-28T17:11:25.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.311+0000 7f90177fe640 1 -- 172.21.15.116:0/1722494113 learned_addr learned my addr 172.21.15.116:0/1722494113 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.311+0000 7f90177fe640 1 -- 172.21.15.116:0/1722494113 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90181f8230 msgr2=0x7f901806c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.311+0000 7f90177fe640 1 --2- 172.21.15.116:0/1722494113 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90181f8230 0x7f901806c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.311+0000 7f90177fe640 1 -- 172.21.15.116:0/1722494113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9018161a40 msgr2=0x7f901815fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.311+0000 7f90177fe640 1 --2- 172.21.15.116:0/1722494113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9018161a40 0x7f901815fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.311+0000 7f90177fe640 1 -- 172.21.15.116:0/1722494113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90181f4eb0 con 0x7f9004004d00 2023-11-28T17:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.312+0000 7f90177fe640 1 --2- 172.21.15.116:0/1722494113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9004004d00 0x7f90181f4970 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f900800f580 tx=0x7f900800fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.312+0000 7f9014ff9640 1 -- 172.21.15.116:0/1722494113 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008013810 con 0x7f9004004d00 2023-11-28T17:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.312+0000 7f9014ff9640 1 -- 172.21.15.116:0/1722494113 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9008013970 con 0x7f9004004d00 2023-11-28T17:11:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.312+0000 7f9014ff9640 1 -- 172.21.15.116:0/1722494113 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9008013ce0 con 0x7f9004004d00 2023-11-28T17:11:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.312+0000 7f901df8a640 1 -- 172.21.15.116:0/1722494113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9018161f20 con 0x7f9004004d00 2023-11-28T17:11:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.312+0000 7f901df8a640 1 -- 172.21.15.116:0/1722494113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90181f7a30 con 0x7f9004004d00 2023-11-28T17:11:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.313+0000 7f901df8a640 1 -- 172.21.15.116:0/1722494113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90180cbe70 con 0x7f9004004d00 2023-11-28T17:11:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.314+0000 7f9014ff9640 1 -- 172.21.15.116:0/1722494113 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f900801f460 con 0x7f9004004d00 2023-11-28T17:11:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.315+0000 7f9014ff9640 1 --2- 172.21.15.116:0/1722494113 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8ff806b2a0 0x7f8ff806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.315+0000 7f9014ff9640 1 -- 172.21.15.116:0/1722494113 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f900808a080 con 0x7f9004004d00 2023-11-28T17:11:25.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.315+0000 7f900ffff640 1 --2- 172.21.15.116:0/1722494113 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8ff806b2a0 0x7f8ff806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:25.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.316+0000 7f900ffff640 1 --2- 172.21.15.116:0/1722494113 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8ff806b2a0 0x7f8ff806d760 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f901806c910 tx=0x7f9000009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:25.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.319+0000 7f9014ff9640 1 -- 172.21.15.116:0/1722494113 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90080bce90 con 0x7f9004004d00 2023-11-28T17:11:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:25.551+0000 7f901df8a640 1 -- 172.21.15.116:0/1722494113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9018069360 con 0x7f9004004d00 2023-11-28T17:11:26.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.241+0000 7f9014ff9640 1 -- 172.21.15.116:0/1722494113 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v23) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9008054e60 con 0x7f9004004d00 2023-11-28T17:11:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.245+0000 7f901df8a640 1 -- 172.21.15.116:0/1722494113 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8ff806b2a0 msgr2=0x7f8ff806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.245+0000 7f901df8a640 1 --2- 172.21.15.116:0/1722494113 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8ff806b2a0 0x7f8ff806d760 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f901806c910 tx=0x7f9000009040 comp rx=0 tx=0).stop 2023-11-28T17:11:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.245+0000 7f901df8a640 1 -- 172.21.15.116:0/1722494113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9004004d00 msgr2=0x7f90181f4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.245+0000 7f901df8a640 1 --2- 172.21.15.116:0/1722494113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9004004d00 0x7f90181f4970 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f900800f580 tx=0x7f900800fa40 comp rx=0 tx=0).stop 2023-11-28T17:11:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.245+0000 7f901df8a640 1 -- 172.21.15.116:0/1722494113 shutdown_connections 2023-11-28T17:11:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.245+0000 7f901df8a640 1 --2- 172.21.15.116:0/1722494113 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8ff806b2a0 0x7f8ff806d760 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.245+0000 7f901df8a640 1 --2- 172.21.15.116:0/1722494113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9018161a40 0x7f901815fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.245+0000 7f901df8a640 1 --2- 172.21.15.116:0/1722494113 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f90181f8230 0x7f901806c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.245+0000 7f901df8a640 1 --2- 172.21.15.116:0/1722494113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9004004d00 0x7f90181f4970 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.245+0000 7f901df8a640 1 -- 172.21.15.116:0/1722494113 >> 172.21.15.116:0/1722494113 conn(0x7f90180b0cf0 msgr2=0x7f90180b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.245+0000 7f901df8a640 1 -- 172.21.15.116:0/1722494113 shutdown_connections 2023-11-28T17:11:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.245+0000 7f901df8a640 1 -- 172.21.15.116:0/1722494113 wait complete. 2023-11-28T17:11:26.268 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-28T17:11:26.268 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-28T17:11:26.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.488+0000 7f670df78640 1 Processor -- start 2023-11-28T17:11:26.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.488+0000 7f670df78640 1 -- start start 2023-11-28T17:11:26.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.489+0000 7f670df78640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f670815d480 0x7f67080d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:26.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.489+0000 7f670df78640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f67080d01e0 0x7f6708160270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:26.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.489+0000 7f670df78640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f670815ddc0 con 0x7f67080d1310 2023-11-28T17:11:26.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.489+0000 7f670df78640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f670815e0a0 con 0x7f67080d01e0 2023-11-28T17:11:26.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.489+0000 7f670df78640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670815e380 con 0x7f670815d480 2023-11-28T17:11:26.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.490+0000 7f6706ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f67080d1310 0x7f67080cfad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53144/0 (socket says 172.21.15.116:53144) 2023-11-28T17:11:26.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.490+0000 7f6706ffd640 1 -- 172.21.15.116:0/4005940028 learned_addr learned my addr 172.21.15.116:0/4005940028 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:26.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.490+0000 7f6707fff640 1 --2- 172.21.15.116:0/4005940028 >> v2:172.21.15.116:3300/0 conn(0x7f67080d01e0 0x7f6708160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:26.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.490+0000 7f67077fe640 1 --2- 172.21.15.116:0/4005940028 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f670815d480 0x7f67080d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:26.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.490+0000 7f6707fff640 1 -- 172.21.15.116:0/4005940028 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f670815d480 msgr2=0x7f67080d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:26.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.490+0000 7f6707fff640 1 --2- 172.21.15.116:0/4005940028 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f670815d480 0x7f67080d0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:26.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.490+0000 7f6707fff640 1 -- 172.21.15.116:0/4005940028 >> v1:172.21.15.116:6789/0 conn(0x7f67080d1310 legacy=0x7f67080cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:26.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.490+0000 7f6707fff640 1 -- 172.21.15.116:0/4005940028 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670815e660 con 0x7f67080d01e0 2023-11-28T17:11:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.491+0000 7f6707fff640 1 --2- 172.21.15.116:0/4005940028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67080d01e0 0x7f6708160270 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6700009d90 tx=0x7f670002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422c00d3a525c21c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.492+0000 7f67067fc640 1 -- 172.21.15.116:0/4005940028 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670002cc50 con 0x7f67080d01e0 2023-11-28T17:11:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.492+0000 7f67067fc640 1 -- 172.21.15.116:0/4005940028 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670002cdb0 con 0x7f67080d01e0 2023-11-28T17:11:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.492+0000 7f67067fc640 1 -- 172.21.15.116:0/4005940028 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6700034660 con 0x7f67080d01e0 2023-11-28T17:11:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.492+0000 7f670df78640 1 -- 172.21.15.116:0/4005940028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67080d01e0 msgr2=0x7f6708160270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.492+0000 7f670df78640 1 --2- 172.21.15.116:0/4005940028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67080d01e0 0x7f6708160270 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6700009d90 tx=0x7f670002bea0 comp rx=0 tx=0).stop 2023-11-28T17:11:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.492+0000 7f670df78640 1 -- 172.21.15.116:0/4005940028 shutdown_connections 2023-11-28T17:11:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.492+0000 7f670df78640 1 --2- 172.21.15.116:0/4005940028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67080d01e0 0x7f6708160270 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:26.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.492+0000 7f670df78640 1 --2- 172.21.15.116:0/4005940028 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f670815d480 0x7f67080d0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:26.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.492+0000 7f670df78640 1 -- 172.21.15.116:0/4005940028 >> 172.21.15.116:0/4005940028 conn(0x7f670809c540 msgr2=0x7f67080d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:26.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.492+0000 7f670df78640 1 -- 172.21.15.116:0/4005940028 shutdown_connections 2023-11-28T17:11:26.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.493+0000 7f670df78640 1 -- 172.21.15.116:0/4005940028 wait complete. 2023-11-28T17:11:26.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.495+0000 7f670df78640 1 Processor -- start 2023-11-28T17:11:26.496 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.495+0000 7f670df78640 1 -- start start 2023-11-28T17:11:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.496+0000 7f670df78640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67080d01e0 0x7f67080c6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.496+0000 7f670df78640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67080d1310 0x7f67080c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.496+0000 7f670df78640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f670815d480 0x7f67080b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.496+0000 7f670df78640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67080c8410 con 0x7f67080d01e0 2023-11-28T17:11:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.496+0000 7f670df78640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6708156130 con 0x7f670815d480 2023-11-28T17:11:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.496+0000 7f670df78640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67080c74d0 con 0x7f67080d1310 2023-11-28T17:11:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.496+0000 7f67077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67080d01e0 0x7f67080c6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.496+0000 7f67077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67080d01e0 0x7f67080c6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34010/0 (socket says 172.21.15.116:34010) 2023-11-28T17:11:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.496+0000 7f67077fe640 1 -- 172.21.15.116:0/2757593337 learned_addr learned my addr 172.21.15.116:0/2757593337 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.497+0000 7f67077fe640 1 -- 172.21.15.116:0/2757593337 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67080d1310 msgr2=0x7f67080c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.497+0000 7f67077fe640 1 --2- 172.21.15.116:0/2757593337 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67080d1310 0x7f67080c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.497+0000 7f67077fe640 1 -- 172.21.15.116:0/2757593337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f670815d480 msgr2=0x7f67080b1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.497+0000 7f67077fe640 1 --2- 172.21.15.116:0/2757593337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f670815d480 0x7f67080b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.497+0000 7f67077fe640 1 -- 172.21.15.116:0/2757593337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6700009a70 con 0x7f67080d01e0 2023-11-28T17:11:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.497+0000 7f67077fe640 1 --2- 172.21.15.116:0/2757593337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67080d01e0 0x7f67080c6f90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f66f000d5f0 tx=0x7f66f000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.497+0000 7f66fffff640 1 -- 172.21.15.116:0/2757593337 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f00125f0 con 0x7f67080d01e0 2023-11-28T17:11:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.497+0000 7f66fffff640 1 -- 172.21.15.116:0/2757593337 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f0012750 con 0x7f67080d01e0 2023-11-28T17:11:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.497+0000 7f66fffff640 1 -- 172.21.15.116:0/2757593337 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f0012b00 con 0x7f67080d01e0 2023-11-28T17:11:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.497+0000 7f670df78640 1 -- 172.21.15.116:0/2757593337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67080c7790 con 0x7f67080d01e0 2023-11-28T17:11:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.497+0000 7f670df78640 1 -- 172.21.15.116:0/2757593337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67080c7d50 con 0x7f67080d01e0 2023-11-28T17:11:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.499+0000 7f670df78640 1 -- 172.21.15.116:0/2757593337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6708067980 con 0x7f67080d01e0 2023-11-28T17:11:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.499+0000 7f66fffff640 1 -- 172.21.15.116:0/2757593337 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f66f0012c60 con 0x7f67080d01e0 2023-11-28T17:11:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.500+0000 7f66fffff640 1 --2- 172.21.15.116:0/2757593337 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f66e806b2a0 0x7f66e806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.500+0000 7f66fffff640 1 -- 172.21.15.116:0/2757593337 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66f008a0b0 con 0x7f67080d01e0 2023-11-28T17:11:26.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.501+0000 7f6706ffd640 1 --2- 172.21.15.116:0/2757593337 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f66e806b2a0 0x7f66e806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:26.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.501+0000 7f6706ffd640 1 --2- 172.21.15.116:0/2757593337 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f66e806b2a0 0x7f66e806d760 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f67080d2770 tx=0x7f66f8007a40 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:26.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.505+0000 7f66fffff640 1 -- 172.21.15.116:0/2757593337 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66f0054f10 con 0x7f67080d01e0 2023-11-28T17:11:26.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:26.729+0000 7f670df78640 1 -- 172.21.15.116:0/2757593337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6708069360 con 0x7f67080d01e0 2023-11-28T17:11:27.034 INFO:tasks.ceph.mds.c.smithi116.stderr:2023-11-28T17:11:27.033+0000 7fec22abf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-28T17:11:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.244+0000 7f66fffff640 1 -- 172.21.15.116:0/2757593337 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f66f0058530 con 0x7f67080d01e0 2023-11-28T17:11:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.247+0000 7f670df78640 1 -- 172.21.15.116:0/2757593337 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f66e806b2a0 msgr2=0x7f66e806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.247+0000 7f670df78640 1 --2- 172.21.15.116:0/2757593337 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f66e806b2a0 0x7f66e806d760 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f67080d2770 tx=0x7f66f8007a40 comp rx=0 tx=0).stop 2023-11-28T17:11:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.247+0000 7f670df78640 1 -- 172.21.15.116:0/2757593337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67080d01e0 msgr2=0x7f67080c6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.247+0000 7f670df78640 1 --2- 172.21.15.116:0/2757593337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67080d01e0 0x7f67080c6f90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f66f000d5f0 tx=0x7f66f000dab0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.248+0000 7f670df78640 1 -- 172.21.15.116:0/2757593337 shutdown_connections 2023-11-28T17:11:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.248+0000 7f670df78640 1 --2- 172.21.15.116:0/2757593337 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f66e806b2a0 0x7f66e806d760 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.248+0000 7f670df78640 1 --2- 172.21.15.116:0/2757593337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f670815d480 0x7f67080b1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.248+0000 7f670df78640 1 --2- 172.21.15.116:0/2757593337 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f67080d1310 0x7f67080c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.248+0000 7f670df78640 1 --2- 172.21.15.116:0/2757593337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67080d01e0 0x7f67080c6f90 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.248+0000 7f670df78640 1 -- 172.21.15.116:0/2757593337 >> 172.21.15.116:0/2757593337 conn(0x7f670809c540 msgr2=0x7f67080b5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.248+0000 7f670df78640 1 -- 172.21.15.116:0/2757593337 shutdown_connections 2023-11-28T17:11:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.248+0000 7f670df78640 1 -- 172.21.15.116:0/2757593337 wait complete. 2023-11-28T17:11:27.269 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-28T17:11:27.270 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:11:27.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.475+0000 7f9db45ce640 1 Processor -- start 2023-11-28T17:11:27.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.475+0000 7f9db45ce640 1 -- start start 2023-11-28T17:11:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.477+0000 7f9db45ce640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9dac1401c0 0x7f9dac071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.477+0000 7f9db45ce640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9dac071710 0x7f9dac06c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.477+0000 7f9db45ce640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dac1405a0 con 0x7f9dac06c790 2023-11-28T17:11:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.477+0000 7f9db45ce640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dac06cba0 con 0x7f9dac1401c0 2023-11-28T17:11:27.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.477+0000 7f9db45ce640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dac140880 con 0x7f9dac071710 2023-11-28T17:11:27.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.477+0000 7f9db236a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9dac1401c0 0x7f9dac071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:27.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.477+0000 7f9db2b6b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9dac06c790 0x7f9dac0703f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53146/0 (socket says 172.21.15.116:53146) 2023-11-28T17:11:27.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.477+0000 7f9db2b6b640 1 -- 172.21.15.116:0/461268635 learned_addr learned my addr 172.21.15.116:0/461268635 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:27.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.477+0000 7f9db1b69640 1 --2- 172.21.15.116:0/461268635 >> v2:172.21.15.156:3300/0 conn(0x7f9dac071710 0x7f9dac06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:27.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.478+0000 7f9db236a640 1 -- 172.21.15.116:0/461268635 >> v2:172.21.15.156:3300/0 conn(0x7f9dac071710 msgr2=0x7f9dac06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:27.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.478+0000 7f9db236a640 1 --2- 172.21.15.116:0/461268635 >> v2:172.21.15.156:3300/0 conn(0x7f9dac071710 0x7f9dac06c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.478+0000 7f9db236a640 1 -- 172.21.15.116:0/461268635 >> v1:172.21.15.116:6789/0 conn(0x7f9dac06c790 legacy=0x7f9dac0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:27.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.478+0000 7f9db236a640 1 -- 172.21.15.116:0/461268635 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac15db00 con 0x7f9dac1401c0 2023-11-28T17:11:27.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.478+0000 7f9db1b69640 1 --2- 172.21.15.116:0/461268635 >> v2:172.21.15.156:3300/0 conn(0x7f9dac071710 0x7f9dac06c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:27.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.478+0000 7f9db1368640 1 -- 172.21.15.116:0/461268635 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525392457 0 0) 0x7f9dac1405a0 con 0x7f9dac06c790 2023-11-28T17:11:27.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.479+0000 7f9db236a640 1 --2- 172.21.15.116:0/461268635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dac1401c0 0x7f9dac071eb0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9dac068020 tx=0x7f9d9c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440a2179da102602 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:27.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.479+0000 7f9db1368640 1 -- 172.21.15.116:0/461268635 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c02cc70 con 0x7f9dac1401c0 2023-11-28T17:11:27.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.479+0000 7f9db1368640 1 -- 172.21.15.116:0/461268635 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9c02cdd0 con 0x7f9dac1401c0 2023-11-28T17:11:27.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.479+0000 7f9db1368640 1 -- 172.21.15.116:0/461268635 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c034660 con 0x7f9dac1401c0 2023-11-28T17:11:27.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.480+0000 7f9db45ce640 1 -- 172.21.15.116:0/461268635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dac1401c0 msgr2=0x7f9dac071eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:27.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.480+0000 7f9db45ce640 1 --2- 172.21.15.116:0/461268635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dac1401c0 0x7f9dac071eb0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9dac068020 tx=0x7f9d9c02bec0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.480+0000 7f9db45ce640 1 -- 172.21.15.116:0/461268635 shutdown_connections 2023-11-28T17:11:27.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.480+0000 7f9db45ce640 1 --2- 172.21.15.116:0/461268635 >> v2:172.21.15.156:3300/0 conn(0x7f9dac071710 0x7f9dac06c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.480+0000 7f9db45ce640 1 --2- 172.21.15.116:0/461268635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dac1401c0 0x7f9dac071eb0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.480+0000 7f9db45ce640 1 -- 172.21.15.116:0/461268635 >> 172.21.15.116:0/461268635 conn(0x7f9dac06a4c0 msgr2=0x7f9dac06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:27.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.480+0000 7f9db45ce640 1 -- 172.21.15.116:0/461268635 shutdown_connections 2023-11-28T17:11:27.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.480+0000 7f9db45ce640 1 -- 172.21.15.116:0/461268635 wait complete. 2023-11-28T17:11:27.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.482+0000 7f9db45ce640 1 Processor -- start 2023-11-28T17:11:27.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.482+0000 7f9db45ce640 1 -- start start 2023-11-28T17:11:27.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.483+0000 7f9db45ce640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dac06c790 0x7f9dac156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:27.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.483+0000 7f9db45ce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dac071710 0x7f9dac141920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:27.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.483+0000 7f9db45ce640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dac1401c0 0x7f9dac15ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:27.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.483+0000 7f9db45ce640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac142350 con 0x7f9dac071710 2023-11-28T17:11:27.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.483+0000 7f9db45ce640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac12f210 con 0x7f9dac06c790 2023-11-28T17:11:27.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.483+0000 7f9db45ce640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac12f4f0 con 0x7f9dac1401c0 2023-11-28T17:11:27.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.483+0000 7f9db1b69640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dac071710 0x7f9dac141920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:27.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.483+0000 7f9db1b69640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dac071710 0x7f9dac141920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34034/0 (socket says 172.21.15.116:34034) 2023-11-28T17:11:27.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.483+0000 7f9db1b69640 1 -- 172.21.15.116:0/1695677940 learned_addr learned my addr 172.21.15.116:0/1695677940 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:27.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.484+0000 7f9db236a640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dac06c790 0x7f9dac156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:27.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.484+0000 7f9db2b6b640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dac1401c0 0x7f9dac15ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:27.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.484+0000 7f9db1b69640 1 -- 172.21.15.116:0/1695677940 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dac1401c0 msgr2=0x7f9dac15ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:27.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.484+0000 7f9db1b69640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dac1401c0 0x7f9dac15ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.484+0000 7f9db1b69640 1 -- 172.21.15.116:0/1695677940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dac06c790 msgr2=0x7f9dac156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:27.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.484+0000 7f9db1b69640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dac06c790 0x7f9dac156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.484+0000 7f9db1b69640 1 -- 172.21.15.116:0/1695677940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c009a70 con 0x7f9dac071710 2023-11-28T17:11:27.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.484+0000 7f9db236a640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dac06c790 0x7f9dac156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:27.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.484+0000 7f9db1b69640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dac071710 0x7f9dac141920 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f9da000c7f0 tx=0x7f9da000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:27.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.484+0000 7f9daaffd640 1 -- 172.21.15.116:0/1695677940 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da0007820 con 0x7f9dac071710 2023-11-28T17:11:27.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.484+0000 7f9daaffd640 1 -- 172.21.15.116:0/1695677940 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da0007980 con 0x7f9dac071710 2023-11-28T17:11:27.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.484+0000 7f9daaffd640 1 -- 172.21.15.116:0/1695677940 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da0007cf0 con 0x7f9dac071710 2023-11-28T17:11:27.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.484+0000 7f9db45ce640 1 -- 172.21.15.116:0/1695677940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dac12f7d0 con 0x7f9dac071710 2023-11-28T17:11:27.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.484+0000 7f9db45ce640 1 -- 172.21.15.116:0/1695677940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dac1568c0 con 0x7f9dac071710 2023-11-28T17:11:27.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.485+0000 7f9db45ce640 1 -- 172.21.15.116:0/1695677940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dac0693d0 con 0x7f9dac071710 2023-11-28T17:11:27.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.486+0000 7f9daaffd640 1 -- 172.21.15.116:0/1695677940 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f9da00113f0 con 0x7f9dac071710 2023-11-28T17:11:27.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.487+0000 7f9daaffd640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9d9806b2a0 0x7f9d9806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:27.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.487+0000 7f9daaffd640 1 -- 172.21.15.116:0/1695677940 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9da0089ef0 con 0x7f9dac071710 2023-11-28T17:11:27.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.488+0000 7f9db236a640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9d9806b2a0 0x7f9d9806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:27.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.488+0000 7f9db236a640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9d9806b2a0 0x7f9d9806d760 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c036000 tx=0x7f9d9c030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:27.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.492+0000 7f9daaffd640 1 -- 172.21.15.116:0/1695677940 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9da0054d50 con 0x7f9dac071710 2023-11-28T17:11:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.701+0000 7f9db45ce640 1 -- 172.21.15.116:0/1695677940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dac06b4d0 con 0x7f9dac071710 2023-11-28T17:11:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.703+0000 7f9daaffd640 1 -- 172.21.15.116:0/1695677940 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4922 (secure 0 0 0) 0x7f9da001f090 con 0x7f9dac071710 2023-11-28T17:11:27.704 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:27.705 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4790,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2474839888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2474839888},{"type":"v1","addr":"172.21.15.156:6835","nonce":2474839888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":4987,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/372407661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":372407661},{"type":"v1","addr":"172.21.15.156:6833","nonce":372407661}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3251347229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3251347229},{"type":"v1","addr":"172.21.15.116:6835","nonce":3251347229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:11:23.668388+0000","modified":"2023-11-28T17:11:26.731191+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5826},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5826":{"gid":5826,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6837/1482499383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1482499383},{"type":"v1","addr":"172.21.15.116:6837","nonce":1482499383}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T17:11:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2023-11-28T17:11:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.705+0000 7f9db45ce640 1 -- 172.21.15.116:0/1695677940 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9d9806b2a0 msgr2=0x7f9d9806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.705+0000 7f9db45ce640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9d9806b2a0 0x7f9d9806d760 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c036000 tx=0x7f9d9c030000 comp rx=0 tx=0).stop 2023-11-28T17:11:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.705+0000 7f9db45ce640 1 -- 172.21.15.116:0/1695677940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dac071710 msgr2=0x7f9dac141920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.705+0000 7f9db45ce640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dac071710 0x7f9dac141920 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f9da000c7f0 tx=0x7f9da000ccb0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.705+0000 7f9db45ce640 1 -- 172.21.15.116:0/1695677940 shutdown_connections 2023-11-28T17:11:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.705+0000 7f9db45ce640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f9d9806b2a0 0x7f9d9806d760 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.705+0000 7f9db45ce640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dac1401c0 0x7f9dac15ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.705+0000 7f9db45ce640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9dac071710 0x7f9dac141920 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.705+0000 7f9db45ce640 1 --2- 172.21.15.116:0/1695677940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dac06c790 0x7f9dac156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.705+0000 7f9db45ce640 1 -- 172.21.15.116:0/1695677940 >> 172.21.15.116:0/1695677940 conn(0x7f9dac06a4c0 msgr2=0x7f9dac06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.706+0000 7f9db45ce640 1 -- 172.21.15.116:0/1695677940 shutdown_connections 2023-11-28T17:11:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.706+0000 7f9db45ce640 1 -- 172.21.15.116:0/1695677940 wait complete. 2023-11-28T17:11:27.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:11:27.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.922+0000 7ff68dd33640 1 Processor -- start 2023-11-28T17:11:27.924 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.922+0000 7ff68dd33640 1 -- start start 2023-11-28T17:11:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.924+0000 7ff68dd33640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff68815f7b0 0x7ff688071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.924+0000 7ff68dd33640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff6880721b0 0x7ff68806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.924+0000 7ff68dd33640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff68815fc10 con 0x7ff688144430 2023-11-28T17:11:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.924+0000 7ff68dd33640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff688144810 con 0x7ff6880721b0 2023-11-28T17:11:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.924+0000 7ff68dd33640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff688070bd0 con 0x7ff68815f7b0 2023-11-28T17:11:27.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.925+0000 7ff686ffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff68815f7b0 0x7ff688071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:27.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.925+0000 7ff687fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff6880721b0 0x7ff68806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:27.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.925+0000 7ff686ffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff68815f7b0 0x7ff688071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:46584/0 (socket says 172.21.15.116:46584) 2023-11-28T17:11:27.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.925+0000 7ff686ffd640 1 -- 172.21.15.116:0/3855122554 learned_addr learned my addr 172.21.15.116:0/3855122554 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:27.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.925+0000 7ff687fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff6880721b0 0x7ff68806c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:36750/0 (socket says 172.21.15.116:36750) 2023-11-28T17:11:27.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.925+0000 7ff686ffd640 1 -- 172.21.15.116:0/3855122554 >> v2:172.21.15.156:3300/0 conn(0x7ff6880721b0 msgr2=0x7ff68806c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:27.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.925+0000 7ff686ffd640 1 --2- 172.21.15.116:0/3855122554 >> v2:172.21.15.156:3300/0 conn(0x7ff6880721b0 0x7ff68806c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.925+0000 7ff686ffd640 1 -- 172.21.15.116:0/3855122554 >> v1:172.21.15.156:6789/0 conn(0x7ff688144430 legacy=0x7ff6880703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:27.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.925+0000 7ff686ffd640 1 -- 172.21.15.116:0/3855122554 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff688070eb0 con 0x7ff68815f7b0 2023-11-28T17:11:27.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.927+0000 7ff686ffd640 1 --2- 172.21.15.116:0/3855122554 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff68815f7b0 0x7ff688071c70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009870 tx=0x7ff67c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1edace44e304f60 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:27.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.927+0000 7ff6867fc640 1 -- 172.21.15.116:0/3855122554 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c030840 con 0x7ff68815f7b0 2023-11-28T17:11:27.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.927+0000 7ff6867fc640 1 -- 172.21.15.116:0/3855122554 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67c0309a0 con 0x7ff68815f7b0 2023-11-28T17:11:27.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.927+0000 7ff6867fc640 1 -- 172.21.15.116:0/3855122554 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c030d30 con 0x7ff68815f7b0 2023-11-28T17:11:27.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.928+0000 7ff68dd33640 1 -- 172.21.15.116:0/3855122554 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff68815f7b0 msgr2=0x7ff688071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:27.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.928+0000 7ff68dd33640 1 --2- 172.21.15.116:0/3855122554 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff68815f7b0 0x7ff688071c70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009870 tx=0x7ff67c02b980 comp rx=0 tx=0).stop 2023-11-28T17:11:27.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.928+0000 7ff68dd33640 1 -- 172.21.15.116:0/3855122554 shutdown_connections 2023-11-28T17:11:27.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.928+0000 7ff68dd33640 1 --2- 172.21.15.116:0/3855122554 >> v2:172.21.15.156:3300/0 conn(0x7ff6880721b0 0x7ff68806c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.928+0000 7ff68dd33640 1 --2- 172.21.15.116:0/3855122554 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff68815f7b0 0x7ff688071c70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.928+0000 7ff68dd33640 1 -- 172.21.15.116:0/3855122554 >> 172.21.15.116:0/3855122554 conn(0x7ff68806a4c0 msgr2=0x7ff68806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:27.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.928+0000 7ff68dd33640 1 -- 172.21.15.116:0/3855122554 shutdown_connections 2023-11-28T17:11:27.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.928+0000 7ff68dd33640 1 -- 172.21.15.116:0/3855122554 wait complete. 2023-11-28T17:11:27.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.930+0000 7ff68dd33640 1 Processor -- start 2023-11-28T17:11:27.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.931+0000 7ff68dd33640 1 -- start start 2023-11-28T17:11:27.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.932+0000 7ff68dd33640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6880721b0 0x7ff688140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:27.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.932+0000 7ff68dd33640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff688144430 0x7ff6880c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:27.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.932+0000 7ff68dd33640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff68815f7b0 0x7ff6880a9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:27.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.932+0000 7ff68dd33640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6880c8bf0 con 0x7ff688144430 2023-11-28T17:11:27.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.932+0000 7ff68dd33640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6880c8760 con 0x7ff6880721b0 2023-11-28T17:11:27.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.932+0000 7ff68dd33640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6880c9c60 con 0x7ff68815f7b0 2023-11-28T17:11:27.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.932+0000 7ff686ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff688144430 0x7ff6880c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:27.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.932+0000 7ff686ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff688144430 0x7ff6880c9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33876/0 (socket says 172.21.15.116:33876) 2023-11-28T17:11:27.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.932+0000 7ff686ffd640 1 -- 172.21.15.116:0/2539248132 learned_addr learned my addr 172.21.15.116:0/2539248132 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:27.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.933+0000 7ff687fff640 1 --2- 172.21.15.116:0/2539248132 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff68815f7b0 0x7ff6880a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:27.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.933+0000 7ff6877fe640 1 --2- 172.21.15.116:0/2539248132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6880721b0 0x7ff688140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:27.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.933+0000 7ff686ffd640 1 -- 172.21.15.116:0/2539248132 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff68815f7b0 msgr2=0x7ff6880a9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:27.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.933+0000 7ff686ffd640 1 --2- 172.21.15.116:0/2539248132 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff68815f7b0 0x7ff6880a9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.933+0000 7ff686ffd640 1 -- 172.21.15.116:0/2539248132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6880721b0 msgr2=0x7ff688140e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:27.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.933+0000 7ff686ffd640 1 --2- 172.21.15.116:0/2539248132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6880721b0 0x7ff688140e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:27.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.933+0000 7ff686ffd640 1 -- 172.21.15.116:0/2539248132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c0094d0 con 0x7ff688144430 2023-11-28T17:11:27.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.933+0000 7ff686ffd640 1 --2- 172.21.15.116:0/2539248132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff688144430 0x7ff6880c9720 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff67c000c00 tx=0x7ff67c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:27.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.933+0000 7ff66ffff640 1 -- 172.21.15.116:0/2539248132 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c039a90 con 0x7ff688144430 2023-11-28T17:11:27.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.933+0000 7ff66ffff640 1 -- 172.21.15.116:0/2539248132 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67c039bf0 con 0x7ff688144430 2023-11-28T17:11:27.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.933+0000 7ff68dd33640 1 -- 172.21.15.116:0/2539248132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6880c7b50 con 0x7ff688144430 2023-11-28T17:11:27.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.934+0000 7ff66ffff640 1 -- 172.21.15.116:0/2539248132 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c043410 con 0x7ff688144430 2023-11-28T17:11:27.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.934+0000 7ff68dd33640 1 -- 172.21.15.116:0/2539248132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6880c8110 con 0x7ff688144430 2023-11-28T17:11:27.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.935+0000 7ff68dd33640 1 -- 172.21.15.116:0/2539248132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6880693d0 con 0x7ff688144430 2023-11-28T17:11:27.937 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.936+0000 7ff66ffff640 1 -- 172.21.15.116:0/2539248132 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff67c02f390 con 0x7ff688144430 2023-11-28T17:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.937+0000 7ff66ffff640 1 --2- 172.21.15.116:0/2539248132 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff67406b2a0 0x7ff67406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.937+0000 7ff66ffff640 1 -- 172.21.15.116:0/2539248132 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff67c0abc80 con 0x7ff688144430 2023-11-28T17:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.937+0000 7ff6877fe640 1 --2- 172.21.15.116:0/2539248132 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff67406b2a0 0x7ff67406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.938+0000 7ff6877fe640 1 --2- 172.21.15.116:0/2539248132 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff67406b2a0 0x7ff67406d760 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff678000f60 tx=0x7ff678009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:27.943 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:27.942+0000 7ff66ffff640 1 -- 172.21.15.116:0/2539248132 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff67c076c00 con 0x7ff688144430 2023-11-28T17:11:28.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.133+0000 7ff68dd33640 1 -- 172.21.15.116:0/2539248132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff68806d100 con 0x7ff688144430 2023-11-28T17:11:28.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.136+0000 7ff66ffff640 1 -- 172.21.15.116:0/2539248132 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16521 (secure 0 0 0) 0x7ff67c07a220 con 0x7ff688144430 2023-11-28T17:11:28.138 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:28.138 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":48,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:11:25.192837+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:11:21.518078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T17:11:22.672451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:11:28.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.138+0000 7ff68dd33640 1 -- 172.21.15.116:0/2539248132 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff67406b2a0 msgr2=0x7ff67406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:28.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.138+0000 7ff68dd33640 1 --2- 172.21.15.116:0/2539248132 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff67406b2a0 0x7ff67406d760 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff678000f60 tx=0x7ff678009040 comp rx=0 tx=0).stop 2023-11-28T17:11:28.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.138+0000 7ff68dd33640 1 -- 172.21.15.116:0/2539248132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff688144430 msgr2=0x7ff6880c9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:28.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.138+0000 7ff68dd33640 1 --2- 172.21.15.116:0/2539248132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff688144430 0x7ff6880c9720 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff67c000c00 tx=0x7ff67c004730 comp rx=0 tx=0).stop 2023-11-28T17:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.139+0000 7ff68dd33640 1 -- 172.21.15.116:0/2539248132 shutdown_connections 2023-11-28T17:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.139+0000 7ff68dd33640 1 --2- 172.21.15.116:0/2539248132 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff67406b2a0 0x7ff67406d760 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.139+0000 7ff68dd33640 1 --2- 172.21.15.116:0/2539248132 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff68815f7b0 0x7ff6880a9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.139+0000 7ff68dd33640 1 --2- 172.21.15.116:0/2539248132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff688144430 0x7ff6880c9720 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.139+0000 7ff68dd33640 1 --2- 172.21.15.116:0/2539248132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6880721b0 0x7ff688140e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.139+0000 7ff68dd33640 1 -- 172.21.15.116:0/2539248132 >> 172.21.15.116:0/2539248132 conn(0x7ff68806a4c0 msgr2=0x7ff68815d150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.139+0000 7ff68dd33640 1 -- 172.21.15.116:0/2539248132 shutdown_connections 2023-11-28T17:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.139+0000 7ff68dd33640 1 -- 172.21.15.116:0/2539248132 wait complete. 2023-11-28T17:11:28.159 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-28T17:11:28.159 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:11:28.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.371+0000 7f5d5c389640 1 Processor -- start 2023-11-28T17:11:28.372 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.371+0000 7f5d5c389640 1 -- start start 2023-11-28T17:11:28.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.373+0000 7f5d5c389640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5d5415f7b0 0x7f5d54071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:28.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.373+0000 7f5d5c389640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5415fc10 con 0x7f5d54144430 2023-11-28T17:11:28.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.373+0000 7f5d5c389640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d54070bd0 con 0x7f5d540721b0 2023-11-28T17:11:28.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.373+0000 7f5d5c389640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d54144810 con 0x7f5d5415f7b0 2023-11-28T17:11:28.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.373+0000 7f5d5a125640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5d54144430 0x7f5d540703f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45218/0 (socket says 172.21.15.116:45218) 2023-11-28T17:11:28.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.373+0000 7f5d59924640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5d5415f7b0 0x7f5d54071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:28.374 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.373+0000 7f5d5a125640 1 -- 172.21.15.116:0/4171763041 learned_addr learned my addr 172.21.15.116:0/4171763041 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:28.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.373+0000 7f5d59924640 1 -- 172.21.15.116:0/4171763041 >> v1:172.21.15.156:6789/0 conn(0x7f5d540721b0 legacy=0x7f5d5406c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:28.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.373+0000 7f5d59924640 1 -- 172.21.15.116:0/4171763041 >> v1:172.21.15.116:6789/0 conn(0x7f5d54144430 legacy=0x7f5d540703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:28.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.373+0000 7f5d59924640 1 -- 172.21.15.116:0/4171763041 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d54070eb0 con 0x7f5d5415f7b0 2023-11-28T17:11:28.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.374+0000 7f5d59924640 1 --2- 172.21.15.116:0/4171763041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d5415f7b0 0x7f5d54071c70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5d440097a0 tx=0x7f5d4402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d30cd3909202f21 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:28.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.375+0000 7f5d59123640 1 -- 172.21.15.116:0/4171763041 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44031830 con 0x7f5d5415f7b0 2023-11-28T17:11:28.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.375+0000 7f5d59123640 1 -- 172.21.15.116:0/4171763041 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d44031990 con 0x7f5d5415f7b0 2023-11-28T17:11:28.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.375+0000 7f5d59123640 1 -- 172.21.15.116:0/4171763041 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44031d20 con 0x7f5d5415f7b0 2023-11-28T17:11:28.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.375+0000 7f5d5c389640 1 -- 172.21.15.116:0/4171763041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d5415f7b0 msgr2=0x7f5d54071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:28.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.375+0000 7f5d5c389640 1 --2- 172.21.15.116:0/4171763041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d5415f7b0 0x7f5d54071c70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5d440097a0 tx=0x7f5d4402b8d0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.376+0000 7f5d5c389640 1 -- 172.21.15.116:0/4171763041 shutdown_connections 2023-11-28T17:11:28.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.376+0000 7f5d5c389640 1 --2- 172.21.15.116:0/4171763041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d5415f7b0 0x7f5d54071c70 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.376+0000 7f5d5c389640 1 -- 172.21.15.116:0/4171763041 >> 172.21.15.116:0/4171763041 conn(0x7f5d5406a4c0 msgr2=0x7f5d5406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:28.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.376+0000 7f5d5c389640 1 -- 172.21.15.116:0/4171763041 shutdown_connections 2023-11-28T17:11:28.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.376+0000 7f5d5c389640 1 -- 172.21.15.116:0/4171763041 wait complete. 2023-11-28T17:11:28.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.378+0000 7f5d5c389640 1 Processor -- start 2023-11-28T17:11:28.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.378+0000 7f5d5c389640 1 -- start start 2023-11-28T17:11:28.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.379+0000 7f5d5c389640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d540721b0 0x7f5d540b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:28.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.379+0000 7f5d5c389640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d54144430 0x7f5d54132080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:28.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.379+0000 7f5d5c389640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d5415f7b0 0x7f5d540cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:28.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.379+0000 7f5d5c389640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d540b0b40 con 0x7f5d5415f7b0 2023-11-28T17:11:28.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.379+0000 7f5d5c389640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d541e6a20 con 0x7f5d540721b0 2023-11-28T17:11:28.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.379+0000 7f5d5c389640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d541325c0 con 0x7f5d54144430 2023-11-28T17:11:28.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.379+0000 7f5d5a926640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d5415f7b0 0x7f5d540cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:28.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.379+0000 7f5d5a926640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d5415f7b0 0x7f5d540cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33888/0 (socket says 172.21.15.116:33888) 2023-11-28T17:11:28.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.379+0000 7f5d5a926640 1 -- 172.21.15.116:0/211641372 learned_addr learned my addr 172.21.15.116:0/211641372 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:28.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.380+0000 7f5d5a125640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d540721b0 0x7f5d540b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:28.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.380+0000 7f5d59924640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d54144430 0x7f5d54132080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:28.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.380+0000 7f5d5a926640 1 -- 172.21.15.116:0/211641372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d54144430 msgr2=0x7f5d54132080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:28.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.380+0000 7f5d5a926640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d54144430 0x7f5d54132080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.380+0000 7f5d5a926640 1 -- 172.21.15.116:0/211641372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d540721b0 msgr2=0x7f5d540b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:28.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.380+0000 7f5d5a926640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d540721b0 0x7f5d540b0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.380+0000 7f5d5a926640 1 -- 172.21.15.116:0/211641372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d44009400 con 0x7f5d5415f7b0 2023-11-28T17:11:28.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.380+0000 7f5d59924640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d54144430 0x7f5d54132080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:28.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.380+0000 7f5d5a926640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d5415f7b0 0x7f5d540cc2c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c00c780 tx=0x7f5d4c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:28.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.380+0000 7f5d4affd640 1 -- 172.21.15.116:0/211641372 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c003d80 con 0x7f5d5415f7b0 2023-11-28T17:11:28.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.380+0000 7f5d4affd640 1 -- 172.21.15.116:0/211641372 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4c00ce30 con 0x7f5d5415f7b0 2023-11-28T17:11:28.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.380+0000 7f5d5c389640 1 -- 172.21.15.116:0/211641372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d541e6c70 con 0x7f5d5415f7b0 2023-11-28T17:11:28.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.381+0000 7f5d4affd640 1 -- 172.21.15.116:0/211641372 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c0052b0 con 0x7f5d5415f7b0 2023-11-28T17:11:28.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.381+0000 7f5d5c389640 1 -- 172.21.15.116:0/211641372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5412f680 con 0x7f5d5415f7b0 2023-11-28T17:11:28.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.382+0000 7f5d5c389640 1 -- 172.21.15.116:0/211641372 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d540693d0 con 0x7f5d5415f7b0 2023-11-28T17:11:28.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.383+0000 7f5d4affd640 1 -- 172.21.15.116:0/211641372 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f5d4c005410 con 0x7f5d5415f7b0 2023-11-28T17:11:28.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.384+0000 7f5d4affd640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5d3806b2a0 0x7f5d3806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:28.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.384+0000 7f5d4affd640 1 -- 172.21.15.116:0/211641372 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d4c089400 con 0x7f5d5415f7b0 2023-11-28T17:11:28.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.384+0000 7f5d5a125640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5d3806b2a0 0x7f5d3806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:28.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.385+0000 7f5d5a125640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5d3806b2a0 0x7f5d3806d760 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5d50005880 tx=0x7f5d50009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.388+0000 7f5d4affd640 1 -- 172.21.15.116:0/211641372 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d4c0541d0 con 0x7f5d5415f7b0 2023-11-28T17:11:28.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.574+0000 7f5d5c389640 1 -- 172.21.15.116:0/211641372 --> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5d54071750 con 0x7f5d3806b2a0 2023-11-28T17:11:28.588 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.586+0000 7f5d4affd640 1 -- 172.21.15.116:0/211641372 <== mgr.4098 v2:172.21.15.116:6832/1077400178 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286799 (secure 0 0 0) 0x7f5d50001c50 con 0x7f5d3806b2a0 2023-11-28T17:11:28.592 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:28.593 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-28T17:11:28.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.593+0000 7f5d5c389640 1 -- 172.21.15.116:0/211641372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5d3806b2a0 msgr2=0x7f5d3806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:28.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.593+0000 7f5d5c389640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5d3806b2a0 0x7f5d3806d760 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5d50005880 tx=0x7f5d50009040 comp rx=0 tx=0).stop 2023-11-28T17:11:28.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.593+0000 7f5d5c389640 1 -- 172.21.15.116:0/211641372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d5415f7b0 msgr2=0x7f5d540cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:28.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.593+0000 7f5d5c389640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d5415f7b0 0x7f5d540cc2c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c00c780 tx=0x7f5d4c00cc40 comp rx=0 tx=0).stop 2023-11-28T17:11:28.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.593+0000 7f5d5c389640 1 -- 172.21.15.116:0/211641372 shutdown_connections 2023-11-28T17:11:28.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.593+0000 7f5d5c389640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f5d3806b2a0 0x7f5d3806d760 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.593+0000 7f5d5c389640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5d5415f7b0 0x7f5d540cc2c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.593+0000 7f5d5c389640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d54144430 0x7f5d54132080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.593+0000 7f5d5c389640 1 --2- 172.21.15.116:0/211641372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d540721b0 0x7f5d540b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.593+0000 7f5d5c389640 1 -- 172.21.15.116:0/211641372 >> 172.21.15.116:0/211641372 conn(0x7f5d5406a4c0 msgr2=0x7f5d540b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:28.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.593+0000 7f5d5c389640 1 -- 172.21.15.116:0/211641372 shutdown_connections 2023-11-28T17:11:28.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.593+0000 7f5d5c389640 1 -- 172.21.15.116:0/211641372 wait complete. 2023-11-28T17:11:28.613 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2023-11-28T17:11:27.084886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238212,"kb_used_data":5384,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736508,"statfs":{"total":773094113280,"available":772850184192,"internally_reserved":0,"allocated":5513216,"data_stored":2935416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.908359"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.227473+0000","last_change":"2023-11-28T17:11:24.226060+0000","last_active":"2023-11-28T17:11:24.227473+0000","last_peered":"2023-11-28T17:11:24.227473+0000","last_clean":"2023-11-28T17:11:24.227473+0000","last_became_active":"2023-11-28T17:11:24.225545+0000","last_became_peered":"2023-11-28T17:11:24.225545+0000","last_unstale":"2023-11-28T17:11:24.227473+0000","last_undegraded":"2023-11-28T17:11:24.227473+0000","last_fullsized":"2023-11-28T17:11:24.227473+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:53:17.135956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.247149+0000","last_change":"2023-11-28T17:11:24.237732+0000","last_active":"2023-11-28T17:11:24.247149+0000","last_peered":"2023-11-28T17:11:24.247149+0000","last_clean":"2023-11-28T17:11:24.247149+0000","last_became_active":"2023-11-28T17:11:24.237443+0000","last_became_peered":"2023-11-28T17:11:24.237443+0000","last_unstale":"2023-11-28T17:11:24.247149+0000","last_undegraded":"2023-11-28T17:11:24.247149+0000","last_fullsized":"2023-11-28T17:11:24.247149+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:51:25.683278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242194+0000","last_change":"2023-11-28T17:11:24.238573+0000","last_active":"2023-11-28T17:11:24.242194+0000","last_peered":"2023-11-28T17:11:24.242194+0000","last_clean":"2023-11-28T17:11:24.242194+0000","last_became_active":"2023-11-28T17:11:24.238159+0000","last_became_peered":"2023-11-28T17:11:24.238159+0000","last_unstale":"2023-11-28T17:11:24.242194+0000","last_undegraded":"2023-11-28T17:11:24.242194+0000","last_fullsized":"2023-11-28T17:11:24.242194+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:42.774254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.250211+0000","last_change":"2023-11-28T17:11:24.243165+0000","last_active":"2023-11-28T17:11:24.250211+0000","last_peered":"2023-11-28T17:11:24.250211+0000","last_clean":"2023-11-28T17:11:24.250211+0000","last_became_active":"2023-11-28T17:11:24.237530+0000","last_became_peered":"2023-11-28T17:11:24.237530+0000","last_unstale":"2023-11-28T17:11:24.250211+0000","last_undegraded":"2023-11-28T17:11:24.250211+0000","last_fullsized":"2023-11-28T17:11:24.250211+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:30.990592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223282+0000","last_change":"2023-11-28T17:11:24.217095+0000","last_active":"2023-11-28T17:11:24.223282+0000","last_peered":"2023-11-28T17:11:24.223282+0000","last_clean":"2023-11-28T17:11:24.223282+0000","last_became_active":"2023-11-28T17:11:24.216795+0000","last_became_peered":"2023-11-28T17:11:24.216795+0000","last_unstale":"2023-11-28T17:11:24.223282+0000","last_undegraded":"2023-11-28T17:11:24.223282+0000","last_fullsized":"2023-11-28T17:11:24.223282+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:12:49.613982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.250564+0000","last_change":"2023-11-28T17:11:24.247005+0000","last_active":"2023-11-28T17:11:24.250564+0000","last_peered":"2023-11-28T17:11:24.250564+0000","last_clean":"2023-11-28T17:11:24.250564+0000","last_became_active":"2023-11-28T17:11:24.246728+0000","last_became_peered":"2023-11-28T17:11:24.246728+0000","last_unstale":"2023-11-28T17:11:24.250564+0000","last_undegraded":"2023-11-28T17:11:24.250564+0000","last_fullsized":"2023-11-28T17:11:24.250564+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:00.298985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.241901+0000","last_change":"2023-11-28T17:11:24.236686+0000","last_active":"2023-11-28T17:11:24.241901+0000","last_peered":"2023-11-28T17:11:24.241901+0000","last_clean":"2023-11-28T17:11:24.241901+0000","last_became_active":"2023-11-28T17:11:24.236436+0000","last_became_peered":"2023-11-28T17:11:24.236436+0000","last_unstale":"2023-11-28T17:11:24.241901+0000","last_undegraded":"2023-11-28T17:11:24.241901+0000","last_fullsized":"2023-11-28T17:11:24.241901+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:13:10.490828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.250065+0000","last_change":"2023-11-28T17:11:24.244148+0000","last_active":"2023-11-28T17:11:24.250065+0000","last_peered":"2023-11-28T17:11:24.250065+0000","last_clean":"2023-11-28T17:11:24.250065+0000","last_became_active":"2023-11-28T17:11:24.243301+0000","last_became_peered":"2023-11-28T17:11:24.243301+0000","last_unstale":"2023-11-28T17:11:24.250065+0000","last_undegraded":"2023-11-28T17:11:24.250065+0000","last_fullsized":"2023-11-28T17:11:24.250065+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:51:25.683278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.249041+0000","last_change":"2023-11-28T17:11:24.241462+0000","last_active":"2023-11-28T17:11:24.249041+0000","last_peered":"2023-11-28T17:11:24.249041+0000","last_clean":"2023-11-28T17:11:24.249041+0000","last_became_active":"2023-11-28T17:11:24.241183+0000","last_became_peered":"2023-11-28T17:11:24.241183+0000","last_unstale":"2023-11-28T17:11:24.249041+0000","last_undegraded":"2023-11-28T17:11:24.249041+0000","last_fullsized":"2023-11-28T17:11:24.249041+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:12:49.613982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.241669+0000","last_change":"2023-11-28T17:11:24.234407+0000","last_active":"2023-11-28T17:11:24.241669+0000","last_peered":"2023-11-28T17:11:24.241669+0000","last_clean":"2023-11-28T17:11:24.241669+0000","last_became_active":"2023-11-28T17:11:24.233160+0000","last_became_peered":"2023-11-28T17:11:24.233160+0000","last_unstale":"2023-11-28T17:11:24.241669+0000","last_undegraded":"2023-11-28T17:11:24.241669+0000","last_fullsized":"2023-11-28T17:11:24.241669+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:55.079230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248864+0000","last_change":"2023-11-28T17:11:24.236457+0000","last_active":"2023-11-28T17:11:24.248864+0000","last_peered":"2023-11-28T17:11:24.248864+0000","last_clean":"2023-11-28T17:11:24.248864+0000","last_became_active":"2023-11-28T17:11:24.235921+0000","last_became_peered":"2023-11-28T17:11:24.235921+0000","last_unstale":"2023-11-28T17:11:24.248864+0000","last_undegraded":"2023-11-28T17:11:24.248864+0000","last_fullsized":"2023-11-28T17:11:24.248864+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:18:08.133197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223241+0000","last_change":"2023-11-28T17:11:24.212809+0000","last_active":"2023-11-28T17:11:24.223241+0000","last_peered":"2023-11-28T17:11:24.223241+0000","last_clean":"2023-11-28T17:11:24.223241+0000","last_became_active":"2023-11-28T17:11:24.212489+0000","last_became_peered":"2023-11-28T17:11:24.212489+0000","last_unstale":"2023-11-28T17:11:24.223241+0000","last_undegraded":"2023-11-28T17:11:24.223241+0000","last_fullsized":"2023-11-28T17:11:24.223241+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:22.356893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.250727+0000","last_change":"2023-11-28T17:11:24.248718+0000","last_active":"2023-11-28T17:11:24.250727+0000","last_peered":"2023-11-28T17:11:24.250727+0000","last_clean":"2023-11-28T17:11:24.250727+0000","last_became_active":"2023-11-28T17:11:24.248126+0000","last_became_peered":"2023-11-28T17:11:24.248126+0000","last_unstale":"2023-11-28T17:11:24.250727+0000","last_undegraded":"2023-11-28T17:11:24.250727+0000","last_fullsized":"2023-11-28T17:11:24.250727+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:58:02.404658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.251048+0000","last_change":"2023-11-28T17:11:24.246379+0000","last_active":"2023-11-28T17:11:24.251048+0000","last_peered":"2023-11-28T17:11:24.251048+0000","last_clean":"2023-11-28T17:11:24.251048+0000","last_became_active":"2023-11-28T17:11:24.246008+0000","last_became_peered":"2023-11-28T17:11:24.246008+0000","last_unstale":"2023-11-28T17:11:24.251048+0000","last_undegraded":"2023-11-28T17:11:24.251048+0000","last_fullsized":"2023-11-28T17:11:24.251048+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:12:49.613982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.251299+0000","last_change":"2023-11-28T17:11:24.247659+0000","last_active":"2023-11-28T17:11:24.251299+0000","last_peered":"2023-11-28T17:11:24.251299+0000","last_clean":"2023-11-28T17:11:24.251299+0000","last_became_active":"2023-11-28T17:11:24.243488+0000","last_became_peered":"2023-11-28T17:11:24.243488+0000","last_unstale":"2023-11-28T17:11:24.251299+0000","last_undegraded":"2023-11-28T17:11:24.251299+0000","last_fullsized":"2023-11-28T17:11:24.251299+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:05.450433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.247521+0000","last_change":"2023-11-28T17:11:24.239489+0000","last_active":"2023-11-28T17:11:24.247521+0000","last_peered":"2023-11-28T17:11:24.247521+0000","last_clean":"2023-11-28T17:11:24.247521+0000","last_became_active":"2023-11-28T17:11:24.239123+0000","last_became_peered":"2023-11-28T17:11:24.239123+0000","last_unstale":"2023-11-28T17:11:24.247521+0000","last_undegraded":"2023-11-28T17:11:24.247521+0000","last_fullsized":"2023-11-28T17:11:24.247521+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:58:02.404658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.224067+0000","last_change":"2023-11-28T17:11:24.219801+0000","last_active":"2023-11-28T17:11:24.224067+0000","last_peered":"2023-11-28T17:11:24.224067+0000","last_clean":"2023-11-28T17:11:24.224067+0000","last_became_active":"2023-11-28T17:11:24.218728+0000","last_became_peered":"2023-11-28T17:11:24.218728+0000","last_unstale":"2023-11-28T17:11:24.224067+0000","last_undegraded":"2023-11-28T17:11:24.224067+0000","last_fullsized":"2023-11-28T17:11:24.224067+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:32.366526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.255226+0000","last_change":"2023-11-28T17:11:24.251802+0000","last_active":"2023-11-28T17:11:24.255226+0000","last_peered":"2023-11-28T17:11:24.255226+0000","last_clean":"2023-11-28T17:11:24.255226+0000","last_became_active":"2023-11-28T17:11:24.251539+0000","last_became_peered":"2023-11-28T17:11:24.251539+0000","last_unstale":"2023-11-28T17:11:24.255226+0000","last_undegraded":"2023-11-28T17:11:24.255226+0000","last_fullsized":"2023-11-28T17:11:24.255226+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:06.002391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223422+0000","last_change":"2023-11-28T17:11:24.220740+0000","last_active":"2023-11-28T17:11:24.223422+0000","last_peered":"2023-11-28T17:11:24.223422+0000","last_clean":"2023-11-28T17:11:24.223422+0000","last_became_active":"2023-11-28T17:11:24.220535+0000","last_became_peered":"2023-11-28T17:11:24.220535+0000","last_unstale":"2023-11-28T17:11:24.223422+0000","last_undegraded":"2023-11-28T17:11:24.223422+0000","last_fullsized":"2023-11-28T17:11:24.223422+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:51:25.683278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242385+0000","last_change":"2023-11-28T17:11:24.239214+0000","last_active":"2023-11-28T17:11:24.242385+0000","last_peered":"2023-11-28T17:11:24.242385+0000","last_clean":"2023-11-28T17:11:24.242385+0000","last_became_active":"2023-11-28T17:11:24.238992+0000","last_became_peered":"2023-11-28T17:11:24.238992+0000","last_unstale":"2023-11-28T17:11:24.242385+0000","last_undegraded":"2023-11-28T17:11:24.242385+0000","last_fullsized":"2023-11-28T17:11:24.242385+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:10:44.265974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.249849+0000","last_change":"2023-11-28T17:11:24.236502+0000","last_active":"2023-11-28T17:11:24.249849+0000","last_peered":"2023-11-28T17:11:24.249849+0000","last_clean":"2023-11-28T17:11:24.249849+0000","last_became_active":"2023-11-28T17:11:24.235482+0000","last_became_peered":"2023-11-28T17:11:24.235482+0000","last_unstale":"2023-11-28T17:11:24.249849+0000","last_undegraded":"2023-11-28T17:11:24.249849+0000","last_fullsized":"2023-11-28T17:11:24.249849+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:22.356893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.227177+0000","last_change":"2023-11-28T17:11:24.219330+0000","last_active":"2023-11-28T17:11:24.227177+0000","last_peered":"2023-11-28T17:11:24.227177+0000","last_clean":"2023-11-28T17:11:24.227177+0000","last_became_active":"2023-11-28T17:11:24.217191+0000","last_became_peered":"2023-11-28T17:11:24.217191+0000","last_unstale":"2023-11-28T17:11:24.227177+0000","last_undegraded":"2023-11-28T17:11:24.227177+0000","last_fullsized":"2023-11-28T17:11:24.227177+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:41:38.593938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.217525+0000","last_change":"2023-11-28T17:11:24.213940+0000","last_active":"2023-11-28T17:11:24.217525+0000","last_peered":"2023-11-28T17:11:24.217525+0000","last_clean":"2023-11-28T17:11:24.217525+0000","last_became_active":"2023-11-28T17:11:24.212127+0000","last_became_peered":"2023-11-28T17:11:24.212127+0000","last_unstale":"2023-11-28T17:11:24.217525+0000","last_undegraded":"2023-11-28T17:11:24.217525+0000","last_fullsized":"2023-11-28T17:11:24.217525+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:26:13.848230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248045+0000","last_change":"2023-11-28T17:11:24.244113+0000","last_active":"2023-11-28T17:11:24.248045+0000","last_peered":"2023-11-28T17:11:24.248045+0000","last_clean":"2023-11-28T17:11:24.248045+0000","last_became_active":"2023-11-28T17:11:24.243833+0000","last_became_peered":"2023-11-28T17:11:24.243833+0000","last_unstale":"2023-11-28T17:11:24.248045+0000","last_undegraded":"2023-11-28T17:11:24.248045+0000","last_fullsized":"2023-11-28T17:11:24.248045+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:58:02.404658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223423+0000","last_change":"2023-11-28T17:11:24.219782+0000","last_active":"2023-11-28T17:11:24.223423+0000","last_peered":"2023-11-28T17:11:24.223423+0000","last_clean":"2023-11-28T17:11:24.223423+0000","last_became_active":"2023-11-28T17:11:24.219209+0000","last_became_peered":"2023-11-28T17:11:24.219209+0000","last_unstale":"2023-11-28T17:11:24.223423+0000","last_undegraded":"2023-11-28T17:11:24.223423+0000","last_fullsized":"2023-11-28T17:11:24.223423+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:30:34.918467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.252215+0000","last_change":"2023-11-28T17:11:24.250257+0000","last_active":"2023-11-28T17:11:24.252215+0000","last_peered":"2023-11-28T17:11:24.252215+0000","last_clean":"2023-11-28T17:11:24.252215+0000","last_became_active":"2023-11-28T17:11:24.245657+0000","last_became_peered":"2023-11-28T17:11:24.245657+0000","last_unstale":"2023-11-28T17:11:24.252215+0000","last_undegraded":"2023-11-28T17:11:24.252215+0000","last_fullsized":"2023-11-28T17:11:24.252215+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:32.366526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223816+0000","last_change":"2023-11-28T17:11:24.222198+0000","last_active":"2023-11-28T17:11:24.223816+0000","last_peered":"2023-11-28T17:11:24.223816+0000","last_clean":"2023-11-28T17:11:24.223816+0000","last_became_active":"2023-11-28T17:11:24.218597+0000","last_became_peered":"2023-11-28T17:11:24.218597+0000","last_unstale":"2023-11-28T17:11:24.223816+0000","last_undegraded":"2023-11-28T17:11:24.223816+0000","last_fullsized":"2023-11-28T17:11:24.223816+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:28.215714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.218097+0000","last_change":"2023-11-28T17:11:24.216110+0000","last_active":"2023-11-28T17:11:24.218097+0000","last_peered":"2023-11-28T17:11:24.218097+0000","last_clean":"2023-11-28T17:11:24.218097+0000","last_became_active":"2023-11-28T17:11:24.215793+0000","last_became_peered":"2023-11-28T17:11:24.215793+0000","last_unstale":"2023-11-28T17:11:24.218097+0000","last_undegraded":"2023-11-28T17:11:24.218097+0000","last_fullsized":"2023-11-28T17:11:24.218097+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:28.388804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.239396+0000","last_change":"2023-11-28T17:11:24.235123+0000","last_active":"2023-11-28T17:11:24.239396+0000","last_peered":"2023-11-28T17:11:24.239396+0000","last_clean":"2023-11-28T17:11:24.239396+0000","last_became_active":"2023-11-28T17:11:24.234739+0000","last_became_peered":"2023-11-28T17:11:24.234739+0000","last_unstale":"2023-11-28T17:11:24.239396+0000","last_undegraded":"2023-11-28T17:11:24.239396+0000","last_fullsized":"2023-11-28T17:11:24.239396+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:05.450433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223837+0000","last_change":"2023-11-28T17:11:24.216242+0000","last_active":"2023-11-28T17:11:24.223837+0000","last_peered":"2023-11-28T17:11:24.223837+0000","last_clean":"2023-11-28T17:11:24.223837+0000","last_became_active":"2023-11-28T17:11:24.215959+0000","last_became_peered":"2023-11-28T17:11:24.215959+0000","last_unstale":"2023-11-28T17:11:24.223837+0000","last_undegraded":"2023-11-28T17:11:24.223837+0000","last_fullsized":"2023-11-28T17:11:24.223837+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:05.450433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.230359+0000","last_change":"2023-11-28T17:11:24.219508+0000","last_active":"2023-11-28T17:11:24.230359+0000","last_peered":"2023-11-28T17:11:24.230359+0000","last_clean":"2023-11-28T17:11:24.230359+0000","last_became_active":"2023-11-28T17:11:24.218725+0000","last_became_peered":"2023-11-28T17:11:24.218725+0000","last_unstale":"2023-11-28T17:11:24.230359+0000","last_undegraded":"2023-11-28T17:11:24.230359+0000","last_fullsized":"2023-11-28T17:11:24.230359+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:43:05.206928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248737+0000","last_change":"2023-11-28T17:11:24.243621+0000","last_active":"2023-11-28T17:11:24.248737+0000","last_peered":"2023-11-28T17:11:24.248737+0000","last_clean":"2023-11-28T17:11:24.248737+0000","last_became_active":"2023-11-28T17:11:24.237176+0000","last_became_peered":"2023-11-28T17:11:24.237176+0000","last_unstale":"2023-11-28T17:11:24.248737+0000","last_undegraded":"2023-11-28T17:11:24.248737+0000","last_fullsized":"2023-11-28T17:11:24.248737+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:13:37.338728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.240656+0000","last_change":"2023-11-28T17:11:24.234796+0000","last_active":"2023-11-28T17:11:24.240656+0000","last_peered":"2023-11-28T17:11:24.240656+0000","last_clean":"2023-11-28T17:11:24.240656+0000","last_became_active":"2023-11-28T17:11:24.234427+0000","last_became_peered":"2023-11-28T17:11:24.234427+0000","last_unstale":"2023-11-28T17:11:24.240656+0000","last_undegraded":"2023-11-28T17:11:24.240656+0000","last_fullsized":"2023-11-28T17:11:24.240656+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:42.774254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"47'4","reported_seq":15,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.250149+0000","last_change":"2023-11-28T17:11:23.216512+0000","last_active":"2023-11-28T17:11:24.250149+0000","last_peered":"2023-11-28T17:11:24.250149+0000","last_clean":"2023-11-28T17:11:24.250149+0000","last_became_active":"2023-11-28T17:11:23.216241+0000","last_became_peered":"2023-11-28T17:11:23.216241+0000","last_unstale":"2023-11-28T17:11:24.250149+0000","last_undegraded":"2023-11-28T17:11:24.250149+0000","last_fullsized":"2023-11-28T17:11:24.250149+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:06:02.961760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.236381+0000","last_change":"2023-11-28T17:11:24.231592+0000","last_active":"2023-11-28T17:11:24.236381+0000","last_peered":"2023-11-28T17:11:24.236381+0000","last_clean":"2023-11-28T17:11:24.236381+0000","last_became_active":"2023-11-28T17:11:24.231316+0000","last_became_peered":"2023-11-28T17:11:24.231316+0000","last_unstale":"2023-11-28T17:11:24.236381+0000","last_undegraded":"2023-11-28T17:11:24.236381+0000","last_fullsized":"2023-11-28T17:11:24.236381+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:18:08.133197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.234878+0000","last_change":"2023-11-28T17:11:23.223688+0000","last_active":"2023-11-28T17:11:24.234878+0000","last_peered":"2023-11-28T17:11:24.234878+0000","last_clean":"2023-11-28T17:11:24.234878+0000","last_became_active":"2023-11-28T17:11:23.223318+0000","last_became_peered":"2023-11-28T17:11:23.223318+0000","last_unstale":"2023-11-28T17:11:24.234878+0000","last_undegraded":"2023-11-28T17:11:24.234878+0000","last_fullsized":"2023-11-28T17:11:24.234878+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:28:09.189390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.247215+0000","last_change":"2023-11-28T17:11:24.240363+0000","last_active":"2023-11-28T17:11:24.247215+0000","last_peered":"2023-11-28T17:11:24.247215+0000","last_clean":"2023-11-28T17:11:24.247215+0000","last_became_active":"2023-11-28T17:11:24.240093+0000","last_became_peered":"2023-11-28T17:11:24.240093+0000","last_unstale":"2023-11-28T17:11:24.247215+0000","last_undegraded":"2023-11-28T17:11:24.247215+0000","last_fullsized":"2023-11-28T17:11:24.247215+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:06:03.975512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.249714+0000","last_change":"2023-11-28T17:11:23.215395+0000","last_active":"2023-11-28T17:11:24.249714+0000","last_peered":"2023-11-28T17:11:24.249714+0000","last_clean":"2023-11-28T17:11:24.249714+0000","last_became_active":"2023-11-28T17:11:23.215112+0000","last_became_peered":"2023-11-28T17:11:23.215112+0000","last_unstale":"2023-11-28T17:11:24.249714+0000","last_undegraded":"2023-11-28T17:11:24.249714+0000","last_fullsized":"2023-11-28T17:11:24.249714+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:09:25.806613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248847+0000","last_change":"2023-11-28T17:11:24.235933+0000","last_active":"2023-11-28T17:11:24.248847+0000","last_peered":"2023-11-28T17:11:24.248847+0000","last_clean":"2023-11-28T17:11:24.248847+0000","last_became_active":"2023-11-28T17:11:24.235622+0000","last_became_peered":"2023-11-28T17:11:24.235622+0000","last_unstale":"2023-11-28T17:11:24.248847+0000","last_undegraded":"2023-11-28T17:11:24.248847+0000","last_fullsized":"2023-11-28T17:11:24.248847+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:06:03.975512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242245+0000","last_change":"2023-11-28T17:11:23.222079+0000","last_active":"2023-11-28T17:11:24.242245+0000","last_peered":"2023-11-28T17:11:24.242245+0000","last_clean":"2023-11-28T17:11:24.242245+0000","last_became_active":"2023-11-28T17:11:23.220772+0000","last_became_peered":"2023-11-28T17:11:23.220772+0000","last_unstale":"2023-11-28T17:11:24.242245+0000","last_undegraded":"2023-11-28T17:11:24.242245+0000","last_fullsized":"2023-11-28T17:11:24.242245+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:30:27.588631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.217037+0000","last_change":"2023-11-28T17:11:24.213431+0000","last_active":"2023-11-28T17:11:24.217037+0000","last_peered":"2023-11-28T17:11:24.217037+0000","last_clean":"2023-11-28T17:11:24.217037+0000","last_became_active":"2023-11-28T17:11:24.212942+0000","last_became_peered":"2023-11-28T17:11:24.212942+0000","last_unstale":"2023-11-28T17:11:24.217037+0000","last_undegraded":"2023-11-28T17:11:24.217037+0000","last_fullsized":"2023-11-28T17:11:24.217037+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:27:40.286894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.250282+0000","last_change":"2023-11-28T17:11:23.213829+0000","last_active":"2023-11-28T17:11:24.250282+0000","last_peered":"2023-11-28T17:11:24.250282+0000","last_clean":"2023-11-28T17:11:24.250282+0000","last_became_active":"2023-11-28T17:11:23.213358+0000","last_became_peered":"2023-11-28T17:11:23.213358+0000","last_unstale":"2023-11-28T17:11:24.250282+0000","last_undegraded":"2023-11-28T17:11:24.250282+0000","last_fullsized":"2023-11-28T17:11:24.250282+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:21.343141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242383+0000","last_change":"2023-11-28T17:11:24.239227+0000","last_active":"2023-11-28T17:11:24.242383+0000","last_peered":"2023-11-28T17:11:24.242383+0000","last_clean":"2023-11-28T17:11:24.242383+0000","last_became_active":"2023-11-28T17:11:24.238796+0000","last_became_peered":"2023-11-28T17:11:24.238796+0000","last_unstale":"2023-11-28T17:11:24.242383+0000","last_undegraded":"2023-11-28T17:11:24.242383+0000","last_fullsized":"2023-11-28T17:11:24.242383+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:06:58.499355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.233066+0000","last_change":"2023-11-28T17:11:23.222991+0000","last_active":"2023-11-28T17:11:24.233066+0000","last_peered":"2023-11-28T17:11:24.233066+0000","last_clean":"2023-11-28T17:11:24.233066+0000","last_became_active":"2023-11-28T17:11:23.222681+0000","last_became_peered":"2023-11-28T17:11:23.222681+0000","last_unstale":"2023-11-28T17:11:24.233066+0000","last_undegraded":"2023-11-28T17:11:24.233066+0000","last_fullsized":"2023-11-28T17:11:24.233066+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:15:23.640505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.239081+0000","last_change":"2023-11-28T17:11:24.227524+0000","last_active":"2023-11-28T17:11:24.239081+0000","last_peered":"2023-11-28T17:11:24.239081+0000","last_clean":"2023-11-28T17:11:24.239081+0000","last_became_active":"2023-11-28T17:11:24.227032+0000","last_became_peered":"2023-11-28T17:11:24.227032+0000","last_unstale":"2023-11-28T17:11:24.239081+0000","last_undegraded":"2023-11-28T17:11:24.239081+0000","last_fullsized":"2023-11-28T17:11:24.239081+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:30:34.918467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.206944+0000","last_change":"2023-11-28T17:11:23.221821+0000","last_active":"2023-11-28T17:11:24.206944+0000","last_peered":"2023-11-28T17:11:24.206944+0000","last_clean":"2023-11-28T17:11:24.206944+0000","last_became_active":"2023-11-28T17:11:23.221506+0000","last_became_peered":"2023-11-28T17:11:23.221506+0000","last_unstale":"2023-11-28T17:11:24.206944+0000","last_undegraded":"2023-11-28T17:11:24.206944+0000","last_fullsized":"2023-11-28T17:11:24.206944+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:21.343141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.218073+0000","last_change":"2023-11-28T17:11:24.216144+0000","last_active":"2023-11-28T17:11:24.218073+0000","last_peered":"2023-11-28T17:11:24.218073+0000","last_clean":"2023-11-28T17:11:24.218073+0000","last_became_active":"2023-11-28T17:11:24.215875+0000","last_became_peered":"2023-11-28T17:11:24.215875+0000","last_unstale":"2023-11-28T17:11:24.218073+0000","last_undegraded":"2023-11-28T17:11:24.218073+0000","last_fullsized":"2023-11-28T17:11:24.218073+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:37:46.364504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.212985+0000","last_change":"2023-11-28T17:11:23.220657+0000","last_active":"2023-11-28T17:11:24.212985+0000","last_peered":"2023-11-28T17:11:24.212985+0000","last_clean":"2023-11-28T17:11:24.212985+0000","last_became_active":"2023-11-28T17:11:23.219057+0000","last_became_peered":"2023-11-28T17:11:23.219057+0000","last_unstale":"2023-11-28T17:11:24.212985+0000","last_undegraded":"2023-11-28T17:11:24.212985+0000","last_fullsized":"2023-11-28T17:11:24.212985+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:29.976839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.221840+0000","last_change":"2023-11-28T17:11:24.219719+0000","last_active":"2023-11-28T17:11:24.221840+0000","last_peered":"2023-11-28T17:11:24.221840+0000","last_clean":"2023-11-28T17:11:24.221840+0000","last_became_active":"2023-11-28T17:11:24.219430+0000","last_became_peered":"2023-11-28T17:11:24.219430+0000","last_unstale":"2023-11-28T17:11:24.221840+0000","last_undegraded":"2023-11-28T17:11:24.221840+0000","last_fullsized":"2023-11-28T17:11:24.221840+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:41:38.593938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.211151+0000","last_change":"2023-11-28T17:11:23.225546+0000","last_active":"2023-11-28T17:11:24.211151+0000","last_peered":"2023-11-28T17:11:24.211151+0000","last_clean":"2023-11-28T17:11:24.211151+0000","last_became_active":"2023-11-28T17:11:23.225200+0000","last_became_peered":"2023-11-28T17:11:23.225200+0000","last_unstale":"2023-11-28T17:11:24.211151+0000","last_undegraded":"2023-11-28T17:11:24.211151+0000","last_fullsized":"2023-11-28T17:11:24.211151+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:12:03.116329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.219487+0000","last_change":"2023-11-28T17:11:24.215538+0000","last_active":"2023-11-28T17:11:24.219487+0000","last_peered":"2023-11-28T17:11:24.219487+0000","last_clean":"2023-11-28T17:11:24.219487+0000","last_became_active":"2023-11-28T17:11:24.215242+0000","last_became_peered":"2023-11-28T17:11:24.215242+0000","last_unstale":"2023-11-28T17:11:24.219487+0000","last_undegraded":"2023-11-28T17:11:24.219487+0000","last_fullsized":"2023-11-28T17:11:24.219487+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:37.638014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.217310+0000","last_change":"2023-11-28T17:11:23.218173+0000","last_active":"2023-11-28T17:11:24.217310+0000","last_peered":"2023-11-28T17:11:24.217310+0000","last_clean":"2023-11-28T17:11:24.217310+0000","last_became_active":"2023-11-28T17:11:23.217882+0000","last_became_peered":"2023-11-28T17:11:23.217882+0000","last_unstale":"2023-11-28T17:11:24.217310+0000","last_undegraded":"2023-11-28T17:11:24.217310+0000","last_fullsized":"2023-11-28T17:11:24.217310+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:05.792484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.252174+0000","last_change":"2023-11-28T17:11:24.250103+0000","last_active":"2023-11-28T17:11:24.252174+0000","last_peered":"2023-11-28T17:11:24.252174+0000","last_clean":"2023-11-28T17:11:24.252174+0000","last_became_active":"2023-11-28T17:11:24.246150+0000","last_became_peered":"2023-11-28T17:11:24.246150+0000","last_unstale":"2023-11-28T17:11:24.252174+0000","last_undegraded":"2023-11-28T17:11:24.252174+0000","last_fullsized":"2023-11-28T17:11:24.252174+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:30:34.918467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"47'3","reported_seq":14,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.232609+0000","last_change":"2023-11-28T17:11:23.214497+0000","last_active":"2023-11-28T17:11:24.232609+0000","last_peered":"2023-11-28T17:11:24.232609+0000","last_clean":"2023-11-28T17:11:24.232609+0000","last_became_active":"2023-11-28T17:11:23.214213+0000","last_became_peered":"2023-11-28T17:11:23.214213+0000","last_unstale":"2023-11-28T17:11:24.232609+0000","last_undegraded":"2023-11-28T17:11:24.232609+0000","last_fullsized":"2023-11-28T17:11:24.232609+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:18:07.119444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.254787+0000","last_change":"2023-11-28T17:11:24.249758+0000","last_active":"2023-11-28T17:11:24.254787+0000","last_peered":"2023-11-28T17:11:24.254787+0000","last_clean":"2023-11-28T17:11:24.254787+0000","last_became_active":"2023-11-28T17:11:24.249467+0000","last_became_peered":"2023-11-28T17:11:24.249467+0000","last_unstale":"2023-11-28T17:11:24.254787+0000","last_undegraded":"2023-11-28T17:11:24.254787+0000","last_fullsized":"2023-11-28T17:11:24.254787+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:51:07.135164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"47'6","reported_seq":17,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.255532+0000","last_change":"2023-11-28T17:11:23.223026+0000","last_active":"2023-11-28T17:11:24.255532+0000","last_peered":"2023-11-28T17:11:24.255532+0000","last_clean":"2023-11-28T17:11:24.255532+0000","last_became_active":"2023-11-28T17:11:23.222728+0000","last_became_peered":"2023-11-28T17:11:23.222728+0000","last_unstale":"2023-11-28T17:11:24.255532+0000","last_undegraded":"2023-11-28T17:11:24.255532+0000","last_fullsized":"2023-11-28T17:11:24.255532+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:05:48.739125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.249688+0000","last_change":"2023-11-28T17:11:24.245271+0000","last_active":"2023-11-28T17:11:24.249688+0000","last_peered":"2023-11-28T17:11:24.249688+0000","last_clean":"2023-11-28T17:11:24.249688+0000","last_became_active":"2023-11-28T17:11:24.243901+0000","last_became_peered":"2023-11-28T17:11:24.243901+0000","last_unstale":"2023-11-28T17:11:24.249688+0000","last_undegraded":"2023-11-28T17:11:24.249688+0000","last_fullsized":"2023-11-28T17:11:24.249688+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:23:32.010363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.249177+0000","last_change":"2023-11-28T17:11:23.224266+0000","last_active":"2023-11-28T17:11:24.249177+0000","last_peered":"2023-11-28T17:11:24.249177+0000","last_clean":"2023-11-28T17:11:24.249177+0000","last_became_active":"2023-11-28T17:11:23.223155+0000","last_became_peered":"2023-11-28T17:11:23.223155+0000","last_unstale":"2023-11-28T17:11:24.249177+0000","last_undegraded":"2023-11-28T17:11:24.249177+0000","last_fullsized":"2023-11-28T17:11:24.249177+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:04.988639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.226916+0000","last_change":"2023-11-28T17:11:24.219896+0000","last_active":"2023-11-28T17:11:24.226916+0000","last_peered":"2023-11-28T17:11:24.226916+0000","last_clean":"2023-11-28T17:11:24.226916+0000","last_became_active":"2023-11-28T17:11:24.218800+0000","last_became_peered":"2023-11-28T17:11:24.218800+0000","last_unstale":"2023-11-28T17:11:24.226916+0000","last_undegraded":"2023-11-28T17:11:24.226916+0000","last_fullsized":"2023-11-28T17:11:24.226916+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:56.943404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.208333+0000","last_change":"2023-11-28T17:11:23.222532+0000","last_active":"2023-11-28T17:11:24.208333+0000","last_peered":"2023-11-28T17:11:24.208333+0000","last_clean":"2023-11-28T17:11:24.208333+0000","last_became_active":"2023-11-28T17:11:23.222273+0000","last_became_peered":"2023-11-28T17:11:23.222273+0000","last_unstale":"2023-11-28T17:11:24.208333+0000","last_undegraded":"2023-11-28T17:11:24.208333+0000","last_fullsized":"2023-11-28T17:11:24.208333+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:31.352773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.233896+0000","last_change":"2023-11-28T17:11:24.231913+0000","last_active":"2023-11-28T17:11:24.233896+0000","last_peered":"2023-11-28T17:11:24.233896+0000","last_clean":"2023-11-28T17:11:24.233896+0000","last_became_active":"2023-11-28T17:11:24.231595+0000","last_became_peered":"2023-11-28T17:11:24.231595+0000","last_unstale":"2023-11-28T17:11:24.233896+0000","last_undegraded":"2023-11-28T17:11:24.233896+0000","last_fullsized":"2023-11-28T17:11:24.233896+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:27:31.482201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.211250+0000","last_change":"2023-11-28T17:11:23.220655+0000","last_active":"2023-11-28T17:11:24.211250+0000","last_peered":"2023-11-28T17:11:24.211250+0000","last_clean":"2023-11-28T17:11:24.211250+0000","last_became_active":"2023-11-28T17:11:23.219314+0000","last_became_peered":"2023-11-28T17:11:23.219314+0000","last_unstale":"2023-11-28T17:11:24.211250+0000","last_undegraded":"2023-11-28T17:11:24.211250+0000","last_fullsized":"2023-11-28T17:11:24.211250+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:13:36.324975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223073+0000","last_change":"2023-11-28T17:11:24.214712+0000","last_active":"2023-11-28T17:11:24.223073+0000","last_peered":"2023-11-28T17:11:24.223073+0000","last_clean":"2023-11-28T17:11:24.223073+0000","last_became_active":"2023-11-28T17:11:24.214461+0000","last_became_peered":"2023-11-28T17:11:24.214461+0000","last_unstale":"2023-11-28T17:11:24.223073+0000","last_undegraded":"2023-11-28T17:11:24.223073+0000","last_fullsized":"2023-11-28T17:11:24.223073+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:13:37.338728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"47'8","reported_seq":19,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248490+0000","last_change":"2023-11-28T17:11:23.220889+0000","last_active":"2023-11-28T17:11:24.248490+0000","last_peered":"2023-11-28T17:11:24.248490+0000","last_clean":"2023-11-28T17:11:24.248490+0000","last_became_active":"2023-11-28T17:11:23.219444+0000","last_became_peered":"2023-11-28T17:11:23.219444+0000","last_unstale":"2023-11-28T17:11:24.248490+0000","last_undegraded":"2023-11-28T17:11:24.248490+0000","last_fullsized":"2023-11-28T17:11:24.248490+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:53:16.122203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.219766+0000","last_change":"2023-11-28T17:11:24.218149+0000","last_active":"2023-11-28T17:11:24.219766+0000","last_peered":"2023-11-28T17:11:24.219766+0000","last_clean":"2023-11-28T17:11:24.219766+0000","last_became_active":"2023-11-28T17:11:24.217819+0000","last_became_peered":"2023-11-28T17:11:24.217819+0000","last_unstale":"2023-11-28T17:11:24.219766+0000","last_undegraded":"2023-11-28T17:11:24.219766+0000","last_fullsized":"2023-11-28T17:11:24.219766+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:07.298149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248729+0000","last_change":"2023-11-28T17:11:23.223512+0000","last_active":"2023-11-28T17:11:24.248729+0000","last_peered":"2023-11-28T17:11:24.248729+0000","last_clean":"2023-11-28T17:11:24.248729+0000","last_became_active":"2023-11-28T17:11:23.222568+0000","last_became_peered":"2023-11-28T17:11:23.222568+0000","last_unstale":"2023-11-28T17:11:24.248729+0000","last_undegraded":"2023-11-28T17:11:24.248729+0000","last_fullsized":"2023-11-28T17:11:24.248729+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:54.065477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.239889+0000","last_change":"2023-11-28T17:11:24.234785+0000","last_active":"2023-11-28T17:11:24.239889+0000","last_peered":"2023-11-28T17:11:24.239889+0000","last_clean":"2023-11-28T17:11:24.239889+0000","last_became_active":"2023-11-28T17:11:24.234447+0000","last_became_peered":"2023-11-28T17:11:24.234447+0000","last_unstale":"2023-11-28T17:11:24.239889+0000","last_undegraded":"2023-11-28T17:11:24.239889+0000","last_fullsized":"2023-11-28T17:11:24.239889+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:12:04.130082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.245405+0000","last_change":"2023-11-28T17:11:23.219015+0000","last_active":"2023-11-28T17:11:24.245405+0000","last_peered":"2023-11-28T17:11:24.245405+0000","last_clean":"2023-11-28T17:11:24.245405+0000","last_became_active":"2023-11-28T17:11:23.218733+0000","last_became_peered":"2023-11-28T17:11:23.218733+0000","last_unstale":"2023-11-28T17:11:24.245405+0000","last_undegraded":"2023-11-28T17:11:24.245405+0000","last_fullsized":"2023-11-28T17:11:24.245405+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:55.929651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248232+0000","last_change":"2023-11-28T17:11:24.246437+0000","last_active":"2023-11-28T17:11:24.248232+0000","last_peered":"2023-11-28T17:11:24.248232+0000","last_clean":"2023-11-28T17:11:24.248232+0000","last_became_active":"2023-11-28T17:11:24.246167+0000","last_became_peered":"2023-11-28T17:11:24.246167+0000","last_unstale":"2023-11-28T17:11:24.248232+0000","last_undegraded":"2023-11-28T17:11:24.248232+0000","last_fullsized":"2023-11-28T17:11:24.248232+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:30:34.918467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242551+0000","last_change":"2023-11-28T17:11:23.214992+0000","last_active":"2023-11-28T17:11:24.242551+0000","last_peered":"2023-11-28T17:11:24.242551+0000","last_clean":"2023-11-28T17:11:24.242551+0000","last_became_active":"2023-11-28T17:11:23.214694+0000","last_became_peered":"2023-11-28T17:11:23.214694+0000","last_unstale":"2023-11-28T17:11:24.242551+0000","last_undegraded":"2023-11-28T17:11:24.242551+0000","last_fullsized":"2023-11-28T17:11:24.242551+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:06.284396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242213+0000","last_change":"2023-11-28T17:11:24.238610+0000","last_active":"2023-11-28T17:11:24.242213+0000","last_peered":"2023-11-28T17:11:24.242213+0000","last_clean":"2023-11-28T17:11:24.242213+0000","last_became_active":"2023-11-28T17:11:24.238168+0000","last_became_peered":"2023-11-28T17:11:24.238168+0000","last_unstale":"2023-11-28T17:11:24.242213+0000","last_undegraded":"2023-11-28T17:11:24.242213+0000","last_fullsized":"2023-11-28T17:11:24.242213+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:27:31.482201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.209087+0000","last_change":"2023-11-28T17:11:23.218564+0000","last_active":"2023-11-28T17:11:24.209087+0000","last_peered":"2023-11-28T17:11:24.209087+0000","last_clean":"2023-11-28T17:11:24.209087+0000","last_became_active":"2023-11-28T17:11:23.218279+0000","last_became_peered":"2023-11-28T17:11:23.218279+0000","last_unstale":"2023-11-28T17:11:24.209087+0000","last_undegraded":"2023-11-28T17:11:24.209087+0000","last_fullsized":"2023-11-28T17:11:24.209087+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:04.436680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242213+0000","last_change":"2023-11-28T17:11:24.238576+0000","last_active":"2023-11-28T17:11:24.242213+0000","last_peered":"2023-11-28T17:11:24.242213+0000","last_clean":"2023-11-28T17:11:24.242213+0000","last_became_active":"2023-11-28T17:11:24.237274+0000","last_became_peered":"2023-11-28T17:11:24.237274+0000","last_unstale":"2023-11-28T17:11:24.242213+0000","last_undegraded":"2023-11-28T17:11:24.242213+0000","last_fullsized":"2023-11-28T17:11:24.242213+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:05:25.971021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.238520+0000","last_change":"2023-11-28T17:11:23.214890+0000","last_active":"2023-11-28T17:11:24.238520+0000","last_peered":"2023-11-28T17:11:24.238520+0000","last_clean":"2023-11-28T17:11:24.238520+0000","last_became_active":"2023-11-28T17:11:23.213727+0000","last_became_peered":"2023-11-28T17:11:23.213727+0000","last_unstale":"2023-11-28T17:11:24.238520+0000","last_undegraded":"2023-11-28T17:11:24.238520+0000","last_fullsized":"2023-11-28T17:11:24.238520+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:07.784341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.231630+0000","last_change":"2023-11-28T17:11:24.226847+0000","last_active":"2023-11-28T17:11:24.231630+0000","last_peered":"2023-11-28T17:11:24.231630+0000","last_clean":"2023-11-28T17:11:24.231630+0000","last_became_active":"2023-11-28T17:11:24.226493+0000","last_became_peered":"2023-11-28T17:11:24.226493+0000","last_unstale":"2023-11-28T17:11:24.231630+0000","last_undegraded":"2023-11-28T17:11:24.231630+0000","last_fullsized":"2023-11-28T17:11:24.231630+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:06:32.767821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.208445+0000","last_change":"2023-11-28T17:11:23.218827+0000","last_active":"2023-11-28T17:11:24.208445+0000","last_peered":"2023-11-28T17:11:24.208445+0000","last_clean":"2023-11-28T17:11:24.208445+0000","last_became_active":"2023-11-28T17:11:23.218545+0000","last_became_peered":"2023-11-28T17:11:23.218545+0000","last_unstale":"2023-11-28T17:11:24.208445+0000","last_undegraded":"2023-11-28T17:11:24.208445+0000","last_fullsized":"2023-11-28T17:11:24.208445+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:29.976839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248757+0000","last_change":"2023-11-28T17:11:24.243810+0000","last_active":"2023-11-28T17:11:24.248757+0000","last_peered":"2023-11-28T17:11:24.248757+0000","last_clean":"2023-11-28T17:11:24.248757+0000","last_became_active":"2023-11-28T17:11:24.243045+0000","last_became_peered":"2023-11-28T17:11:24.243045+0000","last_unstale":"2023-11-28T17:11:24.248757+0000","last_undegraded":"2023-11-28T17:11:24.248757+0000","last_fullsized":"2023-11-28T17:11:24.248757+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:58:02.404658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.212978+0000","last_change":"2023-11-28T17:11:23.212259+0000","last_active":"2023-11-28T17:11:24.212978+0000","last_peered":"2023-11-28T17:11:24.212978+0000","last_clean":"2023-11-28T17:11:24.212978+0000","last_became_active":"2023-11-28T17:11:23.211898+0000","last_became_peered":"2023-11-28T17:11:23.211898+0000","last_unstale":"2023-11-28T17:11:24.212978+0000","last_undegraded":"2023-11-28T17:11:24.212978+0000","last_fullsized":"2023-11-28T17:11:24.212978+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:31.352773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.251318+0000","last_change":"2023-11-28T17:11:24.248225+0000","last_active":"2023-11-28T17:11:24.251318+0000","last_peered":"2023-11-28T17:11:24.251318+0000","last_clean":"2023-11-28T17:11:24.251318+0000","last_became_active":"2023-11-28T17:11:24.247968+0000","last_became_peered":"2023-11-28T17:11:24.247968+0000","last_unstale":"2023-11-28T17:11:24.251318+0000","last_undegraded":"2023-11-28T17:11:24.251318+0000","last_fullsized":"2023-11-28T17:11:24.251318+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:00.298985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.206477+0000","last_change":"2023-11-28T17:11:23.221517+0000","last_active":"2023-11-28T17:11:24.206477+0000","last_peered":"2023-11-28T17:11:24.206477+0000","last_clean":"2023-11-28T17:11:24.206477+0000","last_became_active":"2023-11-28T17:11:23.221239+0000","last_became_peered":"2023-11-28T17:11:23.221239+0000","last_unstale":"2023-11-28T17:11:24.206477+0000","last_undegraded":"2023-11-28T17:11:24.206477+0000","last_fullsized":"2023-11-28T17:11:24.206477+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:44.532346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.233453+0000","last_change":"2023-11-28T17:11:24.228992+0000","last_active":"2023-11-28T17:11:24.233453+0000","last_peered":"2023-11-28T17:11:24.233453+0000","last_clean":"2023-11-28T17:11:24.233453+0000","last_became_active":"2023-11-28T17:11:24.228258+0000","last_became_peered":"2023-11-28T17:11:24.228258+0000","last_unstale":"2023-11-28T17:11:24.233453+0000","last_undegraded":"2023-11-28T17:11:24.233453+0000","last_fullsized":"2023-11-28T17:11:24.233453+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:55:57.542662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.228420+0000","last_change":"2023-11-28T17:11:23.225083+0000","last_active":"2023-11-28T17:11:24.228420+0000","last_peered":"2023-11-28T17:11:24.228420+0000","last_clean":"2023-11-28T17:11:24.228420+0000","last_became_active":"2023-11-28T17:11:23.224633+0000","last_became_peered":"2023-11-28T17:11:23.224633+0000","last_unstale":"2023-11-28T17:11:24.228420+0000","last_undegraded":"2023-11-28T17:11:24.228420+0000","last_fullsized":"2023-11-28T17:11:24.228420+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:06:57.485602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242045+0000","last_change":"2023-11-28T17:11:24.235923+0000","last_active":"2023-11-28T17:11:24.242045+0000","last_peered":"2023-11-28T17:11:24.242045+0000","last_clean":"2023-11-28T17:11:24.242045+0000","last_became_active":"2023-11-28T17:11:24.235637+0000","last_became_peered":"2023-11-28T17:11:24.235637+0000","last_unstale":"2023-11-28T17:11:24.242045+0000","last_undegraded":"2023-11-28T17:11:24.242045+0000","last_fullsized":"2023-11-28T17:11:24.242045+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:39:21.268933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.207641+0000","last_change":"2023-11-28T17:11:23.223135+0000","last_active":"2023-11-28T17:11:24.207641+0000","last_peered":"2023-11-28T17:11:24.207641+0000","last_clean":"2023-11-28T17:11:24.207641+0000","last_became_active":"2023-11-28T17:11:23.222821+0000","last_became_peered":"2023-11-28T17:11:23.222821+0000","last_unstale":"2023-11-28T17:11:24.207641+0000","last_undegraded":"2023-11-28T17:11:24.207641+0000","last_fullsized":"2023-11-28T17:11:24.207641+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:06:02.961760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.254768+0000","last_change":"2023-11-28T17:11:24.249637+0000","last_active":"2023-11-28T17:11:24.254768+0000","last_peered":"2023-11-28T17:11:24.254768+0000","last_clean":"2023-11-28T17:11:24.254768+0000","last_became_active":"2023-11-28T17:11:24.249321+0000","last_became_peered":"2023-11-28T17:11:24.249321+0000","last_unstale":"2023-11-28T17:11:24.254768+0000","last_undegraded":"2023-11-28T17:11:24.254768+0000","last_fullsized":"2023-11-28T17:11:24.254768+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:56:28.171534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.204003+0000","last_change":"2023-11-28T17:11:23.226421+0000","last_active":"2023-11-28T17:11:24.204003+0000","last_peered":"2023-11-28T17:11:24.204003+0000","last_clean":"2023-11-28T17:11:24.204003+0000","last_became_active":"2023-11-28T17:11:23.225639+0000","last_became_peered":"2023-11-28T17:11:23.225639+0000","last_unstale":"2023-11-28T17:11:24.204003+0000","last_undegraded":"2023-11-28T17:11:24.204003+0000","last_fullsized":"2023-11-28T17:11:24.204003+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:36.624262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242268+0000","last_change":"2023-11-28T17:11:24.238875+0000","last_active":"2023-11-28T17:11:24.242268+0000","last_peered":"2023-11-28T17:11:24.242268+0000","last_clean":"2023-11-28T17:11:24.242268+0000","last_became_active":"2023-11-28T17:11:24.237847+0000","last_became_peered":"2023-11-28T17:11:24.237847+0000","last_unstale":"2023-11-28T17:11:24.242268+0000","last_undegraded":"2023-11-28T17:11:24.242268+0000","last_fullsized":"2023-11-28T17:11:24.242268+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:47:10.497869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.207376+0000","last_change":"2023-11-28T17:11:23.223183+0000","last_active":"2023-11-28T17:11:24.207376+0000","last_peered":"2023-11-28T17:11:24.207376+0000","last_clean":"2023-11-28T17:11:24.207376+0000","last_became_active":"2023-11-28T17:11:23.222858+0000","last_became_peered":"2023-11-28T17:11:23.222858+0000","last_unstale":"2023-11-28T17:11:24.207376+0000","last_undegraded":"2023-11-28T17:11:24.207376+0000","last_fullsized":"2023-11-28T17:11:24.207376+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:13:36.324975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.249828+0000","last_change":"2023-11-28T17:11:24.236150+0000","last_active":"2023-11-28T17:11:24.249828+0000","last_peered":"2023-11-28T17:11:24.249828+0000","last_clean":"2023-11-28T17:11:24.249828+0000","last_became_active":"2023-11-28T17:11:24.235383+0000","last_became_peered":"2023-11-28T17:11:24.235383+0000","last_unstale":"2023-11-28T17:11:24.249828+0000","last_undegraded":"2023-11-28T17:11:24.249828+0000","last_fullsized":"2023-11-28T17:11:24.249828+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:09:26.820366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.207063+0000","last_change":"2023-11-28T17:11:23.223551+0000","last_active":"2023-11-28T17:11:24.207063+0000","last_peered":"2023-11-28T17:11:24.207063+0000","last_clean":"2023-11-28T17:11:24.207063+0000","last_became_active":"2023-11-28T17:11:23.223272+0000","last_became_peered":"2023-11-28T17:11:23.223272+0000","last_unstale":"2023-11-28T17:11:24.207063+0000","last_undegraded":"2023-11-28T17:11:24.207063+0000","last_fullsized":"2023-11-28T17:11:24.207063+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.781121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.245466+0000","last_change":"2023-11-28T17:11:24.240208+0000","last_active":"2023-11-28T17:11:24.245466+0000","last_peered":"2023-11-28T17:11:24.245466+0000","last_clean":"2023-11-28T17:11:24.245466+0000","last_became_active":"2023-11-28T17:11:24.239742+0000","last_became_peered":"2023-11-28T17:11:24.239742+0000","last_unstale":"2023-11-28T17:11:24.245466+0000","last_undegraded":"2023-11-28T17:11:24.245466+0000","last_fullsized":"2023-11-28T17:11:24.245466+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:47.263936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.211573+0000","last_change":"2023-11-28T17:11:23.219595+0000","last_active":"2023-11-28T17:11:24.211573+0000","last_peered":"2023-11-28T17:11:24.211573+0000","last_clean":"2023-11-28T17:11:24.211573+0000","last_became_active":"2023-11-28T17:11:23.218954+0000","last_became_peered":"2023-11-28T17:11:23.218954+0000","last_unstale":"2023-11-28T17:11:24.211573+0000","last_undegraded":"2023-11-28T17:11:24.211573+0000","last_fullsized":"2023-11-28T17:11:24.211573+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:51:24.669526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223803+0000","last_change":"2023-11-28T17:11:24.216105+0000","last_active":"2023-11-28T17:11:24.223803+0000","last_peered":"2023-11-28T17:11:24.223803+0000","last_clean":"2023-11-28T17:11:24.223803+0000","last_became_active":"2023-11-28T17:11:24.215824+0000","last_became_peered":"2023-11-28T17:11:24.215824+0000","last_unstale":"2023-11-28T17:11:24.223803+0000","last_undegraded":"2023-11-28T17:11:24.223803+0000","last_fullsized":"2023-11-28T17:11:24.223803+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:12:49.613982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.202251+0000","last_change":"2023-11-28T17:11:23.225850+0000","last_active":"2023-11-28T17:11:24.202251+0000","last_peered":"2023-11-28T17:11:24.202251+0000","last_clean":"2023-11-28T17:11:24.202251+0000","last_became_active":"2023-11-28T17:11:23.225316+0000","last_became_peered":"2023-11-28T17:11:23.225316+0000","last_unstale":"2023-11-28T17:11:24.202251+0000","last_undegraded":"2023-11-28T17:11:24.202251+0000","last_fullsized":"2023-11-28T17:11:24.202251+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:23:30.996611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.252270+0000","last_change":"2023-11-28T17:11:24.250815+0000","last_active":"2023-11-28T17:11:24.252270+0000","last_peered":"2023-11-28T17:11:24.252270+0000","last_clean":"2023-11-28T17:11:24.252270+0000","last_became_active":"2023-11-28T17:11:24.250568+0000","last_became_peered":"2023-11-28T17:11:24.250568+0000","last_unstale":"2023-11-28T17:11:24.252270+0000","last_undegraded":"2023-11-28T17:11:24.252270+0000","last_fullsized":"2023-11-28T17:11:24.252270+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:53:17.135956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.249045+0000","last_change":"2023-11-28T17:11:23.226792+0000","last_active":"2023-11-28T17:11:24.249045+0000","last_peered":"2023-11-28T17:11:24.249045+0000","last_clean":"2023-11-28T17:11:24.249045+0000","last_became_active":"2023-11-28T17:11:23.226442+0000","last_became_peered":"2023-11-28T17:11:23.226442+0000","last_unstale":"2023-11-28T17:11:24.249045+0000","last_undegraded":"2023-11-28T17:11:24.249045+0000","last_fullsized":"2023-11-28T17:11:24.249045+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:50.078274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":191,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-11-28T17:11:23.207303+0000","last_change":"2023-11-28T17:08:51.392782+0000","last_active":"2023-11-28T17:11:23.207303+0000","last_peered":"2023-11-28T17:11:23.207303+0000","last_clean":"2023-11-28T17:11:23.207303+0000","last_became_active":"2023-11-28T17:08:51.390415+0000","last_became_peered":"2023-11-28T17:08:51.390415+0000","last_unstale":"2023-11-28T17:11:23.207303+0000","last_undegraded":"2023-11-28T17:11:23.207303+0000","last_fullsized":"2023-11-28T17:11:23.207303+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_clean_scrub_stamp":"2023-11-28T17:08:49.967682+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:03:32.638685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":540672,"data_stored":219335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.1100000000000001,"15min":1.087},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.337,"5min":1.151,"15min":1.1200000000000001},"min":{"1min":0.78600000000000003,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.294}]},{"osd":1,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.1279999999999999,"15min":1.1339999999999999},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.2969999999999999,"15min":1.296},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.77500000000000002}]},{"osd":2,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.228,"15min":1.2170000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.841,"5min":1.841,"15min":1.841},"last":0.995},{"interface":"front","average":{"1min":1.226,"5min":1.3260000000000001,"15min":1.3420000000000001},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.889,"5min":2.7029999999999998,"15min":2.7029999999999998},"last":1.5309999999999999}]},{"osd":3,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.179,"15min":1.181},"min":{"1min":0.74299999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":1.073},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.423,"15min":1.421},"min":{"1min":0.83299999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.9510000000000001,"5min":2.161,"15min":2.161},"last":1.179}]},{"osd":4,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.335,"15min":1.329},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.821,"5min":2.202,"15min":2.202},"last":1.427},{"interface":"front","average":{"1min":1.327,"5min":1.367,"15min":1.373},"min":{"1min":0.67500000000000004,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.8939999999999999,"5min":2.069,"15min":2.069},"last":1.246}]},{"osd":5,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.4830000000000001,"15min":1.478},"min":{"1min":0.84299999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.161,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.552,"5min":1.5680000000000001,"15min":1.571},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.1720000000000002,"5min":2.7850000000000001,"15min":2.7850000000000001},"last":1.6599999999999999}]},{"osd":6,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.6970000000000001,"5min":1.7330000000000001,"15min":1.7390000000000001},"min":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"max":{"1min":2.3559999999999999,"5min":3.8010000000000002,"15min":3.8010000000000002},"last":1.7370000000000001},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.605,"15min":1.6200000000000001},"min":{"1min":1.1180000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.0419999999999998,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.6220000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":540672,"data_stored":219335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.1779999999999999,"15min":1.2110000000000001},"min":{"1min":0.64600000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6839999999999999,"5min":4.3579999999999997,"15min":4.3579999999999997},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0780000000000001,"15min":1.0880000000000001},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.48,"5min":1.841,"15min":1.841},"last":1.3600000000000001}]},{"osd":1,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.3759999999999999,"15min":1.411},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.1190000000000002,"5min":4.1580000000000004,"15min":4.1580000000000004},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.238,"5min":1.3779999999999999,"15min":1.401},"min":{"1min":0.872,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.6259999999999999,"5min":4.202,"15min":4.202},"last":1.4890000000000001}]},{"osd":2,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1319999999999999,"15min":1.137},"min":{"1min":0.77800000000000002,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.425,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":1.208},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.4830000000000001,"15min":1.5169999999999999},"min":{"1min":0.872,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.2250000000000001,"5min":4.2750000000000004,"15min":4.2750000000000004},"last":1.254}]},{"osd":3,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2609999999999999,"15min":1.262},"min":{"1min":0.81399999999999995,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.74,"5min":1.74,"15min":1.74},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.395,"15min":1.399},"min":{"1min":0.86699999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.1789999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":1.4059999999999999}]},{"osd":4,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.3600000000000001,"15min":1.3620000000000001},"min":{"1min":0.82999999999999996,"5min":0.624,"15min":0.624},"max":{"1min":2.0430000000000001,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.415,"5min":1.4259999999999999,"15min":1.4279999999999999},"min":{"1min":0.97499999999999998,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.0779999999999998,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.3120000000000001}]},{"osd":5,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.528,"15min":1.524},"min":{"1min":1.1759999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":2.0289999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":1.46},{"interface":"front","average":{"1min":1.548,"5min":1.5940000000000001,"15min":1.6020000000000001},"min":{"1min":1.21,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.3919999999999999,"5min":3.0030000000000001,"15min":3.0030000000000001},"last":1.579}]},{"osd":7,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.508,"15min":1.512},"min":{"1min":1.147,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.982,"5min":2.524,"15min":2.524},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.655,"15min":1.675},"min":{"1min":1.2150000000000001,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.97,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.5189999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1134592,"data_stored":809703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":1.0229999999999999,"15min":1.0329999999999999},"min":{"1min":0.67200000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":1.206}]},{"osd":1,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.2030000000000001,"15min":1.2070000000000001},"min":{"1min":0.79700000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.1080000000000001,"15min":1.0800000000000001},"min":{"1min":0.86399999999999999,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.008}]},{"osd":2,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.2370000000000001,"15min":1.232},"min":{"1min":0.80600000000000005,"5min":0.749,"15min":0.749},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.429},{"interface":"front","average":{"1min":1.224,"5min":1.252,"15min":1.256},"min":{"1min":0.78900000000000003,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5429999999999999,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.131}]},{"osd":3,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.2030000000000001,"15min":1.1890000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.7989999999999999,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.365,"5min":1.3859999999999999,"15min":1.3899999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":2.0960000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.266}]},{"osd":4,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.417,"15min":1.421},"min":{"1min":0.82999999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.8440000000000001,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.335},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.3460000000000001,"15min":1.323},"min":{"1min":0.73399999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":1.385}]},{"osd":6,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.4430000000000001,"15min":1.4239999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":2.2839999999999998,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":1.554},{"interface":"front","average":{"1min":1.518,"5min":1.5349999999999999,"15min":1.538},"min":{"1min":0.874,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.964,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.47}]},{"osd":7,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.478,"15min":1.462},"min":{"1min":1.002,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":1.512},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.5580000000000001,"15min":1.546},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.4369999999999998,"5min":2.46,"15min":2.46},"last":1.627}]}]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":540672,"data_stored":219335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":1.02,"15min":1.038},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.3680000000000001,"5min":1.673,"15min":1.673},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.1779999999999999,"15min":1.179},"min":{"1min":0.72799999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.581,"5min":1.837,"15min":1.837},"last":1.397}]},{"osd":1,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.141,"15min":1.1559999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.097,"5min":1.244,"15min":1.268},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.5049999999999999,"5min":1.71,"15min":1.71},"last":0.76000000000000001}]},{"osd":2,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.169,"15min":1.206},"min":{"1min":0.71699999999999997,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.252,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.337,"15min":1.353},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.73,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.5089999999999999}]},{"osd":3,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.3100000000000001,"15min":1.3340000000000001},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.1549999999999998,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.377,"5min":1.405,"15min":1.4099999999999999},"min":{"1min":1.03,"5min":0.87,"15min":0.87},"max":{"1min":1.6439999999999999,"5min":2.048,"15min":2.048},"last":1.615}]},{"osd":5,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.425,"15min":1.4590000000000001},"min":{"1min":0.81499999999999995,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.7410000000000001,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.4490000000000001,"15min":1.4810000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.9870000000000001,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":1.0589999999999999}]},{"osd":6,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.6299999999999999,"15min":1.647},"min":{"1min":1.26,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.8280000000000001,"5min":2.1520000000000001,"15min":2.1520000000000001},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.363,"5min":1.619,"15min":1.6619999999999999},"min":{"1min":1.0269999999999999,"5min":0.749,"15min":0.749},"max":{"1min":1.867,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":1.4490000000000001}]},{"osd":7,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.6339999999999999,"15min":1.673},"min":{"1min":0.80000000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.2879999999999998,"5min":2.669,"15min":2.669},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.6599999999999999,"15min":1.6679999999999999},"min":{"1min":1.248,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.927,"5min":2.387,"15min":2.387},"last":1.806}]}]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27552,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":540672,"data_stored":219335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.95499999999999996,"15min":0.98099999999999998},"min":{"1min":0.34699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.5569999999999999,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.98499999999999999,"15min":0.997},"min":{"1min":0.48399999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.4370000000000001,"5min":2.444,"15min":2.444},"last":0.65700000000000003}]},{"osd":1,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":1.024,"15min":1.0569999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.325,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":0.621},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.1100000000000001,"15min":1.131},"min":{"1min":0.55600000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4810000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.55100000000000005}]},{"osd":2,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.1779999999999999,"15min":1.194},"min":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.587,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.046,"15min":1.0640000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.3540000000000001,"5min":2.319,"15min":2.319},"last":0.72099999999999997}]},{"osd":4,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.268,"15min":1.284},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.744,"5min":2.5699999999999998,"15min":2.5699999999999998},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.355,"15min":1.375},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.7110000000000001,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":1.105}]},{"osd":5,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.4179999999999999,"15min":1.446},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.5580000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.4950000000000001,"15min":1.524},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.706,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.73999999999999999}]},{"osd":6,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.365,"15min":1.3759999999999999},"min":{"1min":0.73199999999999998,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7869999999999999,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.335,"5min":1.5449999999999999,"15min":1.5800000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.825,"5min":2.6000000000000001,"15min":2.6000000000000001},"last":0.82099999999999995}]},{"osd":7,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.5509999999999999,"15min":1.577},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.79,"5min":2.508,"15min":2.508},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.4770000000000001,"15min":1.4890000000000001},"min":{"1min":0.871,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.8580000000000001,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":0.83999999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27552,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":540672,"data_stored":219335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.84799999999999998,"15min":0.86199999999999999},"min":{"1min":0.45700000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.448,"5min":2.6560000000000001,"15min":2.6560000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.79300000000000004,"15min":0.79600000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.65200000000000002}]},{"osd":1,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.88800000000000001,"15min":0.89600000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":1.397},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0569999999999999,"15min":1.0720000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.609,"5min":2.161,"15min":2.161},"last":1.272}]},{"osd":3,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.93300000000000005,"15min":0.94199999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.4119999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.1459999999999999,"15min":1.1679999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.4350000000000001,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":1.1200000000000001}]},{"osd":4,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.23,"15min":1.2509999999999999},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.587,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.236,"15min":1.2470000000000001},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.73,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.0649999999999999}]},{"osd":5,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.246,"15min":1.26},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.5489999999999999,"5min":2.048,"15min":2.048},"last":1.169},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.258,"15min":1.2869999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.323,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.5589999999999999}]},{"osd":6,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.319,"15min":1.351},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.6060000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.629},{"interface":"front","average":{"1min":1.252,"5min":1.3360000000000001,"15min":1.3500000000000001},"min":{"1min":0.90900000000000003,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.5389999999999999,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":1.2410000000000001}]},{"osd":7,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.4039999999999999,"15min":1.4239999999999999},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.6879999999999999,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.4299999999999999},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.367,"15min":1.397},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.5509999999999999,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":1.6759999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28132,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343708,"statfs":{"total":96636764160,"available":96607956992,"internally_reserved":0,"allocated":1134592,"data_stored":809703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":1,"15min":1.038},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.3759999999999999,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.131,"15min":1.1819999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.333,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":1.2130000000000001}]},{"osd":2,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.95299999999999996,"15min":0.97099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.395,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":1.224,"15min":1.274},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.3720000000000001,"5min":4.0090000000000003,"15min":4.0090000000000003},"last":0.90600000000000003}]},{"osd":3,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.3009999999999999,"15min":1.343},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.5369999999999999,"5min":3.9569999999999999,"15min":3.9569999999999999},"last":1.139},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.2030000000000001,"15min":1.2589999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.4610000000000001,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.0780000000000001}]},{"osd":4,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.2290000000000001,"15min":1.2450000000000001},"min":{"1min":0.77100000000000002,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.8839999999999999,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":1.268},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.595,"15min":1.649},"min":{"1min":0.81899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6439999999999999,"5min":4.29,"15min":4.29},"last":1.3149999999999999}]},{"osd":5,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.607,"15min":1.665},"min":{"1min":0.90900000000000003,"5min":0.874,"15min":0.874},"max":{"1min":1.7549999999999999,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.252,"5min":1.5309999999999999,"15min":1.577},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":2.2040000000000002,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":1.411}]},{"osd":6,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.6839999999999999,"15min":1.748},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":1.9690000000000001,"5min":4.3419999999999996,"15min":4.3419999999999996},"last":1.5},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.659,"15min":1.73},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.601,"5min":4.1239999999999997,"15min":4.1239999999999997},"last":1.875}]},{"osd":7,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.706,"15min":1.7709999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.0339999999999998,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":1.8080000000000001},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.486,"15min":1.516},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.0950000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":1.5880000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27552,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":540672,"data_stored":219335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 17:10:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.80700000000000005,"15min":0.82799999999999996},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.006,"5min":1.45,"15min":1.45},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.86199999999999999,"15min":0.874},"min":{"1min":0.40600000000000003,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.1439999999999999,"5min":1.294,"15min":1.294},"last":0.82299999999999995}]},{"osd":2,"last update":"Tue Nov 28 17:10:55 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.97999999999999998,"15min":1.002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.3600000000000001,"5min":1.601,"15min":1.601},"last":1.2949999999999999},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.93500000000000005,"15min":0.95199999999999996},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.4610000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.377}]},{"osd":3,"last update":"Tue Nov 28 17:10:55 2023","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.143,"15min":1.1679999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.4430000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":1.1739999999999999},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.0900000000000001,"15min":1.1279999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.4870000000000001,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.484}]},{"osd":4,"last update":"Tue Nov 28 17:10:55 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.2749999999999999,"15min":1.2989999999999999},"min":{"1min":0.78400000000000003,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.673,"5min":1.764,"15min":1.764},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.4059999999999999,"15min":1.444},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6279999999999999,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.6379999999999999}]},{"osd":5,"last update":"Tue Nov 28 17:10:55 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.407,"15min":1.4350000000000001},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.9690000000000001,"5min":2.2570000000000001,"15min":2.2570000000000001},"last":1.444},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.276,"15min":1.286},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6659999999999999,"5min":1.855,"15min":1.855},"last":1.2350000000000001}]},{"osd":6,"last update":"Tue Nov 28 17:10:55 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.3799999999999999,"15min":1.4079999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.5920000000000001,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.452,"15min":1.476},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":1.762,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.556}]},{"osd":7,"last update":"Tue Nov 28 17:10:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.377,"15min":1.3899999999999999},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.7330000000000001,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.339},{"interface":"front","average":{"1min":1.292,"5min":1.411,"15min":1.431},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.7709999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.774}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T17:11:28.619 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:11:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.869+0000 7f07a280e640 1 Processor -- start 2023-11-28T17:11:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.870+0000 7f07a280e640 1 -- start start 2023-11-28T17:11:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.871+0000 7f07a280e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f079c145c70 0x7f079c0ae1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.871+0000 7f07a280e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079c072620 con 0x7f079c071550 2023-11-28T17:11:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.871+0000 7f07a280e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079c071a10 con 0x7f079c06dbd0 2023-11-28T17:11:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.871+0000 7f07a280e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f079c0b1e10 con 0x7f079c145c70 2023-11-28T17:11:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.872+0000 7f079bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f079c145c70 0x7f079c0ae1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.872+0000 7f07937fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f079c071550 0x7f079c06d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45228/0 (socket says 172.21.15.116:45228) 2023-11-28T17:11:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.872+0000 7f07937fe640 1 -- 172.21.15.116:0/442590004 learned_addr learned my addr 172.21.15.116:0/442590004 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.872+0000 7f079bfff640 1 -- 172.21.15.116:0/442590004 >> v1:172.21.15.156:6789/0 conn(0x7f079c06dbd0 legacy=0x7f079c0ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.872+0000 7f079bfff640 1 -- 172.21.15.116:0/442590004 >> v1:172.21.15.116:6789/0 conn(0x7f079c071550 legacy=0x7f079c06d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.872+0000 7f079bfff640 1 -- 172.21.15.116:0/442590004 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079c071cf0 con 0x7f079c145c70 2023-11-28T17:11:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.873+0000 7f079bfff640 1 --2- 172.21.15.116:0/442590004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f079c145c70 0x7f079c0ae1c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f078c0097a0 tx=0x7f078c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3595f966bbfa870a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.873+0000 7f079b7fe640 1 -- 172.21.15.116:0/442590004 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c031830 con 0x7f079c145c70 2023-11-28T17:11:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.874+0000 7f079b7fe640 1 -- 172.21.15.116:0/442590004 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c031990 con 0x7f079c145c70 2023-11-28T17:11:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.874+0000 7f079b7fe640 1 -- 172.21.15.116:0/442590004 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c031d20 con 0x7f079c145c70 2023-11-28T17:11:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.874+0000 7f07a280e640 1 -- 172.21.15.116:0/442590004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f079c145c70 msgr2=0x7f079c0ae1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.874+0000 7f07a280e640 1 --2- 172.21.15.116:0/442590004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f079c145c70 0x7f079c0ae1c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f078c0097a0 tx=0x7f078c02b8d0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.874+0000 7f07a280e640 1 -- 172.21.15.116:0/442590004 shutdown_connections 2023-11-28T17:11:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.874+0000 7f07a280e640 1 --2- 172.21.15.116:0/442590004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f079c145c70 0x7f079c0ae1c0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.874+0000 7f07a280e640 1 -- 172.21.15.116:0/442590004 >> 172.21.15.116:0/442590004 conn(0x7f079c06aaf0 msgr2=0x7f079c0b0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.874+0000 7f07a280e640 1 -- 172.21.15.116:0/442590004 shutdown_connections 2023-11-28T17:11:28.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.874+0000 7f07a280e640 1 -- 172.21.15.116:0/442590004 wait complete. 2023-11-28T17:11:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.876+0000 7f07a280e640 1 Processor -- start 2023-11-28T17:11:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.876+0000 7f07a280e640 1 -- start start 2023-11-28T17:11:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.877+0000 7f07a280e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f079c06dbd0 0x7f079c1f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.877+0000 7f07a280e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f079c071550 0x7f079c1f6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.877+0000 7f07a280e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f079c145c70 0x7f079c1f6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.877+0000 7f07a280e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c1f6ec0 con 0x7f079c071550 2023-11-28T17:11:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.877+0000 7f07a280e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c1f2900 con 0x7f079c06dbd0 2023-11-28T17:11:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.877+0000 7f07a280e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079c0b0a50 con 0x7f079c145c70 2023-11-28T17:11:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.878+0000 7f07937fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f079c071550 0x7f079c1f6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.878+0000 7f079bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f079c06dbd0 0x7f079c1f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.878+0000 7f07937fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f079c071550 0x7f079c1f6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33908/0 (socket says 172.21.15.116:33908) 2023-11-28T17:11:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.878+0000 7f07937fe640 1 -- 172.21.15.116:0/3757778249 learned_addr learned my addr 172.21.15.116:0/3757778249 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.878+0000 7f07a0dab640 1 --2- 172.21.15.116:0/3757778249 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f079c145c70 0x7f079c1f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.878+0000 7f07937fe640 1 -- 172.21.15.116:0/3757778249 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f079c145c70 msgr2=0x7f079c1f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:28.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.878+0000 7f07937fe640 1 --2- 172.21.15.116:0/3757778249 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f079c145c70 0x7f079c1f6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.878+0000 7f07937fe640 1 -- 172.21.15.116:0/3757778249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f079c06dbd0 msgr2=0x7f079c1f23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.878+0000 7f07937fe640 1 --2- 172.21.15.116:0/3757778249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f079c06dbd0 0x7f079c1f23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.878+0000 7f07937fe640 1 -- 172.21.15.116:0/3757778249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c009400 con 0x7f079c071550 2023-11-28T17:11:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.878+0000 7f07937fe640 1 --2- 172.21.15.116:0/3757778249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f079c071550 0x7f079c1f6040 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f078400c4a0 tx=0x7f078400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.879+0000 7f07997fa640 1 -- 172.21.15.116:0/3757778249 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0784005ce0 con 0x7f079c071550 2023-11-28T17:11:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.879+0000 7f07997fa640 1 -- 172.21.15.116:0/3757778249 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0784005e40 con 0x7f079c071550 2023-11-28T17:11:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.879+0000 7f07997fa640 1 -- 172.21.15.116:0/3757778249 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0784007270 con 0x7f079c071550 2023-11-28T17:11:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.879+0000 7f07a280e640 1 -- 172.21.15.116:0/3757778249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f079c0b0d30 con 0x7f079c071550 2023-11-28T17:11:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.879+0000 7f07a280e640 1 -- 172.21.15.116:0/3757778249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f079c144060 con 0x7f079c071550 2023-11-28T17:11:28.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.880+0000 7f07a280e640 1 -- 172.21.15.116:0/3757778249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f079c144250 con 0x7f079c071550 2023-11-28T17:11:28.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.881+0000 7f07997fa640 1 -- 172.21.15.116:0/3757778249 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0784007490 con 0x7f079c071550 2023-11-28T17:11:28.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.882+0000 7f07997fa640 1 --2- 172.21.15.116:0/3757778249 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f078006b2a0 0x7f078006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:28.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.882+0000 7f07997fa640 1 -- 172.21.15.116:0/3757778249 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0784089010 con 0x7f079c071550 2023-11-28T17:11:28.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.882+0000 7f079bfff640 1 --2- 172.21.15.116:0/3757778249 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f078006b2a0 0x7f078006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:28.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.883+0000 7f079bfff640 1 --2- 172.21.15.116:0/3757778249 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f078006b2a0 0x7f078006d760 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f078c036000 tx=0x7f078c03c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:28.886+0000 7f07997fa640 1 -- 172.21.15.116:0/3757778249 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0784053f30 con 0x7f079c071550 2023-11-28T17:11:29.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.072+0000 7f07a280e640 1 -- 172.21.15.116:0/3757778249 --> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f079c0722b0 con 0x7f078006b2a0 2023-11-28T17:11:29.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.096+0000 7f07997fa640 1 -- 172.21.15.116:0/3757778249 <== mgr.4098 v2:172.21.15.116:6832/1077400178 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286799 (secure 0 0 0) 0x7f078c044400 con 0x7f078006b2a0 2023-11-28T17:11:29.099 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:29.103 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2023-11-28T17:11:29.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.103+0000 7f07a280e640 1 -- 172.21.15.116:0/3757778249 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f078006b2a0 msgr2=0x7f078006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.103+0000 7f07a280e640 1 --2- 172.21.15.116:0/3757778249 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f078006b2a0 0x7f078006d760 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f078c036000 tx=0x7f078c03c040 comp rx=0 tx=0).stop 2023-11-28T17:11:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.103+0000 7f07a280e640 1 -- 172.21.15.116:0/3757778249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f079c071550 msgr2=0x7f079c1f6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.103+0000 7f07a280e640 1 --2- 172.21.15.116:0/3757778249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f079c071550 0x7f079c1f6040 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f078400c4a0 tx=0x7f078400c960 comp rx=0 tx=0).stop 2023-11-28T17:11:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.103+0000 7f07a280e640 1 -- 172.21.15.116:0/3757778249 shutdown_connections 2023-11-28T17:11:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.103+0000 7f07a280e640 1 --2- 172.21.15.116:0/3757778249 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f078006b2a0 0x7f078006d760 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.103+0000 7f07a280e640 1 --2- 172.21.15.116:0/3757778249 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f079c145c70 0x7f079c1f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.103+0000 7f07a280e640 1 --2- 172.21.15.116:0/3757778249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f079c071550 0x7f079c1f6040 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.103+0000 7f07a280e640 1 --2- 172.21.15.116:0/3757778249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f079c06dbd0 0x7f079c1f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.103+0000 7f07a280e640 1 -- 172.21.15.116:0/3757778249 >> 172.21.15.116:0/3757778249 conn(0x7f079c06aaf0 msgr2=0x7f079c1402a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.103+0000 7f07a280e640 1 -- 172.21.15.116:0/3757778249 shutdown_connections 2023-11-28T17:11:29.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.103+0000 7f07a280e640 1 -- 172.21.15.116:0/3757778249 wait complete. 2023-11-28T17:11:29.125 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2023-11-28T17:11:27.084886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238212,"kb_used_data":5384,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736508,"statfs":{"total":773094113280,"available":772850184192,"internally_reserved":0,"allocated":5513216,"data_stored":2935416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.908359"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.227473+0000","last_change":"2023-11-28T17:11:24.226060+0000","last_active":"2023-11-28T17:11:24.227473+0000","last_peered":"2023-11-28T17:11:24.227473+0000","last_clean":"2023-11-28T17:11:24.227473+0000","last_became_active":"2023-11-28T17:11:24.225545+0000","last_became_peered":"2023-11-28T17:11:24.225545+0000","last_unstale":"2023-11-28T17:11:24.227473+0000","last_undegraded":"2023-11-28T17:11:24.227473+0000","last_fullsized":"2023-11-28T17:11:24.227473+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:53:17.135956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.247149+0000","last_change":"2023-11-28T17:11:24.237732+0000","last_active":"2023-11-28T17:11:24.247149+0000","last_peered":"2023-11-28T17:11:24.247149+0000","last_clean":"2023-11-28T17:11:24.247149+0000","last_became_active":"2023-11-28T17:11:24.237443+0000","last_became_peered":"2023-11-28T17:11:24.237443+0000","last_unstale":"2023-11-28T17:11:24.247149+0000","last_undegraded":"2023-11-28T17:11:24.247149+0000","last_fullsized":"2023-11-28T17:11:24.247149+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:51:25.683278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242194+0000","last_change":"2023-11-28T17:11:24.238573+0000","last_active":"2023-11-28T17:11:24.242194+0000","last_peered":"2023-11-28T17:11:24.242194+0000","last_clean":"2023-11-28T17:11:24.242194+0000","last_became_active":"2023-11-28T17:11:24.238159+0000","last_became_peered":"2023-11-28T17:11:24.238159+0000","last_unstale":"2023-11-28T17:11:24.242194+0000","last_undegraded":"2023-11-28T17:11:24.242194+0000","last_fullsized":"2023-11-28T17:11:24.242194+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:42.774254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.250211+0000","last_change":"2023-11-28T17:11:24.243165+0000","last_active":"2023-11-28T17:11:24.250211+0000","last_peered":"2023-11-28T17:11:24.250211+0000","last_clean":"2023-11-28T17:11:24.250211+0000","last_became_active":"2023-11-28T17:11:24.237530+0000","last_became_peered":"2023-11-28T17:11:24.237530+0000","last_unstale":"2023-11-28T17:11:24.250211+0000","last_undegraded":"2023-11-28T17:11:24.250211+0000","last_fullsized":"2023-11-28T17:11:24.250211+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:30.990592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223282+0000","last_change":"2023-11-28T17:11:24.217095+0000","last_active":"2023-11-28T17:11:24.223282+0000","last_peered":"2023-11-28T17:11:24.223282+0000","last_clean":"2023-11-28T17:11:24.223282+0000","last_became_active":"2023-11-28T17:11:24.216795+0000","last_became_peered":"2023-11-28T17:11:24.216795+0000","last_unstale":"2023-11-28T17:11:24.223282+0000","last_undegraded":"2023-11-28T17:11:24.223282+0000","last_fullsized":"2023-11-28T17:11:24.223282+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:12:49.613982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.250564+0000","last_change":"2023-11-28T17:11:24.247005+0000","last_active":"2023-11-28T17:11:24.250564+0000","last_peered":"2023-11-28T17:11:24.250564+0000","last_clean":"2023-11-28T17:11:24.250564+0000","last_became_active":"2023-11-28T17:11:24.246728+0000","last_became_peered":"2023-11-28T17:11:24.246728+0000","last_unstale":"2023-11-28T17:11:24.250564+0000","last_undegraded":"2023-11-28T17:11:24.250564+0000","last_fullsized":"2023-11-28T17:11:24.250564+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:00.298985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.241901+0000","last_change":"2023-11-28T17:11:24.236686+0000","last_active":"2023-11-28T17:11:24.241901+0000","last_peered":"2023-11-28T17:11:24.241901+0000","last_clean":"2023-11-28T17:11:24.241901+0000","last_became_active":"2023-11-28T17:11:24.236436+0000","last_became_peered":"2023-11-28T17:11:24.236436+0000","last_unstale":"2023-11-28T17:11:24.241901+0000","last_undegraded":"2023-11-28T17:11:24.241901+0000","last_fullsized":"2023-11-28T17:11:24.241901+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:13:10.490828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.250065+0000","last_change":"2023-11-28T17:11:24.244148+0000","last_active":"2023-11-28T17:11:24.250065+0000","last_peered":"2023-11-28T17:11:24.250065+0000","last_clean":"2023-11-28T17:11:24.250065+0000","last_became_active":"2023-11-28T17:11:24.243301+0000","last_became_peered":"2023-11-28T17:11:24.243301+0000","last_unstale":"2023-11-28T17:11:24.250065+0000","last_undegraded":"2023-11-28T17:11:24.250065+0000","last_fullsized":"2023-11-28T17:11:24.250065+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:51:25.683278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.249041+0000","last_change":"2023-11-28T17:11:24.241462+0000","last_active":"2023-11-28T17:11:24.249041+0000","last_peered":"2023-11-28T17:11:24.249041+0000","last_clean":"2023-11-28T17:11:24.249041+0000","last_became_active":"2023-11-28T17:11:24.241183+0000","last_became_peered":"2023-11-28T17:11:24.241183+0000","last_unstale":"2023-11-28T17:11:24.249041+0000","last_undegraded":"2023-11-28T17:11:24.249041+0000","last_fullsized":"2023-11-28T17:11:24.249041+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:12:49.613982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.241669+0000","last_change":"2023-11-28T17:11:24.234407+0000","last_active":"2023-11-28T17:11:24.241669+0000","last_peered":"2023-11-28T17:11:24.241669+0000","last_clean":"2023-11-28T17:11:24.241669+0000","last_became_active":"2023-11-28T17:11:24.233160+0000","last_became_peered":"2023-11-28T17:11:24.233160+0000","last_unstale":"2023-11-28T17:11:24.241669+0000","last_undegraded":"2023-11-28T17:11:24.241669+0000","last_fullsized":"2023-11-28T17:11:24.241669+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:55.079230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248864+0000","last_change":"2023-11-28T17:11:24.236457+0000","last_active":"2023-11-28T17:11:24.248864+0000","last_peered":"2023-11-28T17:11:24.248864+0000","last_clean":"2023-11-28T17:11:24.248864+0000","last_became_active":"2023-11-28T17:11:24.235921+0000","last_became_peered":"2023-11-28T17:11:24.235921+0000","last_unstale":"2023-11-28T17:11:24.248864+0000","last_undegraded":"2023-11-28T17:11:24.248864+0000","last_fullsized":"2023-11-28T17:11:24.248864+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:18:08.133197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223241+0000","last_change":"2023-11-28T17:11:24.212809+0000","last_active":"2023-11-28T17:11:24.223241+0000","last_peered":"2023-11-28T17:11:24.223241+0000","last_clean":"2023-11-28T17:11:24.223241+0000","last_became_active":"2023-11-28T17:11:24.212489+0000","last_became_peered":"2023-11-28T17:11:24.212489+0000","last_unstale":"2023-11-28T17:11:24.223241+0000","last_undegraded":"2023-11-28T17:11:24.223241+0000","last_fullsized":"2023-11-28T17:11:24.223241+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:22.356893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.250727+0000","last_change":"2023-11-28T17:11:24.248718+0000","last_active":"2023-11-28T17:11:24.250727+0000","last_peered":"2023-11-28T17:11:24.250727+0000","last_clean":"2023-11-28T17:11:24.250727+0000","last_became_active":"2023-11-28T17:11:24.248126+0000","last_became_peered":"2023-11-28T17:11:24.248126+0000","last_unstale":"2023-11-28T17:11:24.250727+0000","last_undegraded":"2023-11-28T17:11:24.250727+0000","last_fullsized":"2023-11-28T17:11:24.250727+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:58:02.404658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.251048+0000","last_change":"2023-11-28T17:11:24.246379+0000","last_active":"2023-11-28T17:11:24.251048+0000","last_peered":"2023-11-28T17:11:24.251048+0000","last_clean":"2023-11-28T17:11:24.251048+0000","last_became_active":"2023-11-28T17:11:24.246008+0000","last_became_peered":"2023-11-28T17:11:24.246008+0000","last_unstale":"2023-11-28T17:11:24.251048+0000","last_undegraded":"2023-11-28T17:11:24.251048+0000","last_fullsized":"2023-11-28T17:11:24.251048+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:12:49.613982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.251299+0000","last_change":"2023-11-28T17:11:24.247659+0000","last_active":"2023-11-28T17:11:24.251299+0000","last_peered":"2023-11-28T17:11:24.251299+0000","last_clean":"2023-11-28T17:11:24.251299+0000","last_became_active":"2023-11-28T17:11:24.243488+0000","last_became_peered":"2023-11-28T17:11:24.243488+0000","last_unstale":"2023-11-28T17:11:24.251299+0000","last_undegraded":"2023-11-28T17:11:24.251299+0000","last_fullsized":"2023-11-28T17:11:24.251299+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:05.450433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.247521+0000","last_change":"2023-11-28T17:11:24.239489+0000","last_active":"2023-11-28T17:11:24.247521+0000","last_peered":"2023-11-28T17:11:24.247521+0000","last_clean":"2023-11-28T17:11:24.247521+0000","last_became_active":"2023-11-28T17:11:24.239123+0000","last_became_peered":"2023-11-28T17:11:24.239123+0000","last_unstale":"2023-11-28T17:11:24.247521+0000","last_undegraded":"2023-11-28T17:11:24.247521+0000","last_fullsized":"2023-11-28T17:11:24.247521+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:58:02.404658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.224067+0000","last_change":"2023-11-28T17:11:24.219801+0000","last_active":"2023-11-28T17:11:24.224067+0000","last_peered":"2023-11-28T17:11:24.224067+0000","last_clean":"2023-11-28T17:11:24.224067+0000","last_became_active":"2023-11-28T17:11:24.218728+0000","last_became_peered":"2023-11-28T17:11:24.218728+0000","last_unstale":"2023-11-28T17:11:24.224067+0000","last_undegraded":"2023-11-28T17:11:24.224067+0000","last_fullsized":"2023-11-28T17:11:24.224067+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:32.366526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.255226+0000","last_change":"2023-11-28T17:11:24.251802+0000","last_active":"2023-11-28T17:11:24.255226+0000","last_peered":"2023-11-28T17:11:24.255226+0000","last_clean":"2023-11-28T17:11:24.255226+0000","last_became_active":"2023-11-28T17:11:24.251539+0000","last_became_peered":"2023-11-28T17:11:24.251539+0000","last_unstale":"2023-11-28T17:11:24.255226+0000","last_undegraded":"2023-11-28T17:11:24.255226+0000","last_fullsized":"2023-11-28T17:11:24.255226+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:06.002391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223422+0000","last_change":"2023-11-28T17:11:24.220740+0000","last_active":"2023-11-28T17:11:24.223422+0000","last_peered":"2023-11-28T17:11:24.223422+0000","last_clean":"2023-11-28T17:11:24.223422+0000","last_became_active":"2023-11-28T17:11:24.220535+0000","last_became_peered":"2023-11-28T17:11:24.220535+0000","last_unstale":"2023-11-28T17:11:24.223422+0000","last_undegraded":"2023-11-28T17:11:24.223422+0000","last_fullsized":"2023-11-28T17:11:24.223422+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:51:25.683278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242385+0000","last_change":"2023-11-28T17:11:24.239214+0000","last_active":"2023-11-28T17:11:24.242385+0000","last_peered":"2023-11-28T17:11:24.242385+0000","last_clean":"2023-11-28T17:11:24.242385+0000","last_became_active":"2023-11-28T17:11:24.238992+0000","last_became_peered":"2023-11-28T17:11:24.238992+0000","last_unstale":"2023-11-28T17:11:24.242385+0000","last_undegraded":"2023-11-28T17:11:24.242385+0000","last_fullsized":"2023-11-28T17:11:24.242385+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:10:44.265974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.249849+0000","last_change":"2023-11-28T17:11:24.236502+0000","last_active":"2023-11-28T17:11:24.249849+0000","last_peered":"2023-11-28T17:11:24.249849+0000","last_clean":"2023-11-28T17:11:24.249849+0000","last_became_active":"2023-11-28T17:11:24.235482+0000","last_became_peered":"2023-11-28T17:11:24.235482+0000","last_unstale":"2023-11-28T17:11:24.249849+0000","last_undegraded":"2023-11-28T17:11:24.249849+0000","last_fullsized":"2023-11-28T17:11:24.249849+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:22.356893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.227177+0000","last_change":"2023-11-28T17:11:24.219330+0000","last_active":"2023-11-28T17:11:24.227177+0000","last_peered":"2023-11-28T17:11:24.227177+0000","last_clean":"2023-11-28T17:11:24.227177+0000","last_became_active":"2023-11-28T17:11:24.217191+0000","last_became_peered":"2023-11-28T17:11:24.217191+0000","last_unstale":"2023-11-28T17:11:24.227177+0000","last_undegraded":"2023-11-28T17:11:24.227177+0000","last_fullsized":"2023-11-28T17:11:24.227177+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:41:38.593938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.217525+0000","last_change":"2023-11-28T17:11:24.213940+0000","last_active":"2023-11-28T17:11:24.217525+0000","last_peered":"2023-11-28T17:11:24.217525+0000","last_clean":"2023-11-28T17:11:24.217525+0000","last_became_active":"2023-11-28T17:11:24.212127+0000","last_became_peered":"2023-11-28T17:11:24.212127+0000","last_unstale":"2023-11-28T17:11:24.217525+0000","last_undegraded":"2023-11-28T17:11:24.217525+0000","last_fullsized":"2023-11-28T17:11:24.217525+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:26:13.848230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248045+0000","last_change":"2023-11-28T17:11:24.244113+0000","last_active":"2023-11-28T17:11:24.248045+0000","last_peered":"2023-11-28T17:11:24.248045+0000","last_clean":"2023-11-28T17:11:24.248045+0000","last_became_active":"2023-11-28T17:11:24.243833+0000","last_became_peered":"2023-11-28T17:11:24.243833+0000","last_unstale":"2023-11-28T17:11:24.248045+0000","last_undegraded":"2023-11-28T17:11:24.248045+0000","last_fullsized":"2023-11-28T17:11:24.248045+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:58:02.404658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223423+0000","last_change":"2023-11-28T17:11:24.219782+0000","last_active":"2023-11-28T17:11:24.223423+0000","last_peered":"2023-11-28T17:11:24.223423+0000","last_clean":"2023-11-28T17:11:24.223423+0000","last_became_active":"2023-11-28T17:11:24.219209+0000","last_became_peered":"2023-11-28T17:11:24.219209+0000","last_unstale":"2023-11-28T17:11:24.223423+0000","last_undegraded":"2023-11-28T17:11:24.223423+0000","last_fullsized":"2023-11-28T17:11:24.223423+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:30:34.918467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.252215+0000","last_change":"2023-11-28T17:11:24.250257+0000","last_active":"2023-11-28T17:11:24.252215+0000","last_peered":"2023-11-28T17:11:24.252215+0000","last_clean":"2023-11-28T17:11:24.252215+0000","last_became_active":"2023-11-28T17:11:24.245657+0000","last_became_peered":"2023-11-28T17:11:24.245657+0000","last_unstale":"2023-11-28T17:11:24.252215+0000","last_undegraded":"2023-11-28T17:11:24.252215+0000","last_fullsized":"2023-11-28T17:11:24.252215+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:32.366526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223816+0000","last_change":"2023-11-28T17:11:24.222198+0000","last_active":"2023-11-28T17:11:24.223816+0000","last_peered":"2023-11-28T17:11:24.223816+0000","last_clean":"2023-11-28T17:11:24.223816+0000","last_became_active":"2023-11-28T17:11:24.218597+0000","last_became_peered":"2023-11-28T17:11:24.218597+0000","last_unstale":"2023-11-28T17:11:24.223816+0000","last_undegraded":"2023-11-28T17:11:24.223816+0000","last_fullsized":"2023-11-28T17:11:24.223816+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:28.215714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.218097+0000","last_change":"2023-11-28T17:11:24.216110+0000","last_active":"2023-11-28T17:11:24.218097+0000","last_peered":"2023-11-28T17:11:24.218097+0000","last_clean":"2023-11-28T17:11:24.218097+0000","last_became_active":"2023-11-28T17:11:24.215793+0000","last_became_peered":"2023-11-28T17:11:24.215793+0000","last_unstale":"2023-11-28T17:11:24.218097+0000","last_undegraded":"2023-11-28T17:11:24.218097+0000","last_fullsized":"2023-11-28T17:11:24.218097+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:33:28.388804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.239396+0000","last_change":"2023-11-28T17:11:24.235123+0000","last_active":"2023-11-28T17:11:24.239396+0000","last_peered":"2023-11-28T17:11:24.239396+0000","last_clean":"2023-11-28T17:11:24.239396+0000","last_became_active":"2023-11-28T17:11:24.234739+0000","last_became_peered":"2023-11-28T17:11:24.234739+0000","last_unstale":"2023-11-28T17:11:24.239396+0000","last_undegraded":"2023-11-28T17:11:24.239396+0000","last_fullsized":"2023-11-28T17:11:24.239396+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:05.450433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223837+0000","last_change":"2023-11-28T17:11:24.216242+0000","last_active":"2023-11-28T17:11:24.223837+0000","last_peered":"2023-11-28T17:11:24.223837+0000","last_clean":"2023-11-28T17:11:24.223837+0000","last_became_active":"2023-11-28T17:11:24.215959+0000","last_became_peered":"2023-11-28T17:11:24.215959+0000","last_unstale":"2023-11-28T17:11:24.223837+0000","last_undegraded":"2023-11-28T17:11:24.223837+0000","last_fullsized":"2023-11-28T17:11:24.223837+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:05.450433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.230359+0000","last_change":"2023-11-28T17:11:24.219508+0000","last_active":"2023-11-28T17:11:24.230359+0000","last_peered":"2023-11-28T17:11:24.230359+0000","last_clean":"2023-11-28T17:11:24.230359+0000","last_became_active":"2023-11-28T17:11:24.218725+0000","last_became_peered":"2023-11-28T17:11:24.218725+0000","last_unstale":"2023-11-28T17:11:24.230359+0000","last_undegraded":"2023-11-28T17:11:24.230359+0000","last_fullsized":"2023-11-28T17:11:24.230359+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:43:05.206928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248737+0000","last_change":"2023-11-28T17:11:24.243621+0000","last_active":"2023-11-28T17:11:24.248737+0000","last_peered":"2023-11-28T17:11:24.248737+0000","last_clean":"2023-11-28T17:11:24.248737+0000","last_became_active":"2023-11-28T17:11:24.237176+0000","last_became_peered":"2023-11-28T17:11:24.237176+0000","last_unstale":"2023-11-28T17:11:24.248737+0000","last_undegraded":"2023-11-28T17:11:24.248737+0000","last_fullsized":"2023-11-28T17:11:24.248737+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:13:37.338728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.240656+0000","last_change":"2023-11-28T17:11:24.234796+0000","last_active":"2023-11-28T17:11:24.240656+0000","last_peered":"2023-11-28T17:11:24.240656+0000","last_clean":"2023-11-28T17:11:24.240656+0000","last_became_active":"2023-11-28T17:11:24.234427+0000","last_became_peered":"2023-11-28T17:11:24.234427+0000","last_unstale":"2023-11-28T17:11:24.240656+0000","last_undegraded":"2023-11-28T17:11:24.240656+0000","last_fullsized":"2023-11-28T17:11:24.240656+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:42.774254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"47'4","reported_seq":15,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.250149+0000","last_change":"2023-11-28T17:11:23.216512+0000","last_active":"2023-11-28T17:11:24.250149+0000","last_peered":"2023-11-28T17:11:24.250149+0000","last_clean":"2023-11-28T17:11:24.250149+0000","last_became_active":"2023-11-28T17:11:23.216241+0000","last_became_peered":"2023-11-28T17:11:23.216241+0000","last_unstale":"2023-11-28T17:11:24.250149+0000","last_undegraded":"2023-11-28T17:11:24.250149+0000","last_fullsized":"2023-11-28T17:11:24.250149+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:06:02.961760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.236381+0000","last_change":"2023-11-28T17:11:24.231592+0000","last_active":"2023-11-28T17:11:24.236381+0000","last_peered":"2023-11-28T17:11:24.236381+0000","last_clean":"2023-11-28T17:11:24.236381+0000","last_became_active":"2023-11-28T17:11:24.231316+0000","last_became_peered":"2023-11-28T17:11:24.231316+0000","last_unstale":"2023-11-28T17:11:24.236381+0000","last_undegraded":"2023-11-28T17:11:24.236381+0000","last_fullsized":"2023-11-28T17:11:24.236381+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:18:08.133197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.234878+0000","last_change":"2023-11-28T17:11:23.223688+0000","last_active":"2023-11-28T17:11:24.234878+0000","last_peered":"2023-11-28T17:11:24.234878+0000","last_clean":"2023-11-28T17:11:24.234878+0000","last_became_active":"2023-11-28T17:11:23.223318+0000","last_became_peered":"2023-11-28T17:11:23.223318+0000","last_unstale":"2023-11-28T17:11:24.234878+0000","last_undegraded":"2023-11-28T17:11:24.234878+0000","last_fullsized":"2023-11-28T17:11:24.234878+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:28:09.189390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.247215+0000","last_change":"2023-11-28T17:11:24.240363+0000","last_active":"2023-11-28T17:11:24.247215+0000","last_peered":"2023-11-28T17:11:24.247215+0000","last_clean":"2023-11-28T17:11:24.247215+0000","last_became_active":"2023-11-28T17:11:24.240093+0000","last_became_peered":"2023-11-28T17:11:24.240093+0000","last_unstale":"2023-11-28T17:11:24.247215+0000","last_undegraded":"2023-11-28T17:11:24.247215+0000","last_fullsized":"2023-11-28T17:11:24.247215+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:06:03.975512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.249714+0000","last_change":"2023-11-28T17:11:23.215395+0000","last_active":"2023-11-28T17:11:24.249714+0000","last_peered":"2023-11-28T17:11:24.249714+0000","last_clean":"2023-11-28T17:11:24.249714+0000","last_became_active":"2023-11-28T17:11:23.215112+0000","last_became_peered":"2023-11-28T17:11:23.215112+0000","last_unstale":"2023-11-28T17:11:24.249714+0000","last_undegraded":"2023-11-28T17:11:24.249714+0000","last_fullsized":"2023-11-28T17:11:24.249714+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:09:25.806613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248847+0000","last_change":"2023-11-28T17:11:24.235933+0000","last_active":"2023-11-28T17:11:24.248847+0000","last_peered":"2023-11-28T17:11:24.248847+0000","last_clean":"2023-11-28T17:11:24.248847+0000","last_became_active":"2023-11-28T17:11:24.235622+0000","last_became_peered":"2023-11-28T17:11:24.235622+0000","last_unstale":"2023-11-28T17:11:24.248847+0000","last_undegraded":"2023-11-28T17:11:24.248847+0000","last_fullsized":"2023-11-28T17:11:24.248847+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:06:03.975512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1c","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242245+0000","last_change":"2023-11-28T17:11:23.222079+0000","last_active":"2023-11-28T17:11:24.242245+0000","last_peered":"2023-11-28T17:11:24.242245+0000","last_clean":"2023-11-28T17:11:24.242245+0000","last_became_active":"2023-11-28T17:11:23.220772+0000","last_became_peered":"2023-11-28T17:11:23.220772+0000","last_unstale":"2023-11-28T17:11:24.242245+0000","last_undegraded":"2023-11-28T17:11:24.242245+0000","last_fullsized":"2023-11-28T17:11:24.242245+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:30:27.588631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.217037+0000","last_change":"2023-11-28T17:11:24.213431+0000","last_active":"2023-11-28T17:11:24.217037+0000","last_peered":"2023-11-28T17:11:24.217037+0000","last_clean":"2023-11-28T17:11:24.217037+0000","last_became_active":"2023-11-28T17:11:24.212942+0000","last_became_peered":"2023-11-28T17:11:24.212942+0000","last_unstale":"2023-11-28T17:11:24.217037+0000","last_undegraded":"2023-11-28T17:11:24.217037+0000","last_fullsized":"2023-11-28T17:11:24.217037+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:27:40.286894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.250282+0000","last_change":"2023-11-28T17:11:23.213829+0000","last_active":"2023-11-28T17:11:24.250282+0000","last_peered":"2023-11-28T17:11:24.250282+0000","last_clean":"2023-11-28T17:11:24.250282+0000","last_became_active":"2023-11-28T17:11:23.213358+0000","last_became_peered":"2023-11-28T17:11:23.213358+0000","last_unstale":"2023-11-28T17:11:24.250282+0000","last_undegraded":"2023-11-28T17:11:24.250282+0000","last_fullsized":"2023-11-28T17:11:24.250282+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:21.343141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242383+0000","last_change":"2023-11-28T17:11:24.239227+0000","last_active":"2023-11-28T17:11:24.242383+0000","last_peered":"2023-11-28T17:11:24.242383+0000","last_clean":"2023-11-28T17:11:24.242383+0000","last_became_active":"2023-11-28T17:11:24.238796+0000","last_became_peered":"2023-11-28T17:11:24.238796+0000","last_unstale":"2023-11-28T17:11:24.242383+0000","last_undegraded":"2023-11-28T17:11:24.242383+0000","last_fullsized":"2023-11-28T17:11:24.242383+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:06:58.499355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.233066+0000","last_change":"2023-11-28T17:11:23.222991+0000","last_active":"2023-11-28T17:11:24.233066+0000","last_peered":"2023-11-28T17:11:24.233066+0000","last_clean":"2023-11-28T17:11:24.233066+0000","last_became_active":"2023-11-28T17:11:23.222681+0000","last_became_peered":"2023-11-28T17:11:23.222681+0000","last_unstale":"2023-11-28T17:11:24.233066+0000","last_undegraded":"2023-11-28T17:11:24.233066+0000","last_fullsized":"2023-11-28T17:11:24.233066+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:15:23.640505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.239081+0000","last_change":"2023-11-28T17:11:24.227524+0000","last_active":"2023-11-28T17:11:24.239081+0000","last_peered":"2023-11-28T17:11:24.239081+0000","last_clean":"2023-11-28T17:11:24.239081+0000","last_became_active":"2023-11-28T17:11:24.227032+0000","last_became_peered":"2023-11-28T17:11:24.227032+0000","last_unstale":"2023-11-28T17:11:24.239081+0000","last_undegraded":"2023-11-28T17:11:24.239081+0000","last_fullsized":"2023-11-28T17:11:24.239081+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:30:34.918467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.206944+0000","last_change":"2023-11-28T17:11:23.221821+0000","last_active":"2023-11-28T17:11:24.206944+0000","last_peered":"2023-11-28T17:11:24.206944+0000","last_clean":"2023-11-28T17:11:24.206944+0000","last_became_active":"2023-11-28T17:11:23.221506+0000","last_became_peered":"2023-11-28T17:11:23.221506+0000","last_unstale":"2023-11-28T17:11:24.206944+0000","last_undegraded":"2023-11-28T17:11:24.206944+0000","last_fullsized":"2023-11-28T17:11:24.206944+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:50:21.343141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.218073+0000","last_change":"2023-11-28T17:11:24.216144+0000","last_active":"2023-11-28T17:11:24.218073+0000","last_peered":"2023-11-28T17:11:24.218073+0000","last_clean":"2023-11-28T17:11:24.218073+0000","last_became_active":"2023-11-28T17:11:24.215875+0000","last_became_peered":"2023-11-28T17:11:24.215875+0000","last_unstale":"2023-11-28T17:11:24.218073+0000","last_undegraded":"2023-11-28T17:11:24.218073+0000","last_fullsized":"2023-11-28T17:11:24.218073+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:37:46.364504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.212985+0000","last_change":"2023-11-28T17:11:23.220657+0000","last_active":"2023-11-28T17:11:24.212985+0000","last_peered":"2023-11-28T17:11:24.212985+0000","last_clean":"2023-11-28T17:11:24.212985+0000","last_became_active":"2023-11-28T17:11:23.219057+0000","last_became_peered":"2023-11-28T17:11:23.219057+0000","last_unstale":"2023-11-28T17:11:24.212985+0000","last_undegraded":"2023-11-28T17:11:24.212985+0000","last_fullsized":"2023-11-28T17:11:24.212985+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:29.976839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.221840+0000","last_change":"2023-11-28T17:11:24.219719+0000","last_active":"2023-11-28T17:11:24.221840+0000","last_peered":"2023-11-28T17:11:24.221840+0000","last_clean":"2023-11-28T17:11:24.221840+0000","last_became_active":"2023-11-28T17:11:24.219430+0000","last_became_peered":"2023-11-28T17:11:24.219430+0000","last_unstale":"2023-11-28T17:11:24.221840+0000","last_undegraded":"2023-11-28T17:11:24.221840+0000","last_fullsized":"2023-11-28T17:11:24.221840+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:41:38.593938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.211151+0000","last_change":"2023-11-28T17:11:23.225546+0000","last_active":"2023-11-28T17:11:24.211151+0000","last_peered":"2023-11-28T17:11:24.211151+0000","last_clean":"2023-11-28T17:11:24.211151+0000","last_became_active":"2023-11-28T17:11:23.225200+0000","last_became_peered":"2023-11-28T17:11:23.225200+0000","last_unstale":"2023-11-28T17:11:24.211151+0000","last_undegraded":"2023-11-28T17:11:24.211151+0000","last_fullsized":"2023-11-28T17:11:24.211151+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:12:03.116329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.219487+0000","last_change":"2023-11-28T17:11:24.215538+0000","last_active":"2023-11-28T17:11:24.219487+0000","last_peered":"2023-11-28T17:11:24.219487+0000","last_clean":"2023-11-28T17:11:24.219487+0000","last_became_active":"2023-11-28T17:11:24.215242+0000","last_became_peered":"2023-11-28T17:11:24.215242+0000","last_unstale":"2023-11-28T17:11:24.219487+0000","last_undegraded":"2023-11-28T17:11:24.219487+0000","last_fullsized":"2023-11-28T17:11:24.219487+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:37.638014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.217310+0000","last_change":"2023-11-28T17:11:23.218173+0000","last_active":"2023-11-28T17:11:24.217310+0000","last_peered":"2023-11-28T17:11:24.217310+0000","last_clean":"2023-11-28T17:11:24.217310+0000","last_became_active":"2023-11-28T17:11:23.217882+0000","last_became_peered":"2023-11-28T17:11:23.217882+0000","last_unstale":"2023-11-28T17:11:24.217310+0000","last_undegraded":"2023-11-28T17:11:24.217310+0000","last_fullsized":"2023-11-28T17:11:24.217310+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:23:05.792484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.252174+0000","last_change":"2023-11-28T17:11:24.250103+0000","last_active":"2023-11-28T17:11:24.252174+0000","last_peered":"2023-11-28T17:11:24.252174+0000","last_clean":"2023-11-28T17:11:24.252174+0000","last_became_active":"2023-11-28T17:11:24.246150+0000","last_became_peered":"2023-11-28T17:11:24.246150+0000","last_unstale":"2023-11-28T17:11:24.252174+0000","last_undegraded":"2023-11-28T17:11:24.252174+0000","last_fullsized":"2023-11-28T17:11:24.252174+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:30:34.918467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"47'3","reported_seq":14,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.232609+0000","last_change":"2023-11-28T17:11:23.214497+0000","last_active":"2023-11-28T17:11:24.232609+0000","last_peered":"2023-11-28T17:11:24.232609+0000","last_clean":"2023-11-28T17:11:24.232609+0000","last_became_active":"2023-11-28T17:11:23.214213+0000","last_became_peered":"2023-11-28T17:11:23.214213+0000","last_unstale":"2023-11-28T17:11:24.232609+0000","last_undegraded":"2023-11-28T17:11:24.232609+0000","last_fullsized":"2023-11-28T17:11:24.232609+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:18:07.119444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.254787+0000","last_change":"2023-11-28T17:11:24.249758+0000","last_active":"2023-11-28T17:11:24.254787+0000","last_peered":"2023-11-28T17:11:24.254787+0000","last_clean":"2023-11-28T17:11:24.254787+0000","last_became_active":"2023-11-28T17:11:24.249467+0000","last_became_peered":"2023-11-28T17:11:24.249467+0000","last_unstale":"2023-11-28T17:11:24.254787+0000","last_undegraded":"2023-11-28T17:11:24.254787+0000","last_fullsized":"2023-11-28T17:11:24.254787+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:51:07.135164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"47'6","reported_seq":17,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.255532+0000","last_change":"2023-11-28T17:11:23.223026+0000","last_active":"2023-11-28T17:11:24.255532+0000","last_peered":"2023-11-28T17:11:24.255532+0000","last_clean":"2023-11-28T17:11:24.255532+0000","last_became_active":"2023-11-28T17:11:23.222728+0000","last_became_peered":"2023-11-28T17:11:23.222728+0000","last_unstale":"2023-11-28T17:11:24.255532+0000","last_undegraded":"2023-11-28T17:11:24.255532+0000","last_fullsized":"2023-11-28T17:11:24.255532+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:05:48.739125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.249688+0000","last_change":"2023-11-28T17:11:24.245271+0000","last_active":"2023-11-28T17:11:24.249688+0000","last_peered":"2023-11-28T17:11:24.249688+0000","last_clean":"2023-11-28T17:11:24.249688+0000","last_became_active":"2023-11-28T17:11:24.243901+0000","last_became_peered":"2023-11-28T17:11:24.243901+0000","last_unstale":"2023-11-28T17:11:24.249688+0000","last_undegraded":"2023-11-28T17:11:24.249688+0000","last_fullsized":"2023-11-28T17:11:24.249688+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:23:32.010363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.249177+0000","last_change":"2023-11-28T17:11:23.224266+0000","last_active":"2023-11-28T17:11:24.249177+0000","last_peered":"2023-11-28T17:11:24.249177+0000","last_clean":"2023-11-28T17:11:24.249177+0000","last_became_active":"2023-11-28T17:11:23.223155+0000","last_became_peered":"2023-11-28T17:11:23.223155+0000","last_unstale":"2023-11-28T17:11:24.249177+0000","last_undegraded":"2023-11-28T17:11:24.249177+0000","last_fullsized":"2023-11-28T17:11:24.249177+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:04.988639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.226916+0000","last_change":"2023-11-28T17:11:24.219896+0000","last_active":"2023-11-28T17:11:24.226916+0000","last_peered":"2023-11-28T17:11:24.226916+0000","last_clean":"2023-11-28T17:11:24.226916+0000","last_became_active":"2023-11-28T17:11:24.218800+0000","last_became_peered":"2023-11-28T17:11:24.218800+0000","last_unstale":"2023-11-28T17:11:24.226916+0000","last_undegraded":"2023-11-28T17:11:24.226916+0000","last_fullsized":"2023-11-28T17:11:24.226916+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:56.943404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.208333+0000","last_change":"2023-11-28T17:11:23.222532+0000","last_active":"2023-11-28T17:11:24.208333+0000","last_peered":"2023-11-28T17:11:24.208333+0000","last_clean":"2023-11-28T17:11:24.208333+0000","last_became_active":"2023-11-28T17:11:23.222273+0000","last_became_peered":"2023-11-28T17:11:23.222273+0000","last_unstale":"2023-11-28T17:11:24.208333+0000","last_undegraded":"2023-11-28T17:11:24.208333+0000","last_fullsized":"2023-11-28T17:11:24.208333+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:31.352773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.233896+0000","last_change":"2023-11-28T17:11:24.231913+0000","last_active":"2023-11-28T17:11:24.233896+0000","last_peered":"2023-11-28T17:11:24.233896+0000","last_clean":"2023-11-28T17:11:24.233896+0000","last_became_active":"2023-11-28T17:11:24.231595+0000","last_became_peered":"2023-11-28T17:11:24.231595+0000","last_unstale":"2023-11-28T17:11:24.233896+0000","last_undegraded":"2023-11-28T17:11:24.233896+0000","last_fullsized":"2023-11-28T17:11:24.233896+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:27:31.482201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.211250+0000","last_change":"2023-11-28T17:11:23.220655+0000","last_active":"2023-11-28T17:11:24.211250+0000","last_peered":"2023-11-28T17:11:24.211250+0000","last_clean":"2023-11-28T17:11:24.211250+0000","last_became_active":"2023-11-28T17:11:23.219314+0000","last_became_peered":"2023-11-28T17:11:23.219314+0000","last_unstale":"2023-11-28T17:11:24.211250+0000","last_undegraded":"2023-11-28T17:11:24.211250+0000","last_fullsized":"2023-11-28T17:11:24.211250+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:13:36.324975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223073+0000","last_change":"2023-11-28T17:11:24.214712+0000","last_active":"2023-11-28T17:11:24.223073+0000","last_peered":"2023-11-28T17:11:24.223073+0000","last_clean":"2023-11-28T17:11:24.223073+0000","last_became_active":"2023-11-28T17:11:24.214461+0000","last_became_peered":"2023-11-28T17:11:24.214461+0000","last_unstale":"2023-11-28T17:11:24.223073+0000","last_undegraded":"2023-11-28T17:11:24.223073+0000","last_fullsized":"2023-11-28T17:11:24.223073+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:13:37.338728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"47'8","reported_seq":19,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248490+0000","last_change":"2023-11-28T17:11:23.220889+0000","last_active":"2023-11-28T17:11:24.248490+0000","last_peered":"2023-11-28T17:11:24.248490+0000","last_clean":"2023-11-28T17:11:24.248490+0000","last_became_active":"2023-11-28T17:11:23.219444+0000","last_became_peered":"2023-11-28T17:11:23.219444+0000","last_unstale":"2023-11-28T17:11:24.248490+0000","last_undegraded":"2023-11-28T17:11:24.248490+0000","last_fullsized":"2023-11-28T17:11:24.248490+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:53:16.122203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.219766+0000","last_change":"2023-11-28T17:11:24.218149+0000","last_active":"2023-11-28T17:11:24.219766+0000","last_peered":"2023-11-28T17:11:24.219766+0000","last_clean":"2023-11-28T17:11:24.219766+0000","last_became_active":"2023-11-28T17:11:24.217819+0000","last_became_peered":"2023-11-28T17:11:24.217819+0000","last_unstale":"2023-11-28T17:11:24.219766+0000","last_undegraded":"2023-11-28T17:11:24.219766+0000","last_fullsized":"2023-11-28T17:11:24.219766+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:07.298149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.f","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248729+0000","last_change":"2023-11-28T17:11:23.223512+0000","last_active":"2023-11-28T17:11:24.248729+0000","last_peered":"2023-11-28T17:11:24.248729+0000","last_clean":"2023-11-28T17:11:24.248729+0000","last_became_active":"2023-11-28T17:11:23.222568+0000","last_became_peered":"2023-11-28T17:11:23.222568+0000","last_unstale":"2023-11-28T17:11:24.248729+0000","last_undegraded":"2023-11-28T17:11:24.248729+0000","last_fullsized":"2023-11-28T17:11:24.248729+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:16:54.065477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.239889+0000","last_change":"2023-11-28T17:11:24.234785+0000","last_active":"2023-11-28T17:11:24.239889+0000","last_peered":"2023-11-28T17:11:24.239889+0000","last_clean":"2023-11-28T17:11:24.239889+0000","last_became_active":"2023-11-28T17:11:24.234447+0000","last_became_peered":"2023-11-28T17:11:24.234447+0000","last_unstale":"2023-11-28T17:11:24.239889+0000","last_undegraded":"2023-11-28T17:11:24.239889+0000","last_fullsized":"2023-11-28T17:11:24.239889+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:12:04.130082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.245405+0000","last_change":"2023-11-28T17:11:23.219015+0000","last_active":"2023-11-28T17:11:24.245405+0000","last_peered":"2023-11-28T17:11:24.245405+0000","last_clean":"2023-11-28T17:11:24.245405+0000","last_became_active":"2023-11-28T17:11:23.218733+0000","last_became_peered":"2023-11-28T17:11:23.218733+0000","last_unstale":"2023-11-28T17:11:24.245405+0000","last_undegraded":"2023-11-28T17:11:24.245405+0000","last_fullsized":"2023-11-28T17:11:24.245405+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:55.929651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248232+0000","last_change":"2023-11-28T17:11:24.246437+0000","last_active":"2023-11-28T17:11:24.248232+0000","last_peered":"2023-11-28T17:11:24.248232+0000","last_clean":"2023-11-28T17:11:24.248232+0000","last_became_active":"2023-11-28T17:11:24.246167+0000","last_became_peered":"2023-11-28T17:11:24.246167+0000","last_unstale":"2023-11-28T17:11:24.248232+0000","last_undegraded":"2023-11-28T17:11:24.248232+0000","last_fullsized":"2023-11-28T17:11:24.248232+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:30:34.918467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242551+0000","last_change":"2023-11-28T17:11:23.214992+0000","last_active":"2023-11-28T17:11:24.242551+0000","last_peered":"2023-11-28T17:11:24.242551+0000","last_clean":"2023-11-28T17:11:24.242551+0000","last_became_active":"2023-11-28T17:11:23.214694+0000","last_became_peered":"2023-11-28T17:11:23.214694+0000","last_unstale":"2023-11-28T17:11:24.242551+0000","last_undegraded":"2023-11-28T17:11:24.242551+0000","last_fullsized":"2023-11-28T17:11:24.242551+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:34:06.284396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242213+0000","last_change":"2023-11-28T17:11:24.238610+0000","last_active":"2023-11-28T17:11:24.242213+0000","last_peered":"2023-11-28T17:11:24.242213+0000","last_clean":"2023-11-28T17:11:24.242213+0000","last_became_active":"2023-11-28T17:11:24.238168+0000","last_became_peered":"2023-11-28T17:11:24.238168+0000","last_unstale":"2023-11-28T17:11:24.242213+0000","last_undegraded":"2023-11-28T17:11:24.242213+0000","last_fullsized":"2023-11-28T17:11:24.242213+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:27:31.482201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.209087+0000","last_change":"2023-11-28T17:11:23.218564+0000","last_active":"2023-11-28T17:11:24.209087+0000","last_peered":"2023-11-28T17:11:24.209087+0000","last_clean":"2023-11-28T17:11:24.209087+0000","last_became_active":"2023-11-28T17:11:23.218279+0000","last_became_peered":"2023-11-28T17:11:23.218279+0000","last_unstale":"2023-11-28T17:11:24.209087+0000","last_undegraded":"2023-11-28T17:11:24.209087+0000","last_fullsized":"2023-11-28T17:11:24.209087+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:11:04.436680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242213+0000","last_change":"2023-11-28T17:11:24.238576+0000","last_active":"2023-11-28T17:11:24.242213+0000","last_peered":"2023-11-28T17:11:24.242213+0000","last_clean":"2023-11-28T17:11:24.242213+0000","last_became_active":"2023-11-28T17:11:24.237274+0000","last_became_peered":"2023-11-28T17:11:24.237274+0000","last_unstale":"2023-11-28T17:11:24.242213+0000","last_undegraded":"2023-11-28T17:11:24.242213+0000","last_fullsized":"2023-11-28T17:11:24.242213+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:05:25.971021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.238520+0000","last_change":"2023-11-28T17:11:23.214890+0000","last_active":"2023-11-28T17:11:24.238520+0000","last_peered":"2023-11-28T17:11:24.238520+0000","last_clean":"2023-11-28T17:11:24.238520+0000","last_became_active":"2023-11-28T17:11:23.213727+0000","last_became_peered":"2023-11-28T17:11:23.213727+0000","last_unstale":"2023-11-28T17:11:24.238520+0000","last_undegraded":"2023-11-28T17:11:24.238520+0000","last_fullsized":"2023-11-28T17:11:24.238520+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:05:07.784341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.231630+0000","last_change":"2023-11-28T17:11:24.226847+0000","last_active":"2023-11-28T17:11:24.231630+0000","last_peered":"2023-11-28T17:11:24.231630+0000","last_clean":"2023-11-28T17:11:24.231630+0000","last_became_active":"2023-11-28T17:11:24.226493+0000","last_became_peered":"2023-11-28T17:11:24.226493+0000","last_unstale":"2023-11-28T17:11:24.231630+0000","last_undegraded":"2023-11-28T17:11:24.231630+0000","last_fullsized":"2023-11-28T17:11:24.231630+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:06:32.767821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.208445+0000","last_change":"2023-11-28T17:11:23.218827+0000","last_active":"2023-11-28T17:11:24.208445+0000","last_peered":"2023-11-28T17:11:24.208445+0000","last_clean":"2023-11-28T17:11:24.208445+0000","last_became_active":"2023-11-28T17:11:23.218545+0000","last_became_peered":"2023-11-28T17:11:23.218545+0000","last_unstale":"2023-11-28T17:11:24.208445+0000","last_undegraded":"2023-11-28T17:11:24.208445+0000","last_fullsized":"2023-11-28T17:11:24.208445+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T19:25:29.976839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.248757+0000","last_change":"2023-11-28T17:11:24.243810+0000","last_active":"2023-11-28T17:11:24.248757+0000","last_peered":"2023-11-28T17:11:24.248757+0000","last_clean":"2023-11-28T17:11:24.248757+0000","last_became_active":"2023-11-28T17:11:24.243045+0000","last_became_peered":"2023-11-28T17:11:24.243045+0000","last_unstale":"2023-11-28T17:11:24.248757+0000","last_undegraded":"2023-11-28T17:11:24.248757+0000","last_fullsized":"2023-11-28T17:11:24.248757+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:58:02.404658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.212978+0000","last_change":"2023-11-28T17:11:23.212259+0000","last_active":"2023-11-28T17:11:24.212978+0000","last_peered":"2023-11-28T17:11:24.212978+0000","last_clean":"2023-11-28T17:11:24.212978+0000","last_became_active":"2023-11-28T17:11:23.211898+0000","last_became_peered":"2023-11-28T17:11:23.211898+0000","last_unstale":"2023-11-28T17:11:24.212978+0000","last_undegraded":"2023-11-28T17:11:24.212978+0000","last_fullsized":"2023-11-28T17:11:24.212978+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:13:31.352773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.251318+0000","last_change":"2023-11-28T17:11:24.248225+0000","last_active":"2023-11-28T17:11:24.251318+0000","last_peered":"2023-11-28T17:11:24.251318+0000","last_clean":"2023-11-28T17:11:24.251318+0000","last_became_active":"2023-11-28T17:11:24.247968+0000","last_became_peered":"2023-11-28T17:11:24.247968+0000","last_unstale":"2023-11-28T17:11:24.251318+0000","last_undegraded":"2023-11-28T17:11:24.251318+0000","last_fullsized":"2023-11-28T17:11:24.251318+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T02:25:00.298985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.206477+0000","last_change":"2023-11-28T17:11:23.221517+0000","last_active":"2023-11-28T17:11:24.206477+0000","last_peered":"2023-11-28T17:11:24.206477+0000","last_clean":"2023-11-28T17:11:24.206477+0000","last_became_active":"2023-11-28T17:11:23.221239+0000","last_became_peered":"2023-11-28T17:11:23.221239+0000","last_unstale":"2023-11-28T17:11:24.206477+0000","last_undegraded":"2023-11-28T17:11:24.206477+0000","last_fullsized":"2023-11-28T17:11:24.206477+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:16:44.532346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.233453+0000","last_change":"2023-11-28T17:11:24.228992+0000","last_active":"2023-11-28T17:11:24.233453+0000","last_peered":"2023-11-28T17:11:24.233453+0000","last_clean":"2023-11-28T17:11:24.233453+0000","last_became_active":"2023-11-28T17:11:24.228258+0000","last_became_peered":"2023-11-28T17:11:24.228258+0000","last_unstale":"2023-11-28T17:11:24.233453+0000","last_undegraded":"2023-11-28T17:11:24.233453+0000","last_fullsized":"2023-11-28T17:11:24.233453+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:55:57.542662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.228420+0000","last_change":"2023-11-28T17:11:23.225083+0000","last_active":"2023-11-28T17:11:24.228420+0000","last_peered":"2023-11-28T17:11:24.228420+0000","last_clean":"2023-11-28T17:11:24.228420+0000","last_became_active":"2023-11-28T17:11:23.224633+0000","last_became_peered":"2023-11-28T17:11:23.224633+0000","last_unstale":"2023-11-28T17:11:24.228420+0000","last_undegraded":"2023-11-28T17:11:24.228420+0000","last_fullsized":"2023-11-28T17:11:24.228420+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T23:06:57.485602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242045+0000","last_change":"2023-11-28T17:11:24.235923+0000","last_active":"2023-11-28T17:11:24.242045+0000","last_peered":"2023-11-28T17:11:24.242045+0000","last_clean":"2023-11-28T17:11:24.242045+0000","last_became_active":"2023-11-28T17:11:24.235637+0000","last_became_peered":"2023-11-28T17:11:24.235637+0000","last_unstale":"2023-11-28T17:11:24.242045+0000","last_undegraded":"2023-11-28T17:11:24.242045+0000","last_fullsized":"2023-11-28T17:11:24.242045+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T17:39:21.268933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.207641+0000","last_change":"2023-11-28T17:11:23.223135+0000","last_active":"2023-11-28T17:11:24.207641+0000","last_peered":"2023-11-28T17:11:24.207641+0000","last_clean":"2023-11-28T17:11:24.207641+0000","last_became_active":"2023-11-28T17:11:23.222821+0000","last_became_peered":"2023-11-28T17:11:23.222821+0000","last_unstale":"2023-11-28T17:11:24.207641+0000","last_undegraded":"2023-11-28T17:11:24.207641+0000","last_fullsized":"2023-11-28T17:11:24.207641+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:06:02.961760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.254768+0000","last_change":"2023-11-28T17:11:24.249637+0000","last_active":"2023-11-28T17:11:24.254768+0000","last_peered":"2023-11-28T17:11:24.254768+0000","last_clean":"2023-11-28T17:11:24.254768+0000","last_became_active":"2023-11-28T17:11:24.249321+0000","last_became_peered":"2023-11-28T17:11:24.249321+0000","last_unstale":"2023-11-28T17:11:24.254768+0000","last_undegraded":"2023-11-28T17:11:24.254768+0000","last_fullsized":"2023-11-28T17:11:24.254768+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T03:56:28.171534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.204003+0000","last_change":"2023-11-28T17:11:23.226421+0000","last_active":"2023-11-28T17:11:24.204003+0000","last_peered":"2023-11-28T17:11:24.204003+0000","last_clean":"2023-11-28T17:11:24.204003+0000","last_became_active":"2023-11-28T17:11:23.225639+0000","last_became_peered":"2023-11-28T17:11:23.225639+0000","last_unstale":"2023-11-28T17:11:24.204003+0000","last_undegraded":"2023-11-28T17:11:24.204003+0000","last_fullsized":"2023-11-28T17:11:24.204003+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T01:27:36.624262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.242268+0000","last_change":"2023-11-28T17:11:24.238875+0000","last_active":"2023-11-28T17:11:24.242268+0000","last_peered":"2023-11-28T17:11:24.242268+0000","last_clean":"2023-11-28T17:11:24.242268+0000","last_became_active":"2023-11-28T17:11:24.237847+0000","last_became_peered":"2023-11-28T17:11:24.237847+0000","last_unstale":"2023-11-28T17:11:24.242268+0000","last_undegraded":"2023-11-28T17:11:24.242268+0000","last_fullsized":"2023-11-28T17:11:24.242268+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:47:10.497869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.207376+0000","last_change":"2023-11-28T17:11:23.223183+0000","last_active":"2023-11-28T17:11:24.207376+0000","last_peered":"2023-11-28T17:11:24.207376+0000","last_clean":"2023-11-28T17:11:24.207376+0000","last_became_active":"2023-11-28T17:11:23.222858+0000","last_became_peered":"2023-11-28T17:11:23.222858+0000","last_unstale":"2023-11-28T17:11:24.207376+0000","last_undegraded":"2023-11-28T17:11:24.207376+0000","last_fullsized":"2023-11-28T17:11:24.207376+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:13:36.324975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.249828+0000","last_change":"2023-11-28T17:11:24.236150+0000","last_active":"2023-11-28T17:11:24.249828+0000","last_peered":"2023-11-28T17:11:24.249828+0000","last_clean":"2023-11-28T17:11:24.249828+0000","last_became_active":"2023-11-28T17:11:24.235383+0000","last_became_peered":"2023-11-28T17:11:24.235383+0000","last_unstale":"2023-11-28T17:11:24.249828+0000","last_undegraded":"2023-11-28T17:11:24.249828+0000","last_fullsized":"2023-11-28T17:11:24.249828+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T00:09:26.820366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.207063+0000","last_change":"2023-11-28T17:11:23.223551+0000","last_active":"2023-11-28T17:11:24.207063+0000","last_peered":"2023-11-28T17:11:24.207063+0000","last_clean":"2023-11-28T17:11:24.207063+0000","last_became_active":"2023-11-28T17:11:23.223272+0000","last_became_peered":"2023-11-28T17:11:23.223272+0000","last_unstale":"2023-11-28T17:11:24.207063+0000","last_undegraded":"2023-11-28T17:11:24.207063+0000","last_fullsized":"2023-11-28T17:11:24.207063+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T21:07:41.781121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.245466+0000","last_change":"2023-11-28T17:11:24.240208+0000","last_active":"2023-11-28T17:11:24.245466+0000","last_peered":"2023-11-28T17:11:24.245466+0000","last_clean":"2023-11-28T17:11:24.245466+0000","last_became_active":"2023-11-28T17:11:24.239742+0000","last_became_peered":"2023-11-28T17:11:24.239742+0000","last_unstale":"2023-11-28T17:11:24.245466+0000","last_undegraded":"2023-11-28T17:11:24.245466+0000","last_fullsized":"2023-11-28T17:11:24.245466+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:02:47.263936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.211573+0000","last_change":"2023-11-28T17:11:23.219595+0000","last_active":"2023-11-28T17:11:24.211573+0000","last_peered":"2023-11-28T17:11:24.211573+0000","last_clean":"2023-11-28T17:11:24.211573+0000","last_became_active":"2023-11-28T17:11:23.218954+0000","last_became_peered":"2023-11-28T17:11:23.218954+0000","last_unstale":"2023-11-28T17:11:24.211573+0000","last_undegraded":"2023-11-28T17:11:24.211573+0000","last_fullsized":"2023-11-28T17:11:24.211573+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T18:51:24.669526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.223803+0000","last_change":"2023-11-28T17:11:24.216105+0000","last_active":"2023-11-28T17:11:24.223803+0000","last_peered":"2023-11-28T17:11:24.223803+0000","last_clean":"2023-11-28T17:11:24.223803+0000","last_became_active":"2023-11-28T17:11:24.215824+0000","last_became_peered":"2023-11-28T17:11:24.215824+0000","last_unstale":"2023-11-28T17:11:24.223803+0000","last_undegraded":"2023-11-28T17:11:24.223803+0000","last_fullsized":"2023-11-28T17:11:24.223803+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T20:12:49.613982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.202251+0000","last_change":"2023-11-28T17:11:23.225850+0000","last_active":"2023-11-28T17:11:24.202251+0000","last_peered":"2023-11-28T17:11:24.202251+0000","last_clean":"2023-11-28T17:11:24.202251+0000","last_became_active":"2023-11-28T17:11:23.225316+0000","last_became_peered":"2023-11-28T17:11:23.225316+0000","last_unstale":"2023-11-28T17:11:24.202251+0000","last_undegraded":"2023-11-28T17:11:24.202251+0000","last_fullsized":"2023-11-28T17:11:24.202251+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:23:30.996611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.252270+0000","last_change":"2023-11-28T17:11:24.250815+0000","last_active":"2023-11-28T17:11:24.252270+0000","last_peered":"2023-11-28T17:11:24.252270+0000","last_clean":"2023-11-28T17:11:24.252270+0000","last_became_active":"2023-11-28T17:11:24.250568+0000","last_became_peered":"2023-11-28T17:11:24.250568+0000","last_unstale":"2023-11-28T17:11:24.252270+0000","last_undegraded":"2023-11-28T17:11:24.252270+0000","last_fullsized":"2023-11-28T17:11:24.252270+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:23.175298+0000","last_clean_scrub_stamp":"2023-11-28T17:11:23.175298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-29T22:53:17.135956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2023-11-28T17:11:24.249045+0000","last_change":"2023-11-28T17:11:23.226792+0000","last_active":"2023-11-28T17:11:24.249045+0000","last_peered":"2023-11-28T17:11:24.249045+0000","last_clean":"2023-11-28T17:11:24.249045+0000","last_became_active":"2023-11-28T17:11:23.226442+0000","last_became_peered":"2023-11-28T17:11:23.226442+0000","last_unstale":"2023-11-28T17:11:24.249045+0000","last_undegraded":"2023-11-28T17:11:24.249045+0000","last_fullsized":"2023-11-28T17:11:24.249045+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:11:22.161546+0000","last_clean_scrub_stamp":"2023-11-28T17:11:22.161546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T04:59:50.078274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":191,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-11-28T17:11:23.207303+0000","last_change":"2023-11-28T17:08:51.392782+0000","last_active":"2023-11-28T17:11:23.207303+0000","last_peered":"2023-11-28T17:11:23.207303+0000","last_clean":"2023-11-28T17:11:23.207303+0000","last_became_active":"2023-11-28T17:08:51.390415+0000","last_became_peered":"2023-11-28T17:08:51.390415+0000","last_unstale":"2023-11-28T17:11:23.207303+0000","last_undegraded":"2023-11-28T17:11:23.207303+0000","last_fullsized":"2023-11-28T17:11:23.207303+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-28T17:08:49.967682+0000","last_clean_scrub_stamp":"2023-11-28T17:08:49.967682+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-30T05:03:32.638685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":540672,"data_stored":219335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.1100000000000001,"15min":1.087},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.337,"5min":1.151,"15min":1.1200000000000001},"min":{"1min":0.78600000000000003,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.968,"5min":1.968,"15min":1.968},"last":1.294}]},{"osd":1,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.1279999999999999,"15min":1.1339999999999999},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.2969999999999999,"15min":1.296},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.77500000000000002}]},{"osd":2,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.228,"15min":1.2170000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.841,"5min":1.841,"15min":1.841},"last":0.995},{"interface":"front","average":{"1min":1.226,"5min":1.3260000000000001,"15min":1.3420000000000001},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.889,"5min":2.7029999999999998,"15min":2.7029999999999998},"last":1.5309999999999999}]},{"osd":3,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.179,"15min":1.181},"min":{"1min":0.74299999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":1.073},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.423,"15min":1.421},"min":{"1min":0.83299999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"max":{"1min":1.9510000000000001,"5min":2.161,"15min":2.161},"last":1.179}]},{"osd":4,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.335,"15min":1.329},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.821,"5min":2.202,"15min":2.202},"last":1.427},{"interface":"front","average":{"1min":1.327,"5min":1.367,"15min":1.373},"min":{"1min":0.67500000000000004,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.8939999999999999,"5min":2.069,"15min":2.069},"last":1.246}]},{"osd":5,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.4830000000000001,"15min":1.478},"min":{"1min":0.84299999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.161,"5min":2.4420000000000002,"15min":2.4420000000000002},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.552,"5min":1.5680000000000001,"15min":1.571},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":2.1720000000000002,"5min":2.7850000000000001,"15min":2.7850000000000001},"last":1.6599999999999999}]},{"osd":6,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.6970000000000001,"5min":1.7330000000000001,"15min":1.7390000000000001},"min":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"max":{"1min":2.3559999999999999,"5min":3.8010000000000002,"15min":3.8010000000000002},"last":1.7370000000000001},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.605,"15min":1.6200000000000001},"min":{"1min":1.1180000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.0419999999999998,"5min":3.4049999999999998,"15min":3.4049999999999998},"last":1.6220000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":540672,"data_stored":219335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.1779999999999999,"15min":1.2110000000000001},"min":{"1min":0.64600000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6839999999999999,"5min":4.3579999999999997,"15min":4.3579999999999997},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.0780000000000001,"15min":1.0880000000000001},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.48,"5min":1.841,"15min":1.841},"last":1.3600000000000001}]},{"osd":1,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.3759999999999999,"15min":1.411},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.1190000000000002,"5min":4.1580000000000004,"15min":4.1580000000000004},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.238,"5min":1.3779999999999999,"15min":1.401},"min":{"1min":0.872,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.6259999999999999,"5min":4.202,"15min":4.202},"last":1.4890000000000001}]},{"osd":2,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1319999999999999,"15min":1.137},"min":{"1min":0.77800000000000002,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.425,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":1.208},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.4830000000000001,"15min":1.5169999999999999},"min":{"1min":0.872,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.2250000000000001,"5min":4.2750000000000004,"15min":4.2750000000000004},"last":1.254}]},{"osd":3,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2609999999999999,"15min":1.262},"min":{"1min":0.81399999999999995,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.74,"5min":1.74,"15min":1.74},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.395,"15min":1.399},"min":{"1min":0.86699999999999999,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.1789999999999998,"5min":2.9009999999999998,"15min":2.9009999999999998},"last":1.4059999999999999}]},{"osd":4,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.3600000000000001,"15min":1.3620000000000001},"min":{"1min":0.82999999999999996,"5min":0.624,"15min":0.624},"max":{"1min":2.0430000000000001,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.415,"5min":1.4259999999999999,"15min":1.4279999999999999},"min":{"1min":0.97499999999999998,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.0779999999999998,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.3120000000000001}]},{"osd":5,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.528,"15min":1.524},"min":{"1min":1.1759999999999999,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":2.0289999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":1.46},{"interface":"front","average":{"1min":1.548,"5min":1.5940000000000001,"15min":1.6020000000000001},"min":{"1min":1.21,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.3919999999999999,"5min":3.0030000000000001,"15min":3.0030000000000001},"last":1.579}]},{"osd":7,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.508,"15min":1.512},"min":{"1min":1.147,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.982,"5min":2.524,"15min":2.524},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.655,"15min":1.675},"min":{"1min":1.2150000000000001,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.97,"5min":2.4369999999999998,"15min":2.4369999999999998},"last":1.5189999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1134592,"data_stored":809703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":1.0229999999999999,"15min":1.0329999999999999},"min":{"1min":0.67200000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.673,"5min":1.673,"15min":1.673},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":1.206}]},{"osd":1,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.2030000000000001,"15min":1.2070000000000001},"min":{"1min":0.79700000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.1080000000000001,"15min":1.0800000000000001},"min":{"1min":0.86399999999999999,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.008}]},{"osd":2,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.2370000000000001,"15min":1.232},"min":{"1min":0.80600000000000005,"5min":0.749,"15min":0.749},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.429},{"interface":"front","average":{"1min":1.224,"5min":1.252,"15min":1.256},"min":{"1min":0.78900000000000003,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5429999999999999,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.131}]},{"osd":3,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.2030000000000001,"15min":1.1890000000000001},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.7989999999999999,"5min":2.0910000000000002,"15min":2.0910000000000002},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.365,"5min":1.3859999999999999,"15min":1.3899999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":2.0960000000000001,"5min":2.4180000000000001,"15min":2.4180000000000001},"last":1.266}]},{"osd":4,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.417,"15min":1.421},"min":{"1min":0.82999999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.8440000000000001,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.335},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.3460000000000001,"15min":1.323},"min":{"1min":0.73399999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":1.385}]},{"osd":6,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.4430000000000001,"15min":1.4239999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":2.2839999999999998,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":1.554},{"interface":"front","average":{"1min":1.518,"5min":1.5349999999999999,"15min":1.538},"min":{"1min":0.874,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.964,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.47}]},{"osd":7,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.478,"15min":1.462},"min":{"1min":1.002,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":1.512},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.5580000000000001,"15min":1.546},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.4369999999999998,"5min":2.46,"15min":2.46},"last":1.627}]}]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":540672,"data_stored":219335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":1.02,"15min":1.038},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.3680000000000001,"5min":1.673,"15min":1.673},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.1779999999999999,"15min":1.179},"min":{"1min":0.72799999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.581,"5min":1.837,"15min":1.837},"last":1.397}]},{"osd":1,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.141,"15min":1.1559999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.097,"5min":1.244,"15min":1.268},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.5049999999999999,"5min":1.71,"15min":1.71},"last":0.76000000000000001}]},{"osd":2,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.169,"15min":1.206},"min":{"1min":0.71699999999999997,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.252,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.337,"15min":1.353},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"max":{"1min":1.73,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":1.5089999999999999}]},{"osd":3,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.3100000000000001,"15min":1.3340000000000001},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.1549999999999998,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.377,"5min":1.405,"15min":1.4099999999999999},"min":{"1min":1.03,"5min":0.87,"15min":0.87},"max":{"1min":1.6439999999999999,"5min":2.048,"15min":2.048},"last":1.615}]},{"osd":5,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.425,"15min":1.4590000000000001},"min":{"1min":0.81499999999999995,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.7410000000000001,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.4490000000000001,"15min":1.4810000000000001},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.9870000000000001,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":1.0589999999999999}]},{"osd":6,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.6299999999999999,"15min":1.647},"min":{"1min":1.26,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.8280000000000001,"5min":2.1520000000000001,"15min":2.1520000000000001},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.363,"5min":1.619,"15min":1.6619999999999999},"min":{"1min":1.0269999999999999,"5min":0.749,"15min":0.749},"max":{"1min":1.867,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":1.4490000000000001}]},{"osd":7,"last update":"Tue Nov 28 17:10:54 2023","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.6339999999999999,"15min":1.673},"min":{"1min":0.80000000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.2879999999999998,"5min":2.669,"15min":2.669},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.6599999999999999,"15min":1.6679999999999999},"min":{"1min":1.248,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.927,"5min":2.387,"15min":2.387},"last":1.806}]}]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27552,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":540672,"data_stored":219335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.95499999999999996,"15min":0.98099999999999998},"min":{"1min":0.34699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.5569999999999999,"5min":2.2650000000000001,"15min":2.2650000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.98499999999999999,"15min":0.997},"min":{"1min":0.48399999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.4370000000000001,"5min":2.444,"15min":2.444},"last":0.65700000000000003}]},{"osd":1,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":1.024,"15min":1.0569999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.325,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":0.621},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.1100000000000001,"15min":1.131},"min":{"1min":0.55600000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4810000000000001,"5min":2.3900000000000001,"15min":2.3900000000000001},"last":0.55100000000000005}]},{"osd":2,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.1779999999999999,"15min":1.194},"min":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.587,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.046,"15min":1.0640000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.3540000000000001,"5min":2.319,"15min":2.319},"last":0.72099999999999997}]},{"osd":4,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.268,"15min":1.284},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.744,"5min":2.5699999999999998,"15min":2.5699999999999998},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.355,"15min":1.375},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.7110000000000001,"5min":2.5059999999999998,"15min":2.5059999999999998},"last":1.105}]},{"osd":5,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.4179999999999999,"15min":1.446},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":1.5580000000000001,"5min":2.6280000000000001,"15min":2.6280000000000001},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.4950000000000001,"15min":1.524},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.706,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.73999999999999999}]},{"osd":6,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.365,"15min":1.3759999999999999},"min":{"1min":0.73199999999999998,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7869999999999999,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.335,"5min":1.5449999999999999,"15min":1.5800000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.825,"5min":2.6000000000000001,"15min":2.6000000000000001},"last":0.82099999999999995}]},{"osd":7,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.5509999999999999,"15min":1.577},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.79,"5min":2.508,"15min":2.508},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.4770000000000001,"15min":1.4890000000000001},"min":{"1min":0.871,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.8580000000000001,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":0.83999999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27552,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":540672,"data_stored":219335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.84799999999999998,"15min":0.86199999999999999},"min":{"1min":0.45700000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.448,"5min":2.6560000000000001,"15min":2.6560000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.79300000000000004,"15min":0.79600000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.65200000000000002}]},{"osd":1,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.88800000000000001,"15min":0.89600000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":1.397},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0569999999999999,"15min":1.0720000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.609,"5min":2.161,"15min":2.161},"last":1.272}]},{"osd":3,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.93300000000000005,"15min":0.94199999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.4119999999999999,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.1459999999999999,"15min":1.1679999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.4350000000000001,"5min":2.5590000000000002,"15min":2.5590000000000002},"last":1.1200000000000001}]},{"osd":4,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.23,"15min":1.2509999999999999},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.587,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.236,"15min":1.2470000000000001},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.73,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.0649999999999999}]},{"osd":5,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.246,"15min":1.26},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.5489999999999999,"5min":2.048,"15min":2.048},"last":1.169},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.258,"15min":1.2869999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.323,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.5589999999999999}]},{"osd":6,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.319,"15min":1.351},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.6060000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.629},{"interface":"front","average":{"1min":1.252,"5min":1.3360000000000001,"15min":1.3500000000000001},"min":{"1min":0.90900000000000003,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.5389999999999999,"5min":2.7429999999999999,"15min":2.7429999999999999},"last":1.2410000000000001}]},{"osd":7,"last update":"Tue Nov 28 17:10:51 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.4039999999999999,"15min":1.4239999999999999},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.6879999999999999,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.4299999999999999},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.367,"15min":1.397},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"max":{"1min":1.5509999999999999,"5min":1.9530000000000001,"15min":1.9530000000000001},"last":1.6759999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28132,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343708,"statfs":{"total":96636764160,"available":96607956992,"internally_reserved":0,"allocated":1134592,"data_stored":809703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":1,"15min":1.038},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.3759999999999999,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.131,"15min":1.1819999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.333,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":1.2130000000000001}]},{"osd":2,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.95299999999999996,"15min":0.97099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.395,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":1.224,"15min":1.274},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.3720000000000001,"5min":4.0090000000000003,"15min":4.0090000000000003},"last":0.90600000000000003}]},{"osd":3,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.3009999999999999,"15min":1.343},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.5369999999999999,"5min":3.9569999999999999,"15min":3.9569999999999999},"last":1.139},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.2030000000000001,"15min":1.2589999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.4610000000000001,"5min":2.6389999999999998,"15min":2.6389999999999998},"last":1.0780000000000001}]},{"osd":4,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.2290000000000001,"15min":1.2450000000000001},"min":{"1min":0.77100000000000002,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.8839999999999999,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":1.268},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.595,"15min":1.649},"min":{"1min":0.81899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6439999999999999,"5min":4.29,"15min":4.29},"last":1.3149999999999999}]},{"osd":5,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.607,"15min":1.665},"min":{"1min":0.90900000000000003,"5min":0.874,"15min":0.874},"max":{"1min":1.7549999999999999,"5min":4.1589999999999998,"15min":4.1589999999999998},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.252,"5min":1.5309999999999999,"15min":1.577},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":2.2040000000000002,"5min":4.2460000000000004,"15min":4.2460000000000004},"last":1.411}]},{"osd":6,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.6839999999999999,"15min":1.748},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":1.9690000000000001,"5min":4.3419999999999996,"15min":4.3419999999999996},"last":1.5},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.659,"15min":1.73},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"max":{"1min":1.601,"5min":4.1239999999999997,"15min":4.1239999999999997},"last":1.875}]},{"osd":7,"last update":"Tue Nov 28 17:10:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.706,"15min":1.7709999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.0339999999999998,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":1.8080000000000001},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.486,"15min":1.516},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.0950000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":1.5880000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27552,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":540672,"data_stored":219335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Nov 28 17:10:55 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.80700000000000005,"15min":0.82799999999999996},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.006,"5min":1.45,"15min":1.45},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.86199999999999999,"15min":0.874},"min":{"1min":0.40600000000000003,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.1439999999999999,"5min":1.294,"15min":1.294},"last":0.82299999999999995}]},{"osd":2,"last update":"Tue Nov 28 17:10:55 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.97999999999999998,"15min":1.002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.3600000000000001,"5min":1.601,"15min":1.601},"last":1.2949999999999999},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.93500000000000005,"15min":0.95199999999999996},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.4610000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.377}]},{"osd":3,"last update":"Tue Nov 28 17:10:55 2023","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.143,"15min":1.1679999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.4430000000000001,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":1.1739999999999999},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.0900000000000001,"15min":1.1279999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.4870000000000001,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.484}]},{"osd":4,"last update":"Tue Nov 28 17:10:55 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.2749999999999999,"15min":1.2989999999999999},"min":{"1min":0.78400000000000003,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.673,"5min":1.764,"15min":1.764},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.4059999999999999,"15min":1.444},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6279999999999999,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.6379999999999999}]},{"osd":5,"last update":"Tue Nov 28 17:10:55 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.407,"15min":1.4350000000000001},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.9690000000000001,"5min":2.2570000000000001,"15min":2.2570000000000001},"last":1.444},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.276,"15min":1.286},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6659999999999999,"5min":1.855,"15min":1.855},"last":1.2350000000000001}]},{"osd":6,"last update":"Tue Nov 28 17:10:55 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.3799999999999999,"15min":1.4079999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.5920000000000001,"5min":2.1360000000000001,"15min":2.1360000000000001},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.452,"15min":1.476},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":1.762,"5min":2.3069999999999999,"15min":2.3069999999999999},"last":1.556}]},{"osd":7,"last update":"Tue Nov 28 17:10:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.377,"15min":1.3899999999999999},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.7330000000000001,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.339},{"interface":"front","average":{"1min":1.292,"5min":1.411,"15min":1.431},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.7709999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.774}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-28T17:11:29.130 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-28T17:11:29.130 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-28T17:11:29.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.390+0000 7f368cd68640 1 Processor -- start 2023-11-28T17:11:29.392 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.391+0000 7f368cd68640 1 -- start start 2023-11-28T17:11:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.392+0000 7f368cd68640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f36881438e0 0x7f36880ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.393+0000 7f368cd68640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36880acf80 0x7f36880b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.393+0000 7f368cd68640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3688071590 con 0x7f3688071a20 2023-11-28T17:11:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.393+0000 7f368cd68640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3688070980 con 0x7f36881438e0 2023-11-28T17:11:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.393+0000 7f368cd68640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3688070c60 con 0x7f36880acf80 2023-11-28T17:11:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.393+0000 7f3685d9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3688071a20 0x7f36880ac870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45242/0 (socket says 172.21.15.116:45242) 2023-11-28T17:11:29.395 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.393+0000 7f3685d9b640 1 -- 172.21.15.116:0/1147825645 learned_addr learned my addr 172.21.15.116:0/1147825645 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:29.396 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.393+0000 7f368659c640 1 --2- 172.21.15.116:0/1147825645 >> v2:172.21.15.156:3300/0 conn(0x7f36881438e0 0x7f36880ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:29.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.393+0000 7f3686d9d640 1 --2- 172.21.15.116:0/1147825645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36880acf80 0x7f36880b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:29.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.393+0000 7f368559a640 1 -- 172.21.15.116:0/1147825645 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060833859 0 0) 0x7f3688071590 con 0x7f3688071a20 2023-11-28T17:11:29.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 -- 172.21.15.116:0/1147825645 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3674003520 con 0x7f3688071a20 2023-11-28T17:11:29.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 -- 172.21.15.116:0/1147825645 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3503151146 0 0) 0x7f3674003520 con 0x7f3688071a20 2023-11-28T17:11:29.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 -- 172.21.15.116:0/1147825645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36880acf80 msgr2=0x7f36880b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:29.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 --2- 172.21.15.116:0/1147825645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36880acf80 0x7f36880b3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 -- 172.21.15.116:0/1147825645 >> v2:172.21.15.156:3300/0 conn(0x7f36881438e0 msgr2=0x7f36880ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:29.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 --2- 172.21.15.116:0/1147825645 >> v2:172.21.15.156:3300/0 conn(0x7f36881438e0 0x7f36880ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 -- 172.21.15.116:0/1147825645 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3688070f40 con 0x7f3688071a20 2023-11-28T17:11:29.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 -- 172.21.15.116:0/1147825645 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f36700019f0 con 0x7f3688071a20 2023-11-28T17:11:29.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 -- 172.21.15.116:0/1147825645 >> v1:172.21.15.116:6789/0 conn(0x7f3688071a20 legacy=0x7f36880ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:29.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 --2- 172.21.15.116:0/1147825645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36880acf80 0x7f3674003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:29.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 --2- 172.21.15.116:0/1147825645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3674003f40 0x7f36740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:29.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 --2- 172.21.15.116:0/1147825645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674004d00 0x7f3674005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:29.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 -- 172.21.15.116:0/1147825645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3688070980 con 0x7f3674004d00 2023-11-28T17:11:29.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 -- 172.21.15.116:0/1147825645 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3688070c60 con 0x7f3674003f40 2023-11-28T17:11:29.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f368559a640 1 -- 172.21.15.116:0/1147825645 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367400b980 con 0x7f36880acf80 2023-11-28T17:11:29.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.394+0000 7f3686d9d640 1 --2- 172.21.15.116:0/1147825645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674004d00 0x7f3674005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:29.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.395+0000 7f368659c640 1 --2- 172.21.15.116:0/1147825645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36880acf80 0x7f3674003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:29.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.395+0000 7f3686d9d640 1 -- 172.21.15.116:0/1147825645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36880acf80 msgr2=0x7f3674003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:29.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.395+0000 7f3686d9d640 1 --2- 172.21.15.116:0/1147825645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36880acf80 0x7f3674003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.395+0000 7f3686d9d640 1 -- 172.21.15.116:0/1147825645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3674003f40 msgr2=0x7f36740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:29.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.395+0000 7f3686d9d640 1 --2- 172.21.15.116:0/1147825645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3674003f40 0x7f36740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.398 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.395+0000 7f3686d9d640 1 -- 172.21.15.116:0/1147825645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3674003520 con 0x7f3674004d00 2023-11-28T17:11:29.399 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.395+0000 7f368659c640 1 --2- 172.21.15.116:0/1147825645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36880acf80 0x7f3674003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:29.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.396+0000 7f3686d9d640 1 --2- 172.21.15.116:0/1147825645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674004d00 0x7f3674005180 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3680009af0 tx=0x7f36740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff2f2666e46e454 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:29.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.396+0000 7f368559a640 1 -- 172.21.15.116:0/1147825645 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368002c6b0 con 0x7f3674004d00 2023-11-28T17:11:29.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.396+0000 7f368559a640 1 -- 172.21.15.116:0/1147825645 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368002c810 con 0x7f3674004d00 2023-11-28T17:11:29.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.396+0000 7f368559a640 1 -- 172.21.15.116:0/1147825645 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368002cba0 con 0x7f3674004d00 2023-11-28T17:11:29.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.397+0000 7f368cd68640 1 -- 172.21.15.116:0/1147825645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674004d00 msgr2=0x7f3674005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:29.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.397+0000 7f368cd68640 1 --2- 172.21.15.116:0/1147825645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674004d00 0x7f3674005180 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3680009af0 tx=0x7f36740057f0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.397+0000 7f3686d9d640 1 -- 172.21.15.116:0/1147825645 reap_dead start 2023-11-28T17:11:29.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.397+0000 7f368cd68640 1 -- 172.21.15.116:0/1147825645 shutdown_connections 2023-11-28T17:11:29.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.397+0000 7f368cd68640 1 -- 172.21.15.116:0/1147825645 >> 172.21.15.116:0/1147825645 conn(0x7f36880b0cf0 msgr2=0x7f36880b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:29.400 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.397+0000 7f368cd68640 1 -- 172.21.15.116:0/1147825645 shutdown_connections 2023-11-28T17:11:29.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.397+0000 7f368cd68640 1 -- 172.21.15.116:0/1147825645 wait complete. 2023-11-28T17:11:29.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.399+0000 7f368cd68640 1 Processor -- start 2023-11-28T17:11:29.401 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.399+0000 7f368cd68640 1 -- start start 2023-11-28T17:11:29.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.400+0000 7f368cd68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674004d00 0x7f368813f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:29.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.400+0000 7f368cd68640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36880ce130 0x7f36880ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:29.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.400+0000 7f368cd68640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36880cc360 0x7f36880cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:29.402 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.400+0000 7f368cd68640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368813fe70 con 0x7f3674004d00 2023-11-28T17:11:29.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.400+0000 7f368cd68640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36880ceaf0 con 0x7f36880cc360 2023-11-28T17:11:29.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.400+0000 7f368cd68640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36880cb7b0 con 0x7f36880ce130 2023-11-28T17:11:29.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.400+0000 7f368659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674004d00 0x7f368813f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:29.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.400+0000 7f368659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674004d00 0x7f368813f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33938/0 (socket says 172.21.15.116:33938) 2023-11-28T17:11:29.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.400+0000 7f368659c640 1 -- 172.21.15.116:0/1434309266 learned_addr learned my addr 172.21.15.116:0/1434309266 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:29.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.400+0000 7f368659c640 1 -- 172.21.15.116:0/1434309266 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36880ce130 msgr2=0x7f36880ce5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:29.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.400+0000 7f368659c640 1 --2- 172.21.15.116:0/1434309266 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36880ce130 0x7f36880ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.400+0000 7f368659c640 1 -- 172.21.15.116:0/1434309266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36880cc360 msgr2=0x7f36880cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:29.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.400+0000 7f368659c640 1 --2- 172.21.15.116:0/1434309266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36880cc360 0x7f36880cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.400+0000 7f368659c640 1 -- 172.21.15.116:0/1434309266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36880c73a0 con 0x7f3674004d00 2023-11-28T17:11:29.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.401+0000 7f368659c640 1 --2- 172.21.15.116:0/1434309266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674004d00 0x7f368813f930 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f3678010a00 tx=0x7f3678010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:29.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.401+0000 7f367effd640 1 -- 172.21.15.116:0/1434309266 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367800eba0 con 0x7f3674004d00 2023-11-28T17:11:29.403 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.401+0000 7f367effd640 1 -- 172.21.15.116:0/1434309266 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367800ed00 con 0x7f3674004d00 2023-11-28T17:11:29.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.401+0000 7f368cd68640 1 -- 172.21.15.116:0/1434309266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36880c8800 con 0x7f3674004d00 2023-11-28T17:11:29.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.401+0000 7f367effd640 1 -- 172.21.15.116:0/1434309266 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367801c620 con 0x7f3674004d00 2023-11-28T17:11:29.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.401+0000 7f368cd68640 1 -- 172.21.15.116:0/1434309266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36880c81b0 con 0x7f3674004d00 2023-11-28T17:11:29.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.402+0000 7f368cd68640 1 -- 172.21.15.116:0/1434309266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3688069440 con 0x7f3674004d00 2023-11-28T17:11:29.404 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.403+0000 7f367effd640 1 -- 172.21.15.116:0/1434309266 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f367801c780 con 0x7f3674004d00 2023-11-28T17:11:29.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.404+0000 7f367effd640 1 --2- 172.21.15.116:0/1434309266 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f366806b2d0 0x7f366806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:29.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.404+0000 7f367effd640 1 -- 172.21.15.116:0/1434309266 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f367808a170 con 0x7f3674004d00 2023-11-28T17:11:29.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.404+0000 7f3685d9b640 1 --2- 172.21.15.116:0/1434309266 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f366806b2d0 0x7f366806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:29.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.405+0000 7f3685d9b640 1 --2- 172.21.15.116:0/1434309266 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f366806b2d0 0x7f366806d790 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f36880b3830 tx=0x7f3670001940 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:29.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.409+0000 7f367effd640 1 -- 172.21.15.116:0/1434309266 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3678055080 con 0x7f3674004d00 2023-11-28T17:11:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.674+0000 7f368cd68640 1 -- 172.21.15.116:0/1434309266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f36880ae410 con 0x7f3674004d00 2023-11-28T17:11:29.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.679+0000 7f367effd640 1 -- 172.21.15.116:0/1434309266 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7f36780586a0 con 0x7f3674004d00 2023-11-28T17:11:29.681 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2023-11-28T17:11:29.681 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBtHmZlPRewFxAAfLJPZrTljlPIoghT5ZzoBA== 2023-11-28T17:11:29.681 INFO:teuthology.orchestra.run.smithi116.stdout: caps mds = "allow" 2023-11-28T17:11:29.681 INFO:teuthology.orchestra.run.smithi116.stdout: caps mon = "allow r" 2023-11-28T17:11:29.681 INFO:teuthology.orchestra.run.smithi116.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-28T17:11:29.682 INFO:teuthology.orchestra.run.smithi116.stderr:updated caps for client.0 2023-11-28T17:11:29.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.682+0000 7f368cd68640 1 -- 172.21.15.116:0/1434309266 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f366806b2d0 msgr2=0x7f366806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:29.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.682+0000 7f368cd68640 1 --2- 172.21.15.116:0/1434309266 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f366806b2d0 0x7f366806d790 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f36880b3830 tx=0x7f3670001940 comp rx=0 tx=0).stop 2023-11-28T17:11:29.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.682+0000 7f368cd68640 1 -- 172.21.15.116:0/1434309266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674004d00 msgr2=0x7f368813f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:29.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.682+0000 7f368cd68640 1 --2- 172.21.15.116:0/1434309266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674004d00 0x7f368813f930 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f3678010a00 tx=0x7f3678010ec0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.682+0000 7f368cd68640 1 -- 172.21.15.116:0/1434309266 shutdown_connections 2023-11-28T17:11:29.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.682+0000 7f368cd68640 1 --2- 172.21.15.116:0/1434309266 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f366806b2d0 0x7f366806d790 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.682+0000 7f368cd68640 1 --2- 172.21.15.116:0/1434309266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36880cc360 0x7f36880cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.682+0000 7f368cd68640 1 --2- 172.21.15.116:0/1434309266 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f36880ce130 0x7f36880ce5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.682+0000 7f368cd68640 1 --2- 172.21.15.116:0/1434309266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3674004d00 0x7f368813f930 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.682+0000 7f368cd68640 1 -- 172.21.15.116:0/1434309266 >> 172.21.15.116:0/1434309266 conn(0x7f36880b0cf0 msgr2=0x7f36880b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:29.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.683+0000 7f368cd68640 1 -- 172.21.15.116:0/1434309266 shutdown_connections 2023-11-28T17:11:29.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.683+0000 7f368cd68640 1 -- 172.21.15.116:0/1434309266 wait complete. 2023-11-28T17:11:29.704 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:11:29.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.905+0000 7fb2cfbd5640 1 Processor -- start 2023-11-28T17:11:29.907 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.905+0000 7fb2cfbd5640 1 -- start start 2023-11-28T17:11:29.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.907+0000 7fb2cfbd5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb2c8071710 0x7fb2c806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:29.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.907+0000 7fb2cfbd5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c81405a0 con 0x7fb2c806c790 2023-11-28T17:11:29.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.907+0000 7fb2cfbd5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c806cba0 con 0x7fb2c81401c0 2023-11-28T17:11:29.908 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.907+0000 7fb2cfbd5640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c8140880 con 0x7fb2c8071710 2023-11-28T17:11:29.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.907+0000 7fb2ce172640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb2c806c790 0x7fb2c80b3430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45250/0 (socket says 172.21.15.116:45250) 2023-11-28T17:11:29.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.907+0000 7fb2ce172640 1 -- 172.21.15.116:0/2196362320 learned_addr learned my addr 172.21.15.116:0/2196362320 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:29.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.907+0000 7fb2cd170640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb2c8071710 0x7fb2c806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:29.909 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.908+0000 7fb2cd170640 1 -- 172.21.15.116:0/2196362320 >> v1:172.21.15.156:6789/0 conn(0x7fb2c81401c0 legacy=0x7fb2c8071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:29.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.908+0000 7fb2cd170640 1 -- 172.21.15.116:0/2196362320 >> v1:172.21.15.116:6789/0 conn(0x7fb2c806c790 legacy=0x7fb2c80b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:29.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.908+0000 7fb2cc96f640 1 -- 172.21.15.116:0/2196362320 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829413604 0 0) 0x7fb2c81405a0 con 0x7fb2c806c790 2023-11-28T17:11:29.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.908+0000 7fb2cd170640 1 -- 172.21.15.116:0/2196362320 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c815db00 con 0x7fb2c8071710 2023-11-28T17:11:29.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.909+0000 7fb2cd170640 1 --2- 172.21.15.116:0/2196362320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2c8071710 0x7fb2c806c250 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80097a0 tx=0x7fb2b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fcfdfa8718b38ac server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:29.910 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.909+0000 7fb2cc96f640 1 -- 172.21.15.116:0/2196362320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8031830 con 0x7fb2c8071710 2023-11-28T17:11:29.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.909+0000 7fb2cc96f640 1 -- 172.21.15.116:0/2196362320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b8031990 con 0x7fb2c8071710 2023-11-28T17:11:29.911 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.909+0000 7fb2cc96f640 1 -- 172.21.15.116:0/2196362320 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8031d20 con 0x7fb2c8071710 2023-11-28T17:11:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.910+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/2196362320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2c8071710 msgr2=0x7fb2c806c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.910+0000 7fb2cfbd5640 1 --2- 172.21.15.116:0/2196362320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2c8071710 0x7fb2c806c250 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb2b80097a0 tx=0x7fb2b802b8d0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.910+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/2196362320 shutdown_connections 2023-11-28T17:11:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.910+0000 7fb2cfbd5640 1 --2- 172.21.15.116:0/2196362320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2c8071710 0x7fb2c806c250 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.910+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/2196362320 >> 172.21.15.116:0/2196362320 conn(0x7fb2c806a4c0 msgr2=0x7fb2c806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.910+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/2196362320 shutdown_connections 2023-11-28T17:11:29.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.910+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/2196362320 wait complete. 2023-11-28T17:11:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.912+0000 7fb2cfbd5640 1 Processor -- start 2023-11-28T17:11:29.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.912+0000 7fb2cfbd5640 1 -- start start 2023-11-28T17:11:29.915 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.913+0000 7fb2cfbd5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2c806c790 0x7fb2c813d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:29.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.913+0000 7fb2cfbd5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2c8071710 0x7fb2c8070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:29.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.913+0000 7fb2cfbd5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2c81401c0 0x7fb2c8071010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:29.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.913+0000 7fb2cfbd5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c8158350 con 0x7fb2c806c790 2023-11-28T17:11:29.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.913+0000 7fb2cfbd5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c8158630 con 0x7fb2c81401c0 2023-11-28T17:11:29.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.913+0000 7fb2cfbd5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c813de30 con 0x7fb2c8071710 2023-11-28T17:11:29.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.914+0000 7fb2cd971640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2c806c790 0x7fb2c813d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:29.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.914+0000 7fb2cd971640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2c806c790 0x7fb2c813d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33958/0 (socket says 172.21.15.116:33958) 2023-11-28T17:11:29.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.914+0000 7fb2cd971640 1 -- 172.21.15.116:0/4041899592 learned_addr learned my addr 172.21.15.116:0/4041899592 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:29.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.914+0000 7fb2cd170640 1 --2- 172.21.15.116:0/4041899592 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2c8071710 0x7fb2c8070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:29.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.914+0000 7fb2ce172640 1 --2- 172.21.15.116:0/4041899592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2c81401c0 0x7fb2c8071010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:29.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.914+0000 7fb2cd971640 1 -- 172.21.15.116:0/4041899592 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2c8071710 msgr2=0x7fb2c8070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:29.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.914+0000 7fb2cd971640 1 --2- 172.21.15.116:0/4041899592 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2c8071710 0x7fb2c8070ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.914+0000 7fb2cd971640 1 -- 172.21.15.116:0/4041899592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2c81401c0 msgr2=0x7fb2c8071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:29.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.914+0000 7fb2cd971640 1 --2- 172.21.15.116:0/4041899592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2c81401c0 0x7fb2c8071010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:29.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.914+0000 7fb2cd971640 1 -- 172.21.15.116:0/4041899592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b8009400 con 0x7fb2c806c790 2023-11-28T17:11:29.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.914+0000 7fb2cd971640 1 --2- 172.21.15.116:0/4041899592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2c806c790 0x7fb2c813d8f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400c460 tx=0x7fb2c400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.915+0000 7fb2be7fc640 1 -- 172.21.15.116:0/4041899592 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4012830 con 0x7fb2c806c790 2023-11-28T17:11:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.915+0000 7fb2be7fc640 1 -- 172.21.15.116:0/4041899592 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c40129d0 con 0x7fb2c806c790 2023-11-28T17:11:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.915+0000 7fb2be7fc640 1 -- 172.21.15.116:0/4041899592 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4012d80 con 0x7fb2c806c790 2023-11-28T17:11:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.915+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/4041899592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c815a460 con 0x7fb2c806c790 2023-11-28T17:11:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.915+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/4041899592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c815aa20 con 0x7fb2c806c790 2023-11-28T17:11:29.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.916+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/4041899592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c80693d0 con 0x7fb2c806c790 2023-11-28T17:11:29.919 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.917+0000 7fb2be7fc640 1 -- 172.21.15.116:0/4041899592 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb2c401a3b0 con 0x7fb2c806c790 2023-11-28T17:11:29.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.918+0000 7fb2be7fc640 1 --2- 172.21.15.116:0/4041899592 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb2ac06b2d0 0x7fb2ac06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:29.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.918+0000 7fb2be7fc640 1 -- 172.21.15.116:0/4041899592 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2c4089820 con 0x7fb2c806c790 2023-11-28T17:11:29.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.919+0000 7fb2cd170640 1 --2- 172.21.15.116:0/4041899592 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb2ac06b2d0 0x7fb2ac06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:29.920 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.919+0000 7fb2cd170640 1 --2- 172.21.15.116:0/4041899592 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb2ac06b2d0 0x7fb2ac06d790 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8036000 tx=0x7fb2b8008030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:29.923 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:29.923+0000 7fb2be7fc640 1 -- 172.21.15.116:0/4041899592 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2c4054680 con 0x7fb2c806c790 2023-11-28T17:11:30.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.149+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/4041899592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2c815ac10 con 0x7fb2c806c790 2023-11-28T17:11:30.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.151+0000 7fb2be7fc640 1 -- 172.21.15.116:0/4041899592 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4922 (secure 0 0 0) 0x7fb2c4057ca0 con 0x7fb2c806c790 2023-11-28T17:11:30.152 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:30.152 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4790,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2474839888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2474839888},{"type":"v1","addr":"172.21.15.156:6835","nonce":2474839888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":4987,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/372407661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":372407661},{"type":"v1","addr":"172.21.15.156:6833","nonce":372407661}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3251347229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3251347229},{"type":"v1","addr":"172.21.15.116:6835","nonce":3251347229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:11:23.668388+0000","modified":"2023-11-28T17:11:26.731191+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5826},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5826":{"gid":5826,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6837/1482499383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1482499383},{"type":"v1","addr":"172.21.15.116:6837","nonce":1482499383}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T17:11:30.152 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2023-11-28T17:11:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.153+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/4041899592 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb2ac06b2d0 msgr2=0x7fb2ac06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.153+0000 7fb2cfbd5640 1 --2- 172.21.15.116:0/4041899592 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb2ac06b2d0 0x7fb2ac06d790 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8036000 tx=0x7fb2b8008030 comp rx=0 tx=0).stop 2023-11-28T17:11:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.153+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/4041899592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2c806c790 msgr2=0x7fb2c813d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.153+0000 7fb2cfbd5640 1 --2- 172.21.15.116:0/4041899592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2c806c790 0x7fb2c813d8f0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400c460 tx=0x7fb2c400c920 comp rx=0 tx=0).stop 2023-11-28T17:11:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.153+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/4041899592 shutdown_connections 2023-11-28T17:11:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.153+0000 7fb2cfbd5640 1 --2- 172.21.15.116:0/4041899592 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb2ac06b2d0 0x7fb2ac06d790 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.153+0000 7fb2cfbd5640 1 --2- 172.21.15.116:0/4041899592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2c81401c0 0x7fb2c8071010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.153+0000 7fb2cfbd5640 1 --2- 172.21.15.116:0/4041899592 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2c8071710 0x7fb2c8070ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.153+0000 7fb2cfbd5640 1 --2- 172.21.15.116:0/4041899592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb2c806c790 0x7fb2c813d8f0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.154+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/4041899592 >> 172.21.15.116:0/4041899592 conn(0x7fb2c806a4c0 msgr2=0x7fb2c815d290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:30.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.154+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/4041899592 shutdown_connections 2023-11-28T17:11:30.156 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.154+0000 7fb2cfbd5640 1 -- 172.21.15.116:0/4041899592 wait complete. 2023-11-28T17:11:30.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T17:11:23.668388+0000', 'modified': '2023-11-28T17:11:26.731191+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5826}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5826': {'gid': 5826, 'name': 'c', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.116:6837/1482499383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6836', 'nonce': 1482499383}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 1482499383}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T17:11:30.577 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-28T17:11:30.577 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T17:11:30.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.795+0000 7f38413b0640 1 Processor -- start 2023-11-28T17:11:30.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.795+0000 7f38413b0640 1 -- start start 2023-11-28T17:11:30.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.797+0000 7f38413b0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f383c06c250 0x7f383c06c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.797+0000 7f38413b0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c15fc10 con 0x7f383c15f7b0 2023-11-28T17:11:30.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.797+0000 7f38413b0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c070bd0 con 0x7f383c144430 2023-11-28T17:11:30.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.797+0000 7f38413b0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383c144810 con 0x7f383c06c250 2023-11-28T17:11:30.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.797+0000 7f38327fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f383c15f7b0 0x7f383c071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45266/0 (socket says 172.21.15.116:45266) 2023-11-28T17:11:30.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.797+0000 7f38327fc640 1 -- 172.21.15.116:0/2223581617 learned_addr learned my addr 172.21.15.116:0/2223581617 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:30.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383b7fe640 1 --2- 172.21.15.116:0/2223581617 >> v2:172.21.15.156:3300/0 conn(0x7f383c06c250 0x7f383c06c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 -- 172.21.15.116:0/2223581617 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521397996 0 0) 0x7f383c15fc10 con 0x7f383c15f7b0 2023-11-28T17:11:30.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 -- 172.21.15.116:0/2223581617 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f381c003520 con 0x7f383c15f7b0 2023-11-28T17:11:30.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 -- 172.21.15.116:0/2223581617 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3062140720 0 0) 0x7f381c003520 con 0x7f383c15f7b0 2023-11-28T17:11:30.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 -- 172.21.15.116:0/2223581617 >> v2:172.21.15.156:3300/0 conn(0x7f383c06c250 msgr2=0x7f383c06c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:30.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 --2- 172.21.15.116:0/2223581617 >> v2:172.21.15.156:3300/0 conn(0x7f383c06c250 0x7f383c06c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 -- 172.21.15.116:0/2223581617 >> v1:172.21.15.156:6789/0 conn(0x7f383c144430 legacy=0x7f383c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:30.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 -- 172.21.15.116:0/2223581617 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c070eb0 con 0x7f383c15f7b0 2023-11-28T17:11:30.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 -- 172.21.15.116:0/2223581617 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f3824001670 con 0x7f383c15f7b0 2023-11-28T17:11:30.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 -- 172.21.15.116:0/2223581617 >> v1:172.21.15.116:6789/0 conn(0x7f383c15f7b0 legacy=0x7f383c071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:30.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 --2- 172.21.15.116:0/2223581617 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f381c003970 0x7f381c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 --2- 172.21.15.116:0/2223581617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f381c0047c0 0x7f381c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 --2- 172.21.15.116:0/2223581617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f381c005180 0x7f381c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 -- 172.21.15.116:0/2223581617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c144810 con 0x7f381c0047c0 2023-11-28T17:11:30.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 -- 172.21.15.116:0/2223581617 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c00c020 con 0x7f381c005180 2023-11-28T17:11:30.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.798+0000 7f383a7fc640 1 -- 172.21.15.116:0/2223581617 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381c00c1e0 con 0x7f381c003970 2023-11-28T17:11:30.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.799+0000 7f38327fc640 1 --2- 172.21.15.116:0/2223581617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f381c0047c0 0x7f381c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.799+0000 7f383affd640 1 --2- 172.21.15.116:0/2223581617 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f381c003970 0x7f381c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.799+0000 7f383b7fe640 1 --2- 172.21.15.116:0/2223581617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f381c005180 0x7f381c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.799+0000 7f38327fc640 1 -- 172.21.15.116:0/2223581617 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f381c003970 msgr2=0x7f381c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.799+0000 7f38327fc640 1 --2- 172.21.15.116:0/2223581617 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f381c003970 0x7f381c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.799+0000 7f38327fc640 1 -- 172.21.15.116:0/2223581617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f381c005180 msgr2=0x7f381c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.799+0000 7f38327fc640 1 --2- 172.21.15.116:0/2223581617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f381c005180 0x7f381c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.799+0000 7f38327fc640 1 -- 172.21.15.116:0/2223581617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c003520 con 0x7f381c0047c0 2023-11-28T17:11:30.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.799+0000 7f383b7fe640 1 -- 172.21.15.116:0/2223581617 reap_dead start 2023-11-28T17:11:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.801+0000 7f38327fc640 1 --2- 172.21.15.116:0/2223581617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f381c0047c0 0x7f381c004c40 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f382400c240 tx=0x7f381c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3300f7e43d4a900 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.801+0000 7f383a7fc640 1 -- 172.21.15.116:0/2223581617 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824005ad0 con 0x7f381c0047c0 2023-11-28T17:11:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.801+0000 7f383a7fc640 1 -- 172.21.15.116:0/2223581617 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3824005c30 con 0x7f381c0047c0 2023-11-28T17:11:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.801+0000 7f383a7fc640 1 -- 172.21.15.116:0/2223581617 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382402c870 con 0x7f381c0047c0 2023-11-28T17:11:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.801+0000 7f38413b0640 1 -- 172.21.15.116:0/2223581617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f381c0047c0 msgr2=0x7f381c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.801+0000 7f38413b0640 1 --2- 172.21.15.116:0/2223581617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f381c0047c0 0x7f381c004c40 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f382400c240 tx=0x7f381c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.802+0000 7f38413b0640 1 -- 172.21.15.116:0/2223581617 shutdown_connections 2023-11-28T17:11:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.802+0000 7f38413b0640 1 --2- 172.21.15.116:0/2223581617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f381c0047c0 0x7f381c004c40 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.802+0000 7f38413b0640 1 -- 172.21.15.116:0/2223581617 >> 172.21.15.116:0/2223581617 conn(0x7f383c06a4c0 msgr2=0x7f383c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.802+0000 7f38413b0640 1 -- 172.21.15.116:0/2223581617 shutdown_connections 2023-11-28T17:11:30.804 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.802+0000 7f38413b0640 1 -- 172.21.15.116:0/2223581617 wait complete. 2023-11-28T17:11:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.803+0000 7f38413b0640 1 Processor -- start 2023-11-28T17:11:30.805 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.804+0000 7f38413b0640 1 -- start start 2023-11-28T17:11:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.804+0000 7f38413b0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f381c0047c0 0x7f383c12dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.804+0000 7f38413b0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f383c13db90 0x7f383c0c6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.804+0000 7f38413b0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f383c0c74e0 0x7f383c0b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.804+0000 7f38413b0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c12e3e0 con 0x7f381c0047c0 2023-11-28T17:11:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.804+0000 7f38413b0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c15d340 con 0x7f383c13db90 2023-11-28T17:11:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.804+0000 7f38413b0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c15d620 con 0x7f383c0c74e0 2023-11-28T17:11:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.805+0000 7f383affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f381c0047c0 0x7f383c12dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.805+0000 7f383affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f381c0047c0 0x7f383c12dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33968/0 (socket says 172.21.15.116:33968) 2023-11-28T17:11:30.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.805+0000 7f383affd640 1 -- 172.21.15.116:0/1350825906 learned_addr learned my addr 172.21.15.116:0/1350825906 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.805+0000 7f383affd640 1 -- 172.21.15.116:0/1350825906 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f383c0c74e0 msgr2=0x7f383c0b2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.805+0000 7f383affd640 1 --2- 172.21.15.116:0/1350825906 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f383c0c74e0 0x7f383c0b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.805+0000 7f383affd640 1 -- 172.21.15.116:0/1350825906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f383c13db90 msgr2=0x7f383c0c6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.805+0000 7f383affd640 1 --2- 172.21.15.116:0/1350825906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f383c13db90 0x7f383c0c6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.805+0000 7f383affd640 1 -- 172.21.15.116:0/1350825906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c15d900 con 0x7f381c0047c0 2023-11-28T17:11:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.805+0000 7f383affd640 1 --2- 172.21.15.116:0/1350825906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f381c0047c0 0x7f383c12dea0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f382c00e4b0 tx=0x7f382c00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.806+0000 7f3833fff640 1 -- 172.21.15.116:0/1350825906 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c0053a0 con 0x7f381c0047c0 2023-11-28T17:11:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.806+0000 7f3833fff640 1 -- 172.21.15.116:0/1350825906 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c0035f0 con 0x7f381c0047c0 2023-11-28T17:11:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.806+0000 7f3833fff640 1 -- 172.21.15.116:0/1350825906 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c0039a0 con 0x7f381c0047c0 2023-11-28T17:11:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.806+0000 7f38413b0640 1 -- 172.21.15.116:0/1350825906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383c15dbe0 con 0x7f381c0047c0 2023-11-28T17:11:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.806+0000 7f38413b0640 1 -- 172.21.15.116:0/1350825906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383c0b3360 con 0x7f381c0047c0 2023-11-28T17:11:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.808+0000 7f3833fff640 1 -- 172.21.15.116:0/1350825906 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f382c003b00 con 0x7f381c0047c0 2023-11-28T17:11:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.809+0000 7f3833fff640 1 --2- 172.21.15.116:0/1350825906 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f382006b2a0 0x7f382006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.809+0000 7f3833fff640 1 -- 172.21.15.116:0/1350825906 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f382c089980 con 0x7f381c0047c0 2023-11-28T17:11:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.809+0000 7f38327fc640 1 --2- 172.21.15.116:0/1350825906 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f382006b2a0 0x7f382006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.809+0000 7f3844026740 1 Processor -- start 2023-11-28T17:11:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.810+0000 7f3844026740 1 -- start start 2023-11-28T17:11:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.810+0000 7f38327fc640 1 --2- 172.21.15.116:0/1350825906 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f382006b2a0 0x7f382006d760 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3824034000 tx=0x7f3824039040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.810+0000 7f3844026740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67febce10 0x55e67febd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.810+0000 7f3844026740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67fe8d0c0 0x55e67feba080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.810+0000 7f3844026740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fe90990 0x55e67feba5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.811+0000 7f3844026740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e67fdf1880 con 0x55e67fe8d0c0 2023-11-28T17:11:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.811+0000 7f3844026740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e67fe00ae0 con 0x55e67febce10 2023-11-28T17:11:30.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.811+0000 7f3844026740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e67fdfe1d0 con 0x55e67fe90990 2023-11-28T17:11:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.811+0000 7f38327fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67fe8d0c0 0x55e67feba080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.811+0000 7f38327fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67fe8d0c0 0x55e67feba080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33972/0 (socket says 172.21.15.116:33972) 2023-11-28T17:11:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.811+0000 7f38327fc640 1 -- 172.21.15.116:0/2877871259 learned_addr learned my addr 172.21.15.116:0/2877871259 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.811+0000 7f383affd640 1 --2- 172.21.15.116:0/2877871259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67febce10 0x55e67febd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.811+0000 7f383b7fe640 1 --2- 172.21.15.116:0/2877871259 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fe90990 0x55e67feba5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.811+0000 7f38327fc640 1 -- 172.21.15.116:0/2877871259 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fe90990 msgr2=0x55e67feba5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.811+0000 7f38327fc640 1 --2- 172.21.15.116:0/2877871259 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fe90990 0x55e67feba5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.811+0000 7f38327fc640 1 -- 172.21.15.116:0/2877871259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67febce10 msgr2=0x55e67febd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.811+0000 7f38327fc640 1 --2- 172.21.15.116:0/2877871259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67febce10 0x55e67febd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.811+0000 7f38327fc640 1 -- 172.21.15.116:0/2877871259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e67fdc1860 con 0x55e67fe8d0c0 2023-11-28T17:11:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.811+0000 7f383b7fe640 1 --2- 172.21.15.116:0/2877871259 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fe90990 0x55e67feba5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.812+0000 7f38327fc640 1 --2- 172.21.15.116:0/2877871259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67fe8d0c0 0x55e67feba080 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3824038040 tx=0x7f3824007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6388121f669cc84c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:30.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.812+0000 7f3830ff9640 1 -- 172.21.15.116:0/2877871259 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824048070 con 0x55e67fe8d0c0 2023-11-28T17:11:30.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.812+0000 7f3830ff9640 1 -- 172.21.15.116:0/2877871259 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382402cdc0 con 0x55e67fe8d0c0 2023-11-28T17:11:30.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.812+0000 7f3830ff9640 1 -- 172.21.15.116:0/2877871259 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382403c040 con 0x55e67fe8d0c0 2023-11-28T17:11:30.819 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.812+0000 7f3844026740 1 -- 172.21.15.116:0/2877871259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67fe8d0c0 msgr2=0x55e67feba080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.813+0000 7f3844026740 1 --2- 172.21.15.116:0/2877871259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67fe8d0c0 0x55e67feba080 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3824038040 tx=0x7f3824007000 comp rx=0 tx=0).stop 2023-11-28T17:11:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.813+0000 7f3844026740 1 -- 172.21.15.116:0/2877871259 shutdown_connections 2023-11-28T17:11:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.813+0000 7f3844026740 1 --2- 172.21.15.116:0/2877871259 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fe90990 0x55e67feba5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.813+0000 7f3844026740 1 --2- 172.21.15.116:0/2877871259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67fe8d0c0 0x55e67feba080 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.813+0000 7f3844026740 1 --2- 172.21.15.116:0/2877871259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67febce10 0x55e67febd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.813+0000 7f3844026740 1 -- 172.21.15.116:0/2877871259 >> 172.21.15.116:0/2877871259 conn(0x55e67fe8f330 msgr2=0x55e67fed7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.813+0000 7f3844026740 1 -- 172.21.15.116:0/2877871259 shutdown_connections 2023-11-28T17:11:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.813+0000 7f3844026740 1 -- 172.21.15.116:0/2877871259 wait complete. 2023-11-28T17:11:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.813+0000 7f3844026740 1 -- start start 2023-11-28T17:11:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.814+0000 7f3844026740 1 Processor -- start 2023-11-28T17:11:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f3844026740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67fe90990 0x55e67fed82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f3844026740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67febce10 0x55e67fec0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f3844026740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fec1750 0x55e67feb5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f3844026740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e67fdf9ac0 con 0x55e67fe90990 2023-11-28T17:11:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f3844026740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e67fd962f0 con 0x55e67febce10 2023-11-28T17:11:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f3844026740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e67fdb3c70 con 0x55e67fec1750 2023-11-28T17:11:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f3844026740 10 client.0 _ll_register_callbacks cb 0x55e67fb0bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:11:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f383affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67fe90990 0x55e67fed82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f383affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67fe90990 0x55e67fed82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33988/0 (socket says 172.21.15.116:33988) 2023-11-28T17:11:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f383affd640 1 -- 172.21.15.116:0/1247392452 learned_addr learned my addr 172.21.15.116:0/1247392452 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f383affd640 1 -- 172.21.15.116:0/1247392452 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fec1750 msgr2=0x55e67feb5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f383affd640 1 --2- 172.21.15.116:0/1247392452 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fec1750 0x55e67feb5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f383affd640 1 -- 172.21.15.116:0/1247392452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67febce10 msgr2=0x55e67fec0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f383affd640 1 --2- 172.21.15.116:0/1247392452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67febce10 0x55e67fec0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f383affd640 1 -- 172.21.15.116:0/1247392452 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f381c003520 con 0x55e67fe90990 2023-11-28T17:11:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f383affd640 1 --2- 172.21.15.116:0/1247392452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67fe90990 0x55e67fed82b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f382c0601e0 tx=0x7f382c003e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8798b510570896c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.815+0000 7f3829ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-28T17:11:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f3829ffb640 1 -- 172.21.15.116:0/1247392452 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c05c560 con 0x55e67fe90990 2023-11-28T17:11:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f3829ffb640 1 -- 172.21.15.116:0/1247392452 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f382c060710 con 0x55e67fe90990 2023-11-28T17:11:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f3844026740 1 -- 172.21.15.116:0/1247392452 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e67fdf2aa0 con 0x55e67fe90990 2023-11-28T17:11:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f3829ffb640 1 -- 172.21.15.116:0/1247392452 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f382c05c160 con 0x55e67fe90990 2023-11-28T17:11:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f3844026740 10 client.5973 fetch_fsmap learned FSMap version 24 2023-11-28T17:11:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f3844026740 1 -- 172.21.15.116:0/1247392452 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x55e67fdf8140 con 0x55e67fe90990 2023-11-28T17:11:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f3829ffb640 1 -- 172.21.15.116:0/1247392452 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e67fdf2aa0 con 0x55e67fe90990 2023-11-28T17:11:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f3829ffb640 1 -- 172.21.15.116:0/1247392452 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f382c060280 con 0x55e67fe90990 2023-11-28T17:11:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f3844026740 10 client.5973 fetch_fsmap finished waiting for FSMap version 24 2023-11-28T17:11:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f3844026740 10 client.5973 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T17:11:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f3844026740 20 client.5973 populate_metadata read hostname 'smithi116' 2023-11-28T17:11:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f3844026740 1 --2- 172.21.15.116:0/1247392452 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x55e67febf980 0x55e67fef5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f3844026740 4 client.5973 mds_command: new command op to 5826 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T17:11:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f3844026740 1 -- 172.21.15.116:0/1247392452 --> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e67fdf27c0 con 0x55e67febf980 2023-11-28T17:11:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.816+0000 7f38327fc640 1 --2- 172.21.15.116:0/1247392452 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x55e67febf980 0x55e67fef5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.817+0000 7f38327fc640 1 --2- 172.21.15.116:0/1247392452 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x55e67febf980 0x55e67fef5290 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bad40eda84ff36f server_cookie=84d4aa47b89e22f6 in_seq=0 out_seq=0 2023-11-28T17:11:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.817+0000 7f3829ffb640 10 client.5973 ms_handle_connect on v2:172.21.15.116:6836/1482499383 2023-11-28T17:11:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3829ffb640 1 -- 172.21.15.116:0/1247392452 <== mds.0 v2:172.21.15.116:6836/1482499383 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e67fdf27c0 con 0x55e67febf980 2023-11-28T17:11:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3829ffb640 10 client.5973 handle_command_reply: tid=0 2023-11-28T17:11:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3844026740 1 client.5973 shutdown 2023-11-28T17:11:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3844026740 1 -- 172.21.15.116:0/1247392452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67fe90990 msgr2=0x55e67fed82b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3844026740 1 --2- 172.21.15.116:0/1247392452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67fe90990 0x55e67fed82b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f382c0601e0 tx=0x7f382c003e70 comp rx=0 tx=0).stop 2023-11-28T17:11:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3844026740 1 -- 172.21.15.116:0/1247392452 shutdown_connections 2023-11-28T17:11:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3844026740 1 --2- 172.21.15.116:0/1247392452 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x55e67febf980 0x55e67fef5290 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3844026740 1 --2- 172.21.15.116:0/1247392452 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fec1750 0x55e67feb5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3844026740 1 --2- 172.21.15.116:0/1247392452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67febce10 0x55e67fec0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3829ffb640 0 client.5973 ms_handle_reset on v2:172.21.15.116:6836/1482499383 2023-11-28T17:11:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3844026740 1 --2- 172.21.15.116:0/1247392452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67fe90990 0x55e67fed82b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3844026740 1 -- 172.21.15.116:0/1247392452 >> 172.21.15.116:0/1247392452 conn(0x55e67fe8d0c0 msgr2=0x55e67fe91310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3844026740 1 -- 172.21.15.116:0/1247392452 shutdown_connections 2023-11-28T17:11:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3844026740 1 -- 172.21.15.116:0/1247392452 wait complete. 2023-11-28T17:11:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.818+0000 7f3844026740 20 client.5973 trim_cache size 0 max 16384 2023-11-28T17:11:30.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.835+0000 7f3844026740 1 Processor -- start 2023-11-28T17:11:30.836 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.835+0000 7f3844026740 1 -- start start 2023-11-28T17:11:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.836+0000 7f3844026740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67fe90990 0x55e67fef98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.836+0000 7f3844026740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67febce10 0x55e67feb7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.836+0000 7f3844026740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fec1750 0x55e67feb8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.836+0000 7f3844026740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e67fb5cf20 con 0x55e67febce10 2023-11-28T17:11:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.836+0000 7f3844026740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e67fe8cf80 con 0x55e67fe90990 2023-11-28T17:11:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.836+0000 7f3844026740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e67fdfba40 con 0x55e67fec1750 2023-11-28T17:11:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.836+0000 7f38327fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67febce10 0x55e67feb7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.836+0000 7f38327fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67febce10 0x55e67feb7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34000/0 (socket says 172.21.15.116:34000) 2023-11-28T17:11:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.836+0000 7f38327fc640 1 -- 172.21.15.116:0/3906457248 learned_addr learned my addr 172.21.15.116:0/3906457248 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.837+0000 7f383b7fe640 1 --2- 172.21.15.116:0/3906457248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fec1750 0x55e67feb8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.837+0000 7f38327fc640 1 -- 172.21.15.116:0/3906457248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fec1750 msgr2=0x55e67feb8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.837+0000 7f38327fc640 1 --2- 172.21.15.116:0/3906457248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fec1750 0x55e67feb8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.837+0000 7f383affd640 1 --2- 172.21.15.116:0/3906457248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67fe90990 0x55e67fef98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.837+0000 7f38327fc640 1 -- 172.21.15.116:0/3906457248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67fe90990 msgr2=0x55e67fef98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.837+0000 7f38327fc640 1 --2- 172.21.15.116:0/3906457248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67fe90990 0x55e67fef98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.837+0000 7f38327fc640 1 -- 172.21.15.116:0/3906457248 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e67fd95b80 con 0x55e67febce10 2023-11-28T17:11:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.837+0000 7f383b7fe640 1 --2- 172.21.15.116:0/3906457248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fec1750 0x55e67feb8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.837+0000 7f38327fc640 1 --2- 172.21.15.116:0/3906457248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67febce10 0x55e67feb7b10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f382402c4e0 tx=0x7f382402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259ea85b1005f27 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.837+0000 7f38317fa640 1 -- 172.21.15.116:0/3906457248 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824048040 con 0x55e67febce10 2023-11-28T17:11:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.837+0000 7f38317fa640 1 -- 172.21.15.116:0/3906457248 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3824007230 con 0x55e67febce10 2023-11-28T17:11:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.837+0000 7f38317fa640 1 -- 172.21.15.116:0/3906457248 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382403c040 con 0x55e67febce10 2023-11-28T17:11:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.838+0000 7f3844026740 1 -- 172.21.15.116:0/3906457248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67febce10 msgr2=0x55e67feb7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.838+0000 7f3844026740 1 --2- 172.21.15.116:0/3906457248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67febce10 0x55e67feb7b10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f382402c4e0 tx=0x7f382402e000 comp rx=0 tx=0).stop 2023-11-28T17:11:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.838+0000 7f3844026740 1 -- 172.21.15.116:0/3906457248 shutdown_connections 2023-11-28T17:11:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.838+0000 7f3844026740 1 --2- 172.21.15.116:0/3906457248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fec1750 0x55e67feb8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.838+0000 7f3844026740 1 --2- 172.21.15.116:0/3906457248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67febce10 0x55e67feb7b10 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.838+0000 7f3844026740 1 --2- 172.21.15.116:0/3906457248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67fe90990 0x55e67fef98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.838+0000 7f3844026740 1 -- 172.21.15.116:0/3906457248 >> 172.21.15.116:0/3906457248 conn(0x55e67fe8d0c0 msgr2=0x55e67fedb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:30.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.838+0000 7f3844026740 1 -- 172.21.15.116:0/3906457248 shutdown_connections 2023-11-28T17:11:30.841 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.838+0000 7f3844026740 1 -- 172.21.15.116:0/3906457248 wait complete. 2023-11-28T17:11:30.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.839+0000 7f3844026740 1 -- start start 2023-11-28T17:11:30.844 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.839+0000 7f3844026740 1 Processor -- start 2023-11-28T17:11:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f3844026740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fe90990 0x55e67ff279e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f3844026740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67febce10 0x55e67fec0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f3844026740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67fec1750 0x55e67fec09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f3844026740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e67fe8cb50 con 0x55e67febce10 2023-11-28T17:11:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f3844026740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e67ff41b90 con 0x55e67fec1750 2023-11-28T17:11:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f3844026740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e67ff41de0 con 0x55e67fe90990 2023-11-28T17:11:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f3844026740 10 client.0 _ll_register_callbacks cb 0x7f382c007ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:11:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f38327fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67febce10 0x55e67fec0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f38327fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67febce10 0x55e67fec0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34004/0 (socket says 172.21.15.116:34004) 2023-11-28T17:11:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f38327fc640 1 -- 172.21.15.116:0/487318945 learned_addr learned my addr 172.21.15.116:0/487318945 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f38327fc640 1 -- 172.21.15.116:0/487318945 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fe90990 msgr2=0x55e67ff279e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f38327fc640 1 --2- 172.21.15.116:0/487318945 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fe90990 0x55e67ff279e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f38327fc640 1 -- 172.21.15.116:0/487318945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67fec1750 msgr2=0x55e67fec09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f38327fc640 1 --2- 172.21.15.116:0/487318945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67fec1750 0x55e67fec09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.840+0000 7f38327fc640 1 -- 172.21.15.116:0/487318945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e67fdc1860 con 0x55e67febce10 2023-11-28T17:11:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.841+0000 7f38327fc640 1 --2- 172.21.15.116:0/487318945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67febce10 0x55e67fec0460 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3824001930 tx=0x7f382402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dec058fcd13a227 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.841+0000 7f3829ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-28T17:11:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.841+0000 7f3829ffb640 1 -- 172.21.15.116:0/487318945 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824048040 con 0x55e67febce10 2023-11-28T17:11:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.841+0000 7f3829ffb640 1 -- 172.21.15.116:0/487318945 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38240447c0 con 0x55e67febce10 2023-11-28T17:11:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.841+0000 7f3844026740 1 -- 172.21.15.116:0/487318945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f382402cdc0 con 0x55e67febce10 2023-11-28T17:11:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.841+0000 7f3829ffb640 1 -- 172.21.15.116:0/487318945 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f382403c040 con 0x55e67febce10 2023-11-28T17:11:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.841+0000 7f3844026740 10 client.5979 fetch_fsmap learned FSMap version 24 2023-11-28T17:11:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.841+0000 7f3844026740 1 -- 172.21.15.116:0/487318945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x55e67fdbb170 con 0x55e67febce10 2023-11-28T17:11:30.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.841+0000 7f3829ffb640 1 -- 172.21.15.116:0/487318945 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382402cdc0 con 0x55e67febce10 2023-11-28T17:11:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.841+0000 7f3829ffb640 1 -- 172.21.15.116:0/487318945 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3824061070 con 0x55e67febce10 2023-11-28T17:11:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.842+0000 7f3844026740 10 client.5979 fetch_fsmap finished waiting for FSMap version 24 2023-11-28T17:11:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.842+0000 7f3844026740 10 client.5979 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T17:11:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.842+0000 7f3844026740 20 client.5979 populate_metadata read hostname 'smithi116' 2023-11-28T17:11:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.842+0000 7f3844026740 1 --2- 172.21.15.116:0/487318945 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x55e67fea9d20 0x55e67feaa100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.842+0000 7f3844026740 4 client.5979 mds_command: new command op to 5826 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T17:11:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.842+0000 7f3844026740 1 -- 172.21.15.116:0/487318945 --> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e67fafa6c0 con 0x55e67fea9d20 2023-11-28T17:11:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.842+0000 7f383affd640 1 --2- 172.21.15.116:0/487318945 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x55e67fea9d20 0x55e67feaa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.842+0000 7f383affd640 1 --2- 172.21.15.116:0/487318945 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x55e67fea9d20 0x55e67feaa100 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a432c33b70a2218 server_cookie=7aa9d8b2defe02cf in_seq=0 out_seq=0 2023-11-28T17:11:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.842+0000 7f3829ffb640 10 client.5979 ms_handle_connect on v2:172.21.15.116:6836/1482499383 2023-11-28T17:11:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3829ffb640 1 -- 172.21.15.116:0/487318945 <== mds.0 v2:172.21.15.116:6836/1482499383 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55e67fafa6c0 con 0x55e67fea9d20 2023-11-28T17:11:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3829ffb640 10 client.5979 handle_command_reply: tid=0 2023-11-28T17:11:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3844026740 1 client.5979 shutdown 2023-11-28T17:11:30.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3844026740 1 -- 172.21.15.116:0/487318945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67febce10 msgr2=0x55e67fec0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3844026740 1 --2- 172.21.15.116:0/487318945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67febce10 0x55e67fec0460 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3824001930 tx=0x7f382402e070 comp rx=0 tx=0).stop 2023-11-28T17:11:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3844026740 1 -- 172.21.15.116:0/487318945 shutdown_connections 2023-11-28T17:11:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3844026740 1 --2- 172.21.15.116:0/487318945 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x55e67fea9d20 0x55e67feaa100 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3844026740 1 --2- 172.21.15.116:0/487318945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e67fec1750 0x55e67fec09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3844026740 1 --2- 172.21.15.116:0/487318945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e67febce10 0x55e67fec0460 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3844026740 1 --2- 172.21.15.116:0/487318945 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55e67fe90990 0x55e67ff279e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3829ffb640 0 client.5979 ms_handle_reset on v2:172.21.15.116:6836/1482499383 2023-11-28T17:11:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3844026740 1 -- 172.21.15.116:0/487318945 >> 172.21.15.116:0/487318945 conn(0x55e67fe8d0c0 msgr2=0x55e67feb9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3844026740 1 -- 172.21.15.116:0/487318945 shutdown_connections 2023-11-28T17:11:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3844026740 1 -- 172.21.15.116:0/487318945 wait complete. 2023-11-28T17:11:30.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.843+0000 7f3844026740 20 client.5979 trim_cache size 0 max 16384 2023-11-28T17:11:30.849 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-28T17:11:30.849 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a5f2fbcf-fde4-4c38-94da-08178784d9fc", 2023-11-28T17:11:30.849 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2023-11-28T17:11:30.849 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 5826, 2023-11-28T17:11:30.849 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2023-11-28T17:11:30.849 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2023-11-28T17:11:30.849 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2023-11-28T17:11:30.849 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 6.6482841849999996, 2023-11-28T17:11:30.849 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 24, 2023-11-28T17:11:30.849 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 48, 2023-11-28T17:11:30.850 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 48, 2023-11-28T17:11:30.850 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 19.568205291000002 2023-11-28T17:11:30.850 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-28T17:11:30.850 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.844+0000 7f38413b0640 1 -- 172.21.15.116:0/1350825906 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f382006b2a0 msgr2=0x7f382006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.844+0000 7f38413b0640 1 --2- 172.21.15.116:0/1350825906 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f382006b2a0 0x7f382006d760 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3824034000 tx=0x7f3824039040 comp rx=0 tx=0).stop 2023-11-28T17:11:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.844+0000 7f38413b0640 1 -- 172.21.15.116:0/1350825906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f381c0047c0 msgr2=0x7f383c12dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.844+0000 7f38413b0640 1 --2- 172.21.15.116:0/1350825906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f381c0047c0 0x7f383c12dea0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f382c00e4b0 tx=0x7f382c00e970 comp rx=0 tx=0).stop 2023-11-28T17:11:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.844+0000 7f38413b0640 1 -- 172.21.15.116:0/1350825906 shutdown_connections 2023-11-28T17:11:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.844+0000 7f38413b0640 1 --2- 172.21.15.116:0/1350825906 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f382006b2a0 0x7f382006d760 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.844+0000 7f38413b0640 1 --2- 172.21.15.116:0/1350825906 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f383c0c74e0 0x7f383c0b2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.844+0000 7f38413b0640 1 --2- 172.21.15.116:0/1350825906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f383c13db90 0x7f383c0c6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.844+0000 7f38413b0640 1 --2- 172.21.15.116:0/1350825906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f381c0047c0 0x7f383c12dea0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.844+0000 7f38413b0640 1 -- 172.21.15.116:0/1350825906 >> 172.21.15.116:0/1350825906 conn(0x7f383c06a4c0 msgr2=0x7f383c0cc200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.844+0000 7f38413b0640 1 -- 172.21.15.116:0/1350825906 shutdown_connections 2023-11-28T17:11:30.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:30.844+0000 7f38413b0640 1 -- 172.21.15.116:0/1350825906 wait complete. 2023-11-28T17:11:30.861 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T17:11:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.056+0000 7f280111d640 1 Processor -- start 2023-11-28T17:11:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.057+0000 7f280111d640 1 -- start start 2023-11-28T17:11:31.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.058+0000 7f280111d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f27fc15aba0 0x7f27fc164e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:31.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.058+0000 7f280111d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc0aea20 con 0x7f27fc13cee0 2023-11-28T17:11:31.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.058+0000 7f280111d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc13d2c0 con 0x7f27fc06c960 2023-11-28T17:11:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.058+0000 7f280111d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc15b3a0 con 0x7f27fc15aba0 2023-11-28T17:11:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.059+0000 7f27fad9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f27fc13cee0 0x7f27fc06c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45276/0 (socket says 172.21.15.116:45276) 2023-11-28T17:11:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.059+0000 7f27fad9d640 1 -- 172.21.15.116:0/2009320627 learned_addr learned my addr 172.21.15.116:0/2009320627 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.059+0000 7f27fa59c640 1 -- 172.21.15.116:0/2009320627 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522810969 0 0) 0x7f27fc0aea20 con 0x7f27fc13cee0 2023-11-28T17:11:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.059+0000 7f27fa59c640 1 -- 172.21.15.116:0/2009320627 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e8003520 con 0x7f27fc13cee0 2023-11-28T17:11:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.059+0000 7f27fb59e640 1 --2- 172.21.15.116:0/2009320627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f27fc15aba0 0x7f27fc164e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.059+0000 7f27fa59c640 1 -- 172.21.15.116:0/2009320627 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 413491475 0 0) 0x7f27e8003520 con 0x7f27fc13cee0 2023-11-28T17:11:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.059+0000 7f27fa59c640 1 -- 172.21.15.116:0/2009320627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f27fc15aba0 msgr2=0x7f27fc164e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.059+0000 7f27fa59c640 1 --2- 172.21.15.116:0/2009320627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f27fc15aba0 0x7f27fc164e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.059+0000 7f27fa59c640 1 -- 172.21.15.116:0/2009320627 >> v1:172.21.15.156:6789/0 conn(0x7f27fc06c960 legacy=0x7f27fc15a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.059+0000 7f27fa59c640 1 -- 172.21.15.116:0/2009320627 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc15b0c0 con 0x7f27fc13cee0 2023-11-28T17:11:31.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.059+0000 7f27fa59c640 1 -- 172.21.15.116:0/2009320627 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f27ec001670 con 0x7f27fc13cee0 2023-11-28T17:11:31.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.059+0000 7f27fa59c640 1 -- 172.21.15.116:0/2009320627 >> v1:172.21.15.116:6789/0 conn(0x7f27fc13cee0 legacy=0x7f27fc06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:31.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.059+0000 7f27fa59c640 1 --2- 172.21.15.116:0/2009320627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f27fc15aba0 0x7f27e80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:31.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.059+0000 7f27fa59c640 1 --2- 172.21.15.116:0/2009320627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e8004000 0x7f27e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:31.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.060+0000 7f27fa59c640 1 --2- 172.21.15.116:0/2009320627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27e8004d00 0x7f27e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:31.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.060+0000 7f27fa59c640 1 -- 172.21.15.116:0/2009320627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc15b3a0 con 0x7f27e8004000 2023-11-28T17:11:31.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.060+0000 7f27fa59c640 1 -- 172.21.15.116:0/2009320627 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e800bbe0 con 0x7f27e8004d00 2023-11-28T17:11:31.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.060+0000 7f27fa59c640 1 -- 172.21.15.116:0/2009320627 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e800dfb0 con 0x7f27fc15aba0 2023-11-28T17:11:31.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.060+0000 7f27f3fff640 1 --2- 172.21.15.116:0/2009320627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e8004000 0x7f27e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:31.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.060+0000 7f27fad9d640 1 --2- 172.21.15.116:0/2009320627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f27fc15aba0 0x7f27e80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:31.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.060+0000 7f27fb59e640 1 --2- 172.21.15.116:0/2009320627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27e8004d00 0x7f27e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:31.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.060+0000 7f27f3fff640 1 -- 172.21.15.116:0/2009320627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f27fc15aba0 msgr2=0x7f27e80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.060+0000 7f27f3fff640 1 --2- 172.21.15.116:0/2009320627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f27fc15aba0 0x7f27e80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.060+0000 7f27f3fff640 1 -- 172.21.15.116:0/2009320627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27e8004d00 msgr2=0x7f27e8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.060+0000 7f27f3fff640 1 --2- 172.21.15.116:0/2009320627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27e8004d00 0x7f27e8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.060+0000 7f27f3fff640 1 -- 172.21.15.116:0/2009320627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e8003520 con 0x7f27e8004000 2023-11-28T17:11:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.061+0000 7f27f3fff640 1 --2- 172.21.15.116:0/2009320627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e8004000 0x7f27e80047c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f27e4003db0 tx=0x7f27e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41088b2babb0c661 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.062+0000 7f27fa59c640 1 -- 172.21.15.116:0/2009320627 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e4032600 con 0x7f27e8004000 2023-11-28T17:11:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.062+0000 7f27fa59c640 1 -- 172.21.15.116:0/2009320627 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e402e050 con 0x7f27e8004000 2023-11-28T17:11:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.062+0000 7f27fa59c640 1 -- 172.21.15.116:0/2009320627 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e40329e0 con 0x7f27e8004000 2023-11-28T17:11:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.062+0000 7f280111d640 1 -- 172.21.15.116:0/2009320627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e8004000 msgr2=0x7f27e80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.062+0000 7f280111d640 1 --2- 172.21.15.116:0/2009320627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e8004000 0x7f27e80047c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f27e4003db0 tx=0x7f27e80057f0 comp rx=0 tx=0).stop 2023-11-28T17:11:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.062+0000 7f27fb59e640 1 -- 172.21.15.116:0/2009320627 reap_dead start 2023-11-28T17:11:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.062+0000 7f280111d640 1 -- 172.21.15.116:0/2009320627 shutdown_connections 2023-11-28T17:11:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.062+0000 7f280111d640 1 -- 172.21.15.116:0/2009320627 >> 172.21.15.116:0/2009320627 conn(0x7f27fc06a4c0 msgr2=0x7f27fc06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.062+0000 7f280111d640 1 -- 172.21.15.116:0/2009320627 shutdown_connections 2023-11-28T17:11:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.062+0000 7f280111d640 1 -- 172.21.15.116:0/2009320627 wait complete. 2023-11-28T17:11:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.064+0000 7f280111d640 1 Processor -- start 2023-11-28T17:11:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.064+0000 7f280111d640 1 -- start start 2023-11-28T17:11:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.065+0000 7f280111d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e8004000 0x7f27fc1f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.065+0000 7f280111d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f27fc0afa90 0x7f27fc13b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.065+0000 7f280111d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27fc13bc90 0x7f27fc13d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.065+0000 7f280111d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc13d350 con 0x7f27e8004000 2023-11-28T17:11:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.065+0000 7f280111d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc0b0000 con 0x7f27fc13bc90 2023-11-28T17:11:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.065+0000 7f280111d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc0b02e0 con 0x7f27fc0afa90 2023-11-28T17:11:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.066+0000 7f27fad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e8004000 0x7f27fc1f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.066+0000 7f27fad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e8004000 0x7f27fc1f2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34024/0 (socket says 172.21.15.116:34024) 2023-11-28T17:11:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.066+0000 7f27fad9d640 1 -- 172.21.15.116:0/4250803379 learned_addr learned my addr 172.21.15.116:0/4250803379 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.066+0000 7f27f3fff640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f27fc0afa90 0x7f27fc13b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.066+0000 7f27fb59e640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27fc13bc90 0x7f27fc13d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.066+0000 7f27fad9d640 1 -- 172.21.15.116:0/4250803379 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f27fc0afa90 msgr2=0x7f27fc13b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.066+0000 7f27fad9d640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f27fc0afa90 0x7f27fc13b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.066+0000 7f27fad9d640 1 -- 172.21.15.116:0/4250803379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27fc13bc90 msgr2=0x7f27fc13d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.066+0000 7f27fad9d640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27fc13bc90 0x7f27fc13d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.066+0000 7f27fad9d640 1 -- 172.21.15.116:0/4250803379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc0b05c0 con 0x7f27e8004000 2023-11-28T17:11:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.066+0000 7f27fb59e640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27fc13bc90 0x7f27fc13d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.066+0000 7f27fad9d640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e8004000 0x7f27fc1f2e30 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f27ec001050 tx=0x7f27ec00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.067+0000 7f27f37fe640 1 -- 172.21.15.116:0/4250803379 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec00eeb0 con 0x7f27e8004000 2023-11-28T17:11:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.067+0000 7f27f37fe640 1 -- 172.21.15.116:0/4250803379 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ec00c8d0 con 0x7f27e8004000 2023-11-28T17:11:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.067+0000 7f27f37fe640 1 -- 172.21.15.116:0/4250803379 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec00cc80 con 0x7f27e8004000 2023-11-28T17:11:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.067+0000 7f280111d640 1 -- 172.21.15.116:0/4250803379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27fc0b08a0 con 0x7f27e8004000 2023-11-28T17:11:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.067+0000 7f280111d640 1 -- 172.21.15.116:0/4250803379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27fc149640 con 0x7f27e8004000 2023-11-28T17:11:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.068+0000 7f280111d640 1 -- 172.21.15.116:0/4250803379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27fc0b0a90 con 0x7f27e8004000 2023-11-28T17:11:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.069+0000 7f27f37fe640 1 -- 172.21.15.116:0/4250803379 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f27ec01d460 con 0x7f27e8004000 2023-11-28T17:11:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.070+0000 7f27f37fe640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f27dc06b2a0 0x7f27dc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.070+0000 7f27f37fe640 1 -- 172.21.15.116:0/4250803379 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27ec089c30 con 0x7f27e8004000 2023-11-28T17:11:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.070+0000 7f27f3fff640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f27dc06b2a0 0x7f27dc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.071+0000 7f27f3fff640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f27dc06b2a0 0x7f27dc06d760 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f27e4034000 tx=0x7f27e4009030 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.075+0000 7f27f37fe640 1 -- 172.21.15.116:0/4250803379 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27ec054aa0 con 0x7f27e8004000 2023-11-28T17:11:31.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.303+0000 7f280111d640 1 -- 172.21.15.116:0/4250803379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f27fc0b0420 con 0x7f27e8004000 2023-11-28T17:11:31.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.303+0000 7f27f37fe640 1 -- 172.21.15.116:0/4250803379 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+73 (secure 0 0 0) 0x7f27fc0b0420 con 0x7f27e8004000 2023-11-28T17:11:31.305 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T17:11:31.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.306+0000 7f280111d640 1 -- 172.21.15.116:0/4250803379 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f27dc06b2a0 msgr2=0x7f27dc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:31.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.306+0000 7f280111d640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f27dc06b2a0 0x7f27dc06d760 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f27e4034000 tx=0x7f27e4009030 comp rx=0 tx=0).stop 2023-11-28T17:11:31.307 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.306+0000 7f280111d640 1 -- 172.21.15.116:0/4250803379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e8004000 msgr2=0x7f27fc1f2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.306+0000 7f280111d640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e8004000 0x7f27fc1f2e30 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f27ec001050 tx=0x7f27ec00ec60 comp rx=0 tx=0).stop 2023-11-28T17:11:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.306+0000 7f280111d640 1 -- 172.21.15.116:0/4250803379 shutdown_connections 2023-11-28T17:11:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.306+0000 7f280111d640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f27dc06b2a0 0x7f27dc06d760 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.306+0000 7f280111d640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27fc13bc90 0x7f27fc13d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.306+0000 7f280111d640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f27fc0afa90 0x7f27fc13b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.306+0000 7f280111d640 1 --2- 172.21.15.116:0/4250803379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e8004000 0x7f27fc1f2e30 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.306+0000 7f280111d640 1 -- 172.21.15.116:0/4250803379 >> 172.21.15.116:0/4250803379 conn(0x7f27fc06a4c0 msgr2=0x7f27fc15cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.306+0000 7f280111d640 1 -- 172.21.15.116:0/4250803379 shutdown_connections 2023-11-28T17:11:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:31.306+0000 7f280111d640 1 -- 172.21.15.116:0/4250803379 wait complete. 2023-11-28T17:11:31.325 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T17:11:31.325 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T17:11:31.325 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2023-11-28T17:11:31.325 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-28T17:11:31.325 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-28T17:11:31.325 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T17:11:31.325 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T17:11:31.325 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T17:11:31.325 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T17:11:31.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:31.326 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2023-11-28T17:11:31.378 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T17:11:31.378 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T17:11:31.378 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T17:11:31.378 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:11:31.378 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2023-11-28T17:11:31.378 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:11:31.378 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:11:31.378 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:11:31.379 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2023-11-28T17:11:31.379 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:11:31.379 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:11:31.379 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2023-11-28T17:11:31.379 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:11:31.379 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:11:31.379 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T17:11:31.379 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:11:31.379 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T17:11:31.379 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42247sec preferred_lft 42247sec 2023-11-28T17:11:31.379 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2023-11-28T17:11:31.379 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:11:31.379 INFO:teuthology.orchestra.run.smithi116.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T17:11:31.380 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 66:db:2d:c1:d7:e7 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:11:31.380 INFO:teuthology.orchestra.run.smithi116.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T17:11:31.380 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:11:31.380 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::c06c:35ff:fe15:dc8e/64 scope link 2023-11-28T17:11:31.380 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:11:31.380 INFO:teuthology.orchestra.run.smithi116.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T17:11:31.380 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 66:db:2d:c1:d7:e7 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:11:31.380 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::64db:2dff:fec1:d7e7/64 scope link 2023-11-28T17:11:31.380 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:11:31.380 INFO:teuthology.orchestra.run.smithi116.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T17:11:31.380 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether c2:79:c2:c8:b7:18 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:11:31.380 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::c079:c2ff:fec8:b718/64 scope link 2023-11-28T17:11:31.380 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:11:31.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:31.381 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T17:11:31.463 INFO:teuthology.orchestra.run.smithi116.stdout:1 2023-11-28T17:11:31.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:31.467 DEBUG:teuthology.orchestra.run.smithi116:> route 2023-11-28T17:11:31.516 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2023-11-28T17:11:31.517 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T17:11:31.517 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T17:11:31.517 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T17:11:31.517 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T17:11:31.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:31.518 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec bash -c ' 2023-11-28T17:11:31.518 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2023-11-28T17:11:31.518 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T17:11:31.518 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T17:11:31.518 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T17:11:31.518 DEBUG:teuthology.orchestra.run.smithi116:> ') 2023-11-28T17:11:31.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:31.698 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2023-11-28T17:11:31.749 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T17:11:31.749 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T17:11:31.750 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-28T17:11:31.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:31.751 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2023-11-28T17:11:31.842 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-11-28T17:11:31.842 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:11:31.842 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-28T17:11:31.896 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-28T17:11:31.896 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:11:31.896 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-28T17:11:31.944 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-28T17:11:32.086 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:11:32.140 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2023-11-28T17:11:32.141 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2023-11-28T17:11:32.141 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2023-11-28T17:11:32.141 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2023-11-28T17:11:32.141 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2023-11-28T17:11:32.141 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2023-11-28T17:11:32.141 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2023-11-28T17:11:32.141 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2023-11-28T17:11:32.141 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2023-11-28T17:11:32.141 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2023-11-28T17:11:32.141 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2023-11-28T17:11:32.141 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:11:32.141 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2023-11-28T17:11:32.142 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2023-11-28T17:11:32.142 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2023-11-28T17:11:32.142 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2023-11-28T17:11:32.142 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2023-11-28T17:11:32.142 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2023-11-28T17:11:32.142 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:11:32.142 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:11:32.145 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:11:32.145 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:11:32.146 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2023-11-28T17:11:32.146 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2023-11-28T17:11:32.146 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T17:11:32.146 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T17:11:32.146 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T17:11:32.146 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T17:11:32.146 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2023-11-28T17:11:32.146 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:11:32.146 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:11:32.146 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:11:32.146 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:11:32.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:32.147 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-28T17:11:32.201 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-28T17:11:32.201 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-11-28T17:11:32.206 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2023-11-28T17:11:32.285 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:11:32.326 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2023-11-28T17:11:32.326 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2023-11-28T17:11:32.326 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2023-11-28T17:11:32.326 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2023-11-28T17:11:32.326 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2023-11-28T17:11:32.326 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2023-11-28T17:11:32.326 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2023-11-28T17:11:32.327 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2023-11-28T17:11:32.327 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2023-11-28T17:11:32.327 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2023-11-28T17:11:32.327 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2023-11-28T17:11:32.327 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:11:32.327 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2023-11-28T17:11:32.327 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2023-11-28T17:11:32.327 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2023-11-28T17:11:32.327 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2023-11-28T17:11:32.327 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2023-11-28T17:11:32.327 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2023-11-28T17:11:32.327 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:11:32.328 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:11:32.328 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:11:32.328 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:11:32.328 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2023-11-28T17:11:32.328 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2023-11-28T17:11:32.328 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-0 2023-11-28T17:11:32.328 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-1 2023-11-28T17:11:32.328 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-2 2023-11-28T17:11:32.328 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-3 2023-11-28T17:11:32.328 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2023-11-28T17:11:32.328 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:11:32.328 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:11:32.329 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:11:32.329 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:11:32.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:32.330 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2023-11-28T17:11:32.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:2023-11-28T17:11:32.393+0000 7f63f261f480 -1 init, newargv = 0x5573eaa3d5a0 newargc=15 2023-11-28T17:11:32.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[107062]: starting ceph client 2023-11-28T17:11:32.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[107062]: starting fuse 2023-11-28T17:11:32.418 INFO:teuthology.orchestra.run.smithi116.stdout:39 2023-11-28T17:11:32.418 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> sudo stdin-killer -- python3 -c ' 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> import re 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> import os 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2023-11-28T17:11:32.419 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2023-11-28T17:11:32.420 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:11:32.420 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2023-11-28T17:11:32.420 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-28T17:11:32.420 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-28T17:11:32.420 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-28T17:11:32.420 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2023-11-28T17:11:32.420 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2023-11-28T17:11:32.420 DEBUG:teuthology.orchestra.run.smithi116:> return f 2023-11-28T17:11:32.420 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-28T17:11:32.420 DEBUG:teuthology.orchestra.run.smithi116:> 2023-11-28T17:11:32.420 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.0")) 2023-11-28T17:11:32.420 DEBUG:teuthology.orchestra.run.smithi116:> ' 2023-11-28T17:11:32.618 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.0.107062.asok 2023-11-28T17:11:32.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-11-28T17:11:32.634 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.107062.asok 2023-11-28T17:11:32.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:32.635 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.107062.asok status 2023-11-28T17:11:32.874 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-28T17:11:32.874 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2023-11-28T17:11:32.874 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "47c1c547372ed5bff7984647dc200bac0e019caf", 2023-11-28T17:11:32.874 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev)", 2023-11-28T17:11:32.874 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "0", 2023-11-28T17:11:32.874 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2023-11-28T17:11:32.874 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-11-28T17:11:32.875 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "107062", 2023-11-28T17:11:32.875 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2023-11-28T17:11:32.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:32.875 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2023-11-28T17:11:32.875 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2023-11-28T17:11:32.875 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 5994, 2023-11-28T17:11:32.875 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2023-11-28T17:11:32.875 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2023-11-28T17:11:32.875 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2023-11-28T17:11:32.875 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 5994 2023-11-28T17:11:32.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:32.875 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-28T17:11:32.875 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-28T17:11:32.875 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-28T17:11:32.876 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2188629007 2023-11-28T17:11:32.876 INFO:teuthology.orchestra.run.smithi116.stdout: } 2023-11-28T17:11:32.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:32.876 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2023-11-28T17:11:32.876 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2023-11-28T17:11:32.876 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2023-11-28T17:11:32.876 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2188629007 2023-11-28T17:11:32.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2023-11-28T17:11:32.876 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.5994 192.168.0.1:0/2188629007", 2023-11-28T17:11:32.876 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.1:0/2188629007", 2023-11-28T17:11:32.876 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2023-11-28T17:11:32.876 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 24, 2023-11-28T17:11:32.877 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 48, 2023-11-28T17:11:32.877 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2023-11-28T17:11:32.877 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2023-11-28T17:11:32.877 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2023-11-28T17:11:32.877 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-28T17:11:32.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:32.889 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2023-11-28T17:11:32.889 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.0 2023-11-28T17:11:32.943 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2023-11-28T17:11:32.943 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-11-28T17:11:32.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:32.944 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-28T17:11:33.037 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-28T17:11:33.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.250+0000 7f8c0171d640 1 Processor -- start 2023-11-28T17:11:33.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.251+0000 7f8c0171d640 1 -- start start 2023-11-28T17:11:33.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.252+0000 7f8c0171d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bfc15d2c0 0x7f8bfc0cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:33.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.252+0000 7f8c0171d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8bfc0d2f40 0x7f8bfc0b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:33.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.252+0000 7f8c0171d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc0b6640 con 0x7f8bfc0cdf70 2023-11-28T17:11:33.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.252+0000 7f8c0171d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bfc0ce3f0 con 0x7f8bfc0d2f40 2023-11-28T17:11:33.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.252+0000 7f8c0171d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc0d3480 con 0x7f8bfc15d2c0 2023-11-28T17:11:33.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.253+0000 7f8bfb7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8bfc0d2f40 0x7f8bfc0b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:33.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.253+0000 7f8bfb7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8bfc0d2f40 0x7f8bfc0b6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34036/0 (socket says 172.21.15.116:34036) 2023-11-28T17:11:33.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.253+0000 7f8bfb7fe640 1 -- 172.21.15.116:0/3635450011 learned_addr learned my addr 172.21.15.116:0/3635450011 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:33.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.253+0000 7f8bfaffd640 1 --2- 172.21.15.116:0/3635450011 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bfc15d2c0 0x7f8bfc0cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:33.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.253+0000 7f8bfb7fe640 1 -- 172.21.15.116:0/3635450011 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bfc15d2c0 msgr2=0x7f8bfc0cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:33.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.253+0000 7f8bfb7fe640 1 --2- 172.21.15.116:0/3635450011 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bfc15d2c0 0x7f8bfc0cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:33.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.253+0000 7f8bfb7fe640 1 -- 172.21.15.116:0/3635450011 >> v1:172.21.15.156:6789/0 conn(0x7f8bfc0cdf70 legacy=0x7f8bfc0d2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:33.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.253+0000 7f8bfb7fe640 1 -- 172.21.15.116:0/3635450011 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc0d3760 con 0x7f8bfc0d2f40 2023-11-28T17:11:33.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.254+0000 7f8bfb7fe640 1 --2- 172.21.15.116:0/3635450011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc0d2f40 0x7f8bfc0b6ad0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009870 tx=0x7f8bf402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb7712521f0b9ad server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:33.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.254+0000 7f8bf9ffb640 1 -- 172.21.15.116:0/3635450011 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4031640 con 0x7f8bfc0d2f40 2023-11-28T17:11:33.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.255+0000 7f8bf9ffb640 1 -- 172.21.15.116:0/3635450011 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf40317a0 con 0x7f8bfc0d2f40 2023-11-28T17:11:33.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.255+0000 7f8bf9ffb640 1 -- 172.21.15.116:0/3635450011 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4031b30 con 0x7f8bfc0d2f40 2023-11-28T17:11:33.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.255+0000 7f8c0171d640 1 -- 172.21.15.116:0/3635450011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc0d2f40 msgr2=0x7f8bfc0b6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:33.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.255+0000 7f8c0171d640 1 --2- 172.21.15.116:0/3635450011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc0d2f40 0x7f8bfc0b6ad0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009870 tx=0x7f8bf402b8f0 comp rx=0 tx=0).stop 2023-11-28T17:11:33.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.255+0000 7f8c0171d640 1 -- 172.21.15.116:0/3635450011 shutdown_connections 2023-11-28T17:11:33.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.255+0000 7f8c0171d640 1 --2- 172.21.15.116:0/3635450011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc0d2f40 0x7f8bfc0b6ad0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.255+0000 7f8c0171d640 1 --2- 172.21.15.116:0/3635450011 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bfc15d2c0 0x7f8bfc0cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.255+0000 7f8c0171d640 1 -- 172.21.15.116:0/3635450011 >> 172.21.15.116:0/3635450011 conn(0x7f8bfc072010 msgr2=0x7f8bfc072420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.255+0000 7f8c0171d640 1 -- 172.21.15.116:0/3635450011 shutdown_connections 2023-11-28T17:11:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.255+0000 7f8c0171d640 1 -- 172.21.15.116:0/3635450011 wait complete. 2023-11-28T17:11:33.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.257+0000 7f8c0171d640 1 Processor -- start 2023-11-28T17:11:33.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.257+0000 7f8c0171d640 1 -- start start 2023-11-28T17:11:33.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.258+0000 7f8c0171d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bfc0cdf70 0x7f8bfc150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:33.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.258+0000 7f8c0171d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bfc0d2f40 0x7f8bfc0c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:33.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.258+0000 7f8c0171d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc15d2c0 0x7f8bfc0c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:33.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.258+0000 7f8c0171d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc0c87f0 con 0x7f8bfc15d2c0 2023-11-28T17:11:33.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.258+0000 7f8c0171d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc0c7750 con 0x7f8bfc0cdf70 2023-11-28T17:11:33.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.258+0000 7f8c0171d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc0c7a30 con 0x7f8bfc0d2f40 2023-11-28T17:11:33.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.259+0000 7f8bfb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc15d2c0 0x7f8bfc0c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:33.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.259+0000 7f8bfb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc15d2c0 0x7f8bfc0c9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34048/0 (socket says 172.21.15.116:34048) 2023-11-28T17:11:33.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.259+0000 7f8bfb7fe640 1 -- 172.21.15.116:0/1757346489 learned_addr learned my addr 172.21.15.116:0/1757346489 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:33.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.259+0000 7f8bfaffd640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bfc0cdf70 0x7f8bfc150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:33.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.259+0000 7f8bfa7fc640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bfc0d2f40 0x7f8bfc0c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:33.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.259+0000 7f8bfb7fe640 1 -- 172.21.15.116:0/1757346489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bfc0d2f40 msgr2=0x7f8bfc0c6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:33.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.259+0000 7f8bfb7fe640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bfc0d2f40 0x7f8bfc0c6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:33.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.259+0000 7f8bfb7fe640 1 -- 172.21.15.116:0/1757346489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bfc0cdf70 msgr2=0x7f8bfc150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:33.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.259+0000 7f8bfb7fe640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bfc0cdf70 0x7f8bfc150b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:33.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.259+0000 7f8bfb7fe640 1 -- 172.21.15.116:0/1757346489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf40094d0 con 0x7f8bfc15d2c0 2023-11-28T17:11:33.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.259+0000 7f8bfa7fc640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bfc0d2f40 0x7f8bfc0c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:11:33.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.259+0000 7f8bfaffd640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bfc0cdf70 0x7f8bfc150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:11:33.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.259+0000 7f8bfb7fe640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc15d2c0 0x7f8bfc0c9040 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009870 tx=0x7f8bf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:33.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.260+0000 7f8bf37fe640 1 -- 172.21.15.116:0/1757346489 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf403a930 con 0x7f8bfc15d2c0 2023-11-28T17:11:33.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.260+0000 7f8bf37fe640 1 -- 172.21.15.116:0/1757346489 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf403aab0 con 0x7f8bfc15d2c0 2023-11-28T17:11:33.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.260+0000 7f8bf37fe640 1 -- 172.21.15.116:0/1757346489 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf403ae60 con 0x7f8bfc15d2c0 2023-11-28T17:11:33.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.260+0000 7f8c0171d640 1 -- 172.21.15.116:0/1757346489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bfc0c7d10 con 0x7f8bfc15d2c0 2023-11-28T17:11:33.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.260+0000 7f8c0171d640 1 -- 172.21.15.116:0/1757346489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bfc0c82d0 con 0x7f8bfc15d2c0 2023-11-28T17:11:33.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.261+0000 7f8c0171d640 1 -- 172.21.15.116:0/1757346489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bfc0693d0 con 0x7f8bfc15d2c0 2023-11-28T17:11:33.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.262+0000 7f8bf37fe640 1 -- 172.21.15.116:0/1757346489 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f8bf402f050 con 0x7f8bfc15d2c0 2023-11-28T17:11:33.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.263+0000 7f8bf37fe640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8bdc06b2a0 0x7f8bdc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.263+0000 7f8bf37fe640 1 -- 172.21.15.116:0/1757346489 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8bf40abaa0 con 0x7f8bfc15d2c0 2023-11-28T17:11:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.263+0000 7f8bfaffd640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8bdc06b2a0 0x7f8bdc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.264+0000 7f8bfaffd640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8bdc06b2a0 0x7f8bdc06d760 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009af0 tx=0x7f8bec008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.267+0000 7f8bf37fe640 1 -- 172.21.15.116:0/1757346489 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8bf4076870 con 0x7f8bfc15d2c0 2023-11-28T17:11:33.499 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.497+0000 7f8c0171d640 1 -- 172.21.15.116:0/1757346489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8bfc0c9f60 con 0x7f8bfc15d2c0 2023-11-28T17:11:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.498+0000 7f8bf37fe640 1 -- 172.21.15.116:0/1757346489 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8bf4035020 con 0x7f8bfc15d2c0 2023-11-28T17:11:33.590 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-28T17:11:33.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.500+0000 7f8c0171d640 1 -- 172.21.15.116:0/1757346489 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8bdc06b2a0 msgr2=0x7f8bdc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:33.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.500+0000 7f8c0171d640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8bdc06b2a0 0x7f8bdc06d760 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8bec009af0 tx=0x7f8bec008040 comp rx=0 tx=0).stop 2023-11-28T17:11:33.591 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.501+0000 7f8c0171d640 1 -- 172.21.15.116:0/1757346489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc15d2c0 msgr2=0x7f8bfc0c9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.501+0000 7f8c0171d640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc15d2c0 0x7f8bfc0c9040 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009870 tx=0x7f8bf40046c0 comp rx=0 tx=0).stop 2023-11-28T17:11:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.501+0000 7f8c0171d640 1 -- 172.21.15.116:0/1757346489 shutdown_connections 2023-11-28T17:11:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.501+0000 7f8c0171d640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8bdc06b2a0 0x7f8bdc06d760 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.501+0000 7f8c0171d640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bfc15d2c0 0x7f8bfc0c9040 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.501+0000 7f8c0171d640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bfc0d2f40 0x7f8bfc0c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.501+0000 7f8c0171d640 1 --2- 172.21.15.116:0/1757346489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bfc0cdf70 0x7f8bfc150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.501+0000 7f8c0171d640 1 -- 172.21.15.116:0/1757346489 >> 172.21.15.116:0/1757346489 conn(0x7f8bfc072010 msgr2=0x7f8bfc15fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.501+0000 7f8c0171d640 1 -- 172.21.15.116:0/1757346489 shutdown_connections 2023-11-28T17:11:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:33.501+0000 7f8c0171d640 1 -- 172.21.15.116:0/1757346489 wait complete. 2023-11-28T17:11:33.592 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-28T17:11:33.592 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-28T17:11:33.593 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi156.front.sepia.ceph.com 2023-11-28T17:11:33.593 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-28T17:11:33.593 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-28T17:11:33.593 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-28T17:11:33.593 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-28T17:11:33.593 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-28T17:11:33.593 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-28T17:11:33.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:33.593 DEBUG:teuthology.orchestra.run.smithi156:> ip addr 2023-11-28T17:11:33.653 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-28T17:11:33.653 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-28T17:11:33.654 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2023-11-28T17:11:33.654 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:11:33.654 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2023-11-28T17:11:33.654 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:11:33.654 INFO:teuthology.orchestra.run.smithi156.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:11:33.654 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c0 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:11:33.654 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f0 2023-11-28T17:11:33.654 INFO:teuthology.orchestra.run.smithi156.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:11:33.654 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c1 brd ff:ff:ff:ff:ff:ff 2023-11-28T17:11:33.654 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f1 2023-11-28T17:11:33.654 INFO:teuthology.orchestra.run.smithi156.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-28T17:11:33.654 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fc brd ff:ff:ff:ff:ff:ff 2023-11-28T17:11:33.655 INFO:teuthology.orchestra.run.smithi156.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-28T17:11:33.655 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fd brd ff:ff:ff:ff:ff:ff 2023-11-28T17:11:33.655 INFO:teuthology.orchestra.run.smithi156.stdout: inet 172.21.15.156/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-28T17:11:33.655 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft 42242sec preferred_lft 42242sec 2023-11-28T17:11:33.655 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::ec4:7aff:fe88:6dfd/64 scope link 2023-11-28T17:11:33.655 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:11:33.655 INFO:teuthology.orchestra.run.smithi156.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-28T17:11:33.655 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 4a:53:60:71:42:cd brd ff:ff:ff:ff:ff:ff 2023-11-28T17:11:33.655 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-28T17:11:33.655 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:11:33.655 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::54f0:12ff:fe79:1e2c/64 scope link 2023-11-28T17:11:33.655 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:11:33.655 INFO:teuthology.orchestra.run.smithi156.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T17:11:33.655 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether e6:68:69:2b:d0:94 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:11:33.656 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::e468:69ff:fe2b:d094/64 scope link 2023-11-28T17:11:33.656 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:11:33.656 INFO:teuthology.orchestra.run.smithi156.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-28T17:11:33.656 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 4a:53:60:71:42:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:11:33.656 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::4853:60ff:fe71:42cd/64 scope link 2023-11-28T17:11:33.656 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2023-11-28T17:11:33.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:33.657 DEBUG:teuthology.orchestra.run.smithi156:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-28T17:11:33.753 INFO:teuthology.orchestra.run.smithi156.stdout:1 2023-11-28T17:11:33.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:33.757 DEBUG:teuthology.orchestra.run.smithi156:> route 2023-11-28T17:11:33.808 INFO:teuthology.orchestra.run.smithi156.stdout:Kernel IP routing table 2023-11-28T17:11:33.809 INFO:teuthology.orchestra.run.smithi156.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-28T17:11:33.809 INFO:teuthology.orchestra.run.smithi156.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-28T17:11:33.809 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-28T17:11:33.809 INFO:teuthology.orchestra.run.smithi156.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-28T17:11:33.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:33.809 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec bash -c ' 2023-11-28T17:11:33.809 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2023-11-28T17:11:33.809 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-28T17:11:33.809 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-28T17:11:33.810 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-28T17:11:33.810 DEBUG:teuthology.orchestra.run.smithi156:> ') 2023-11-28T17:11:33.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:33.995 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2023-11-28T17:11:34.047 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-28T17:11:34.048 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-28T17:11:34.049 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-28T17:11:34.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:34.049 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set brx.0 up 2023-11-28T17:11:34.146 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-11-28T17:11:34.146 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:11:34.146 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-28T17:11:34.198 INFO:teuthology.orchestra.run.smithi156.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-28T17:11:34.198 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-28T17:11:34.198 DEBUG:teuthology.orchestra.run.smithi156:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-28T17:11:34.251 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2023-11-28T17:11:34.346 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:11:34.404 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2023-11-28T17:11:34.404 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2023-11-28T17:11:34.404 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2023-11-28T17:11:34.404 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2023-11-28T17:11:34.405 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2023-11-28T17:11:34.405 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2023-11-28T17:11:34.405 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2023-11-28T17:11:34.405 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2023-11-28T17:11:34.405 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2023-11-28T17:11:34.405 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2023-11-28T17:11:34.405 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2023-11-28T17:11:34.405 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:11:34.405 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2023-11-28T17:11:34.405 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2023-11-28T17:11:34.405 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2023-11-28T17:11:34.405 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2023-11-28T17:11:34.406 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2023-11-28T17:11:34.406 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2023-11-28T17:11:34.406 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:11:34.406 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:11:34.406 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:11:34.406 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:11:34.406 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2023-11-28T17:11:34.406 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2023-11-28T17:11:34.406 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2023-11-28T17:11:34.406 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2023-11-28T17:11:34.406 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2023-11-28T17:11:34.406 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2023-11-28T17:11:34.407 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2023-11-28T17:11:34.407 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:11:34.407 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:11:34.407 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:11:34.407 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:11:34.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:34.408 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2023-11-28T17:11:34.463 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-11-28T17:11:34.464 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-11-28T17:11:34.469 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2023-11-28T17:11:34.549 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2023-11-28T17:11:34.592 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2023-11-28T17:11:34.592 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2023-11-28T17:11:34.593 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2023-11-28T17:11:34.593 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2023-11-28T17:11:34.593 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2023-11-28T17:11:34.593 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2023-11-28T17:11:34.593 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2023-11-28T17:11:34.593 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2023-11-28T17:11:34.593 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2023-11-28T17:11:34.593 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2023-11-28T17:11:34.593 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2023-11-28T17:11:34.593 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2023-11-28T17:11:34.593 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2023-11-28T17:11:34.593 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2023-11-28T17:11:34.593 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2023-11-28T17:11:34.594 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2023-11-28T17:11:34.594 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2023-11-28T17:11:34.594 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2023-11-28T17:11:34.594 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2023-11-28T17:11:34.594 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-11-28T17:11:34.594 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-11-28T17:11:34.594 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2023-11-28T17:11:34.594 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2023-11-28T17:11:34.594 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2023-11-28T17:11:34.594 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2023-11-28T17:11:34.594 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2023-11-28T17:11:34.594 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2023-11-28T17:11:34.594 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2023-11-28T17:11:34.595 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2023-11-28T17:11:34.595 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:11:34.595 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:11:34.595 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:11:34.595 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:11:34.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:34.596 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2023-11-28T17:11:34.655 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2023-11-28T17:11:34.654+0000 7f75a12b0480 -1 init, newargv = 0x560eaa91f8d0 newargc=15 2023-11-28T17:11:34.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[95090]: starting ceph client 2023-11-28T17:11:34.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[95090]: starting fuse 2023-11-28T17:11:34.682 INFO:teuthology.orchestra.run.smithi156.stdout:39 2023-11-28T17:11:34.682 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2023-11-28T17:11:34.682 DEBUG:teuthology.orchestra.run.smithi156:> sudo stdin-killer -- python3 -c ' 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> import re 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> import os 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> def _find_admin_socket(client_name): 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2023-11-28T17:11:34.683 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:11:34.684 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2023-11-28T17:11:34.684 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-11-28T17:11:34.684 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2023-11-28T17:11:34.684 DEBUG:teuthology.orchestra.run.smithi156:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-11-28T17:11:34.684 DEBUG:teuthology.orchestra.run.smithi156:> contents = proc_f.read() 2023-11-28T17:11:34.684 DEBUG:teuthology.orchestra.run.smithi156:> if mountpoint in contents: 2023-11-28T17:11:34.684 DEBUG:teuthology.orchestra.run.smithi156:> return f 2023-11-28T17:11:34.684 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-11-28T17:11:34.684 DEBUG:teuthology.orchestra.run.smithi156:> 2023-11-28T17:11:34.684 DEBUG:teuthology.orchestra.run.smithi156:> print(_find_admin_socket("client.1")) 2023-11-28T17:11:34.684 DEBUG:teuthology.orchestra.run.smithi156:> ' 2023-11-28T17:11:34.887 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.95090.asok 2023-11-28T17:11:34.891 INFO:teuthology.orchestra.run.smithi156.stderr:2023-11-28T17:11:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-11-28T17:11:34.904 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.95090.asok 2023-11-28T17:11:34.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:34.904 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.95090.asok status 2023-11-28T17:11:35.143 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-11-28T17:11:35.143 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2023-11-28T17:11:35.144 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "47c1c547372ed5bff7984647dc200bac0e019caf", 2023-11-28T17:11:35.144 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev)", 2023-11-28T17:11:35.144 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2023-11-28T17:11:35.144 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2023-11-28T17:11:35.144 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-11-28T17:11:35.144 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "95090", 2023-11-28T17:11:35.144 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2023-11-28T17:11:35.144 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:11:35.144 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2023-11-28T17:11:35.144 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2023-11-28T17:11:35.144 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 5056, 2023-11-28T17:11:35.145 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2023-11-28T17:11:35.145 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2023-11-28T17:11:35.145 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2023-11-28T17:11:35.145 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 5056 2023-11-28T17:11:35.145 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:11:35.145 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2023-11-28T17:11:35.145 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2023-11-28T17:11:35.145 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2023-11-28T17:11:35.145 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 405152349 2023-11-28T17:11:35.145 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-11-28T17:11:35.145 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:11:35.146 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2023-11-28T17:11:35.146 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2023-11-28T17:11:35.146 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2023-11-28T17:11:35.146 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 405152349 2023-11-28T17:11:35.146 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-11-28T17:11:35.146 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.5056 192.168.0.1:0/405152349", 2023-11-28T17:11:35.146 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "192.168.0.1:0/405152349", 2023-11-28T17:11:35.146 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2023-11-28T17:11:35.146 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 24, 2023-11-28T17:11:35.146 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 48, 2023-11-28T17:11:35.146 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2023-11-28T17:11:35.146 INFO:teuthology.orchestra.run.smithi156.stdout: "blocklisted": false, 2023-11-28T17:11:35.147 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2023-11-28T17:11:35.147 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-11-28T17:11:35.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:35.161 DEBUG:teuthology.orchestra.run.smithi156:> stat --file-system '--printf=%T 2023-11-28T17:11:35.161 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1 2023-11-28T17:11:35.211 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2023-11-28T17:11:35.211 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-11-28T17:11:35.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:35.212 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-28T17:11:35.302 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2023-11-28T17:11:35.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.528+0000 7f131b0da640 1 Processor -- start 2023-11-28T17:11:35.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.528+0000 7f131b0da640 1 -- start start 2023-11-28T17:11:35.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.530+0000 7f131b0da640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f131413cee0 0x7f131406c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:35.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.530+0000 7f131b0da640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f131415ae70 0x7f13140cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:35.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.530+0000 7f131b0da640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131415b290 con 0x7f131406c770 2023-11-28T17:11:35.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.530+0000 7f131b0da640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f131415e5d0 con 0x7f131415ae70 2023-11-28T17:11:35.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.530+0000 7f131b0da640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f131406cc20 con 0x7f131413cee0 2023-11-28T17:11:35.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.530+0000 7f1319677640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f131415ae70 0x7f13140cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:35.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.530+0000 7f1319677640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f131415ae70 0x7f13140cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34056/0 (socket says 172.21.15.116:34056) 2023-11-28T17:11:35.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.530+0000 7f1319677640 1 -- 172.21.15.116:0/2531017129 learned_addr learned my addr 172.21.15.116:0/2531017129 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.530+0000 7f1318e76640 1 --2- 172.21.15.116:0/2531017129 >> v2:172.21.15.156:3300/0 conn(0x7f131413cee0 0x7f131406c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.531+0000 7f1319677640 1 -- 172.21.15.116:0/2531017129 >> v2:172.21.15.156:3300/0 conn(0x7f131413cee0 msgr2=0x7f131406c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.531+0000 7f1319677640 1 --2- 172.21.15.116:0/2531017129 >> v2:172.21.15.156:3300/0 conn(0x7f131413cee0 0x7f131406c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.531+0000 7f1319677640 1 -- 172.21.15.116:0/2531017129 >> v1:172.21.15.156:6789/0 conn(0x7f131406c770 legacy=0x7f131415a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.531+0000 7f1319677640 1 -- 172.21.15.116:0/2531017129 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131415b4d0 con 0x7f131415ae70 2023-11-28T17:11:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.531+0000 7f1318e76640 1 --2- 172.21.15.116:0/2531017129 >> v2:172.21.15.156:3300/0 conn(0x7f131413cee0 0x7f131406c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:35.533 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.532+0000 7f1319677640 1 --2- 172.21.15.116:0/2531017129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131415ae70 0x7f13140cd840 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f13140ceea0 tx=0x7f130c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be04dc646a31f2a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:35.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.532+0000 7f13137fe640 1 -- 172.21.15.116:0/2531017129 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c02c9f0 con 0x7f131415ae70 2023-11-28T17:11:35.535 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.532+0000 7f13137fe640 1 -- 172.21.15.116:0/2531017129 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130c02cb50 con 0x7f131415ae70 2023-11-28T17:11:35.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.532+0000 7f13137fe640 1 -- 172.21.15.116:0/2531017129 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c0343f0 con 0x7f131415ae70 2023-11-28T17:11:35.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.533+0000 7f131b0da640 1 -- 172.21.15.116:0/2531017129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131415ae70 msgr2=0x7f13140cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:35.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.533+0000 7f131b0da640 1 --2- 172.21.15.116:0/2531017129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131415ae70 0x7f13140cd840 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f13140ceea0 tx=0x7f130c02be50 comp rx=0 tx=0).stop 2023-11-28T17:11:35.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.533+0000 7f131b0da640 1 -- 172.21.15.116:0/2531017129 shutdown_connections 2023-11-28T17:11:35.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.533+0000 7f131b0da640 1 --2- 172.21.15.116:0/2531017129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131415ae70 0x7f13140cd840 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:35.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.533+0000 7f131b0da640 1 --2- 172.21.15.116:0/2531017129 >> v2:172.21.15.156:3300/0 conn(0x7f131413cee0 0x7f131406c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:35.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.533+0000 7f131b0da640 1 -- 172.21.15.116:0/2531017129 >> 172.21.15.116:0/2531017129 conn(0x7f131406a4c0 msgr2=0x7f131406a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:35.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.533+0000 7f131b0da640 1 -- 172.21.15.116:0/2531017129 shutdown_connections 2023-11-28T17:11:35.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.533+0000 7f131b0da640 1 -- 172.21.15.116:0/2531017129 wait complete. 2023-11-28T17:11:35.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.535+0000 7f131b0da640 1 Processor -- start 2023-11-28T17:11:35.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.535+0000 7f131b0da640 1 -- start start 2023-11-28T17:11:35.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.536+0000 7f131b0da640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f131406c770 0x7f13141f5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:35.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.536+0000 7f131b0da640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f131413cee0 0x7f13140a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:35.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.536+0000 7f131b0da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131415ae70 0x7f1314158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:35.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.536+0000 7f131b0da640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13141529f0 con 0x7f131415ae70 2023-11-28T17:11:35.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.536+0000 7f131b0da640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1314152cd0 con 0x7f131413cee0 2023-11-28T17:11:35.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.536+0000 7f131b0da640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1314152fb0 con 0x7f131406c770 2023-11-28T17:11:35.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.536+0000 7f1319677640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131415ae70 0x7f1314158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:35.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.536+0000 7f1319677640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131415ae70 0x7f1314158fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34070/0 (socket says 172.21.15.116:34070) 2023-11-28T17:11:35.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.536+0000 7f1319677640 1 -- 172.21.15.116:0/4187498032 learned_addr learned my addr 172.21.15.116:0/4187498032 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:35.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.536+0000 7f1319677640 1 -- 172.21.15.116:0/4187498032 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f131406c770 msgr2=0x7f13141f5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:35.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.536+0000 7f1319677640 1 --2- 172.21.15.116:0/4187498032 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f131406c770 0x7f13141f5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:35.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.536+0000 7f1319677640 1 -- 172.21.15.116:0/4187498032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f131413cee0 msgr2=0x7f13140a8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:35.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.536+0000 7f1319677640 1 --2- 172.21.15.116:0/4187498032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f131413cee0 0x7f13140a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:35.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.536+0000 7f1319677640 1 -- 172.21.15.116:0/4187498032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c009b30 con 0x7f131415ae70 2023-11-28T17:11:35.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.537+0000 7f1319677640 1 --2- 172.21.15.116:0/4187498032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131415ae70 0x7f1314158fd0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f130c036040 tx=0x7f130c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:35.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.537+0000 7f13117fa640 1 -- 172.21.15.116:0/4187498032 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c03e450 con 0x7f131415ae70 2023-11-28T17:11:35.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.537+0000 7f13117fa640 1 -- 172.21.15.116:0/4187498032 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130c03e5d0 con 0x7f131415ae70 2023-11-28T17:11:35.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.537+0000 7f131b0da640 1 -- 172.21.15.116:0/4187498032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1314153290 con 0x7f131415ae70 2023-11-28T17:11:35.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.537+0000 7f13117fa640 1 -- 172.21.15.116:0/4187498032 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c03e980 con 0x7f131415ae70 2023-11-28T17:11:35.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.537+0000 7f131b0da640 1 -- 172.21.15.116:0/4187498032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13140a89a0 con 0x7f131415ae70 2023-11-28T17:11:35.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.538+0000 7f131b0da640 1 -- 172.21.15.116:0/4187498032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13141537b0 con 0x7f131415ae70 2023-11-28T17:11:35.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.539+0000 7f13117fa640 1 -- 172.21.15.116:0/4187498032 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f130c03eae0 con 0x7f131415ae70 2023-11-28T17:11:35.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.540+0000 7f13117fa640 1 --2- 172.21.15.116:0/4187498032 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f12f806b2d0 0x7f12f806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:35.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.540+0000 7f13117fa640 1 -- 172.21.15.116:0/4187498032 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f130c0ac5e0 con 0x7f131415ae70 2023-11-28T17:11:35.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.540+0000 7f1318e76640 1 --2- 172.21.15.116:0/4187498032 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f12f806b2d0 0x7f12f806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:35.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.541+0000 7f1318e76640 1 --2- 172.21.15.116:0/4187498032 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f12f806b2d0 0x7f12f806d790 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1308006fd0 tx=0x7f1308008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:35.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.545+0000 7f13117fa640 1 -- 172.21.15.116:0/4187498032 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13141537b0 con 0x7f131415ae70 2023-11-28T17:11:35.775 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:35.773+0000 7f131b0da640 1 -- 172.21.15.116:0/4187498032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f131415b290 con 0x7f131415ae70 2023-11-28T17:11:36.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.021+0000 7f13117fa640 1 -- 172.21.15.116:0/4187498032 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v25) v1 ==== 105+0+0 (secure 0 0 0) 0x7f131415b290 con 0x7f131415ae70 2023-11-28T17:11:36.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.025+0000 7f131b0da640 1 -- 172.21.15.116:0/4187498032 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f12f806b2d0 msgr2=0x7f12f806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.025+0000 7f131b0da640 1 --2- 172.21.15.116:0/4187498032 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f12f806b2d0 0x7f12f806d790 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1308006fd0 tx=0x7f1308008040 comp rx=0 tx=0).stop 2023-11-28T17:11:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.025+0000 7f131b0da640 1 -- 172.21.15.116:0/4187498032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131415ae70 msgr2=0x7f1314158fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.025+0000 7f131b0da640 1 --2- 172.21.15.116:0/4187498032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131415ae70 0x7f1314158fd0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f130c036040 tx=0x7f130c004a40 comp rx=0 tx=0).stop 2023-11-28T17:11:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.025+0000 7f131b0da640 1 -- 172.21.15.116:0/4187498032 shutdown_connections 2023-11-28T17:11:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.025+0000 7f131b0da640 1 --2- 172.21.15.116:0/4187498032 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f12f806b2d0 0x7f12f806d790 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.025+0000 7f131b0da640 1 --2- 172.21.15.116:0/4187498032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f131415ae70 0x7f1314158fd0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.025+0000 7f131b0da640 1 --2- 172.21.15.116:0/4187498032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f131413cee0 0x7f13140a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.025+0000 7f131b0da640 1 --2- 172.21.15.116:0/4187498032 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f131406c770 0x7f13141f5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.025+0000 7f131b0da640 1 -- 172.21.15.116:0/4187498032 >> 172.21.15.116:0/4187498032 conn(0x7f131406a4c0 msgr2=0x7f1314070ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.025+0000 7f131b0da640 1 -- 172.21.15.116:0/4187498032 shutdown_connections 2023-11-28T17:11:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.025+0000 7f131b0da640 1 -- 172.21.15.116:0/4187498032 wait complete. 2023-11-28T17:11:36.050 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:11:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.231+0000 7fc553078640 1 Processor -- start 2023-11-28T17:11:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.231+0000 7fc553078640 1 -- start start 2023-11-28T17:11:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.233+0000 7fc553078640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc54c159f30 0x7fc54c0701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.233+0000 7fc553078640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54c0cd8e0 con 0x7fc54c0cf2a0 2023-11-28T17:11:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.233+0000 7fc553078640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54c15b2f0 con 0x7fc54c15ae90 2023-11-28T17:11:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.233+0000 7fc553078640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc54c070730 con 0x7fc54c159f30 2023-11-28T17:11:36.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.233+0000 7fc552877640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc54c0cf2a0 0x7fc54c0b1f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45292/0 (socket says 172.21.15.116:45292) 2023-11-28T17:11:36.235 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.233+0000 7fc552877640 1 -- 172.21.15.116:0/3214154875 learned_addr learned my addr 172.21.15.116:0/3214154875 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:36.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc552076640 1 --2- 172.21.15.116:0/3214154875 >> v2:172.21.15.156:3300/0 conn(0x7fc54c159f30 0x7fc54c0701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:36.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 -- 172.21.15.116:0/3214154875 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485660006 0 0) 0x7fc54c0cd8e0 con 0x7fc54c0cf2a0 2023-11-28T17:11:36.236 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 -- 172.21.15.116:0/3214154875 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc538003520 con 0x7fc54c0cf2a0 2023-11-28T17:11:36.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 -- 172.21.15.116:0/3214154875 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 71547630 0 0) 0x7fc538003520 con 0x7fc54c0cf2a0 2023-11-28T17:11:36.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 -- 172.21.15.116:0/3214154875 >> v2:172.21.15.156:3300/0 conn(0x7fc54c159f30 msgr2=0x7fc54c0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:36.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 --2- 172.21.15.116:0/3214154875 >> v2:172.21.15.156:3300/0 conn(0x7fc54c159f30 0x7fc54c0701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 -- 172.21.15.116:0/3214154875 >> v1:172.21.15.156:6789/0 conn(0x7fc54c15ae90 legacy=0x7fc54c0ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:36.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 -- 172.21.15.116:0/3214154875 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c070920 con 0x7fc54c0cf2a0 2023-11-28T17:11:36.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 -- 172.21.15.116:0/3214154875 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fc5440019f0 con 0x7fc54c0cf2a0 2023-11-28T17:11:36.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 -- 172.21.15.116:0/3214154875 >> v1:172.21.15.116:6789/0 conn(0x7fc54c0cf2a0 legacy=0x7fc54c0b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:36.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 --2- 172.21.15.116:0/3214154875 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc538003970 0x7fc538003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:36.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 --2- 172.21.15.116:0/3214154875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5380047c0 0x7fc538004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:36.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 --2- 172.21.15.116:0/3214154875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc538005180 0x7fc53800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:36.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 -- 172.21.15.116:0/3214154875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53800e200 con 0x7fc538005180 2023-11-28T17:11:36.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 -- 172.21.15.116:0/3214154875 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53800e370 con 0x7fc5380047c0 2023-11-28T17:11:36.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.234+0000 7fc551074640 1 -- 172.21.15.116:0/3214154875 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc53800e530 con 0x7fc538003970 2023-11-28T17:11:36.237 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.235+0000 7fc552877640 1 --2- 172.21.15.116:0/3214154875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc538005180 0x7fc53800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:36.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.235+0000 7fc552076640 1 --2- 172.21.15.116:0/3214154875 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc538003970 0x7fc538003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:36.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.235+0000 7fc552877640 1 -- 172.21.15.116:0/3214154875 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc538003970 msgr2=0x7fc538003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:36.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.235+0000 7fc552877640 1 --2- 172.21.15.116:0/3214154875 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc538003970 0x7fc538003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.235+0000 7fc552877640 1 -- 172.21.15.116:0/3214154875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5380047c0 msgr2=0x7fc538004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:36.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.235+0000 7fc552877640 1 --2- 172.21.15.116:0/3214154875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5380047c0 0x7fc538004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.235+0000 7fc552877640 1 -- 172.21.15.116:0/3214154875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc538003520 con 0x7fc538005180 2023-11-28T17:11:36.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.236+0000 7fc552877640 1 -- 172.21.15.116:0/3214154875 reap_dead start 2023-11-28T17:11:36.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.236+0000 7fc552877640 1 --2- 172.21.15.116:0/3214154875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc538005180 0x7fc53800db90 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc54400a830 tx=0x7fc5380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb9e040097fac98 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:36.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.236+0000 7fc551074640 1 -- 172.21.15.116:0/3214154875 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5440072d0 con 0x7fc538005180 2023-11-28T17:11:36.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.236+0000 7fc551074640 1 -- 172.21.15.116:0/3214154875 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54402f070 con 0x7fc538005180 2023-11-28T17:11:36.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.236+0000 7fc551074640 1 -- 172.21.15.116:0/3214154875 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5440076b0 con 0x7fc538005180 2023-11-28T17:11:36.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.237+0000 7fc553078640 1 -- 172.21.15.116:0/3214154875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc538005180 msgr2=0x7fc53800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:36.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.237+0000 7fc553078640 1 --2- 172.21.15.116:0/3214154875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc538005180 0x7fc53800db90 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc54400a830 tx=0x7fc5380057f0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.237+0000 7fc553078640 1 -- 172.21.15.116:0/3214154875 shutdown_connections 2023-11-28T17:11:36.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.237+0000 7fc553078640 1 --2- 172.21.15.116:0/3214154875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc538005180 0x7fc53800db90 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.237+0000 7fc553078640 1 -- 172.21.15.116:0/3214154875 >> 172.21.15.116:0/3214154875 conn(0x7fc54c0b0950 msgr2=0x7fc54c0b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:36.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.237+0000 7fc553078640 1 -- 172.21.15.116:0/3214154875 shutdown_connections 2023-11-28T17:11:36.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.237+0000 7fc553078640 1 -- 172.21.15.116:0/3214154875 wait complete. 2023-11-28T17:11:36.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.239+0000 7fc553078640 1 Processor -- start 2023-11-28T17:11:36.240 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.239+0000 7fc553078640 1 -- start start 2023-11-28T17:11:36.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.240+0000 7fc553078640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc538005180 0x7fc54c09bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:36.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.240+0000 7fc553078640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc54c155320 0x7fc54c1413a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:36.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.240+0000 7fc553078640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc54c143d60 0x7fc54c1441e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:36.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.240+0000 7fc553078640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c144720 con 0x7fc538005180 2023-11-28T17:11:36.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.240+0000 7fc553078640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c09b500 con 0x7fc54c143d60 2023-11-28T17:11:36.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.240+0000 7fc553078640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c15b4e0 con 0x7fc54c155320 2023-11-28T17:11:36.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.240+0000 7fc552076640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc538005180 0x7fc54c09bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:36.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.240+0000 7fc552076640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc538005180 0x7fc54c09bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34080/0 (socket says 172.21.15.116:34080) 2023-11-28T17:11:36.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.240+0000 7fc552076640 1 -- 172.21.15.116:0/4003487560 learned_addr learned my addr 172.21.15.116:0/4003487560 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:36.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.240+0000 7fc552076640 1 -- 172.21.15.116:0/4003487560 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc54c155320 msgr2=0x7fc54c1413a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:36.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.240+0000 7fc552076640 1 --2- 172.21.15.116:0/4003487560 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc54c155320 0x7fc54c1413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.240+0000 7fc552076640 1 -- 172.21.15.116:0/4003487560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc54c143d60 msgr2=0x7fc54c1441e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.240+0000 7fc552076640 1 --2- 172.21.15.116:0/4003487560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc54c143d60 0x7fc54c1441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.240+0000 7fc552076640 1 -- 172.21.15.116:0/4003487560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c15b7c0 con 0x7fc538005180 2023-11-28T17:11:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.241+0000 7fc552076640 1 --2- 172.21.15.116:0/4003487560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc538005180 0x7fc54c09bf70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fc548011570 tx=0x7fc548011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.241+0000 7fc542ffd640 1 -- 172.21.15.116:0/4003487560 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc548015930 con 0x7fc538005180 2023-11-28T17:11:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.241+0000 7fc542ffd640 1 -- 172.21.15.116:0/4003487560 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc548015a90 con 0x7fc538005180 2023-11-28T17:11:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.241+0000 7fc542ffd640 1 -- 172.21.15.116:0/4003487560 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc548015e60 con 0x7fc538005180 2023-11-28T17:11:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.241+0000 7fc553078640 1 -- 172.21.15.116:0/4003487560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54c09b770 con 0x7fc538005180 2023-11-28T17:11:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.241+0000 7fc553078640 1 -- 172.21.15.116:0/4003487560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54c1591a0 con 0x7fc538005180 2023-11-28T17:11:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.242+0000 7fc553078640 1 -- 172.21.15.116:0/4003487560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc54c09d0b0 con 0x7fc538005180 2023-11-28T17:11:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.243+0000 7fc542ffd640 1 -- 172.21.15.116:0/4003487560 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fc54801d400 con 0x7fc538005180 2023-11-28T17:11:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.244+0000 7fc542ffd640 1 --2- 172.21.15.116:0/4003487560 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc53006b2d0 0x7fc53006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.245+0000 7fc542ffd640 1 -- 172.21.15.116:0/4003487560 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc54808a3b0 con 0x7fc538005180 2023-11-28T17:11:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.245+0000 7fc551875640 1 --2- 172.21.15.116:0/4003487560 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc53006b2d0 0x7fc53006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.245+0000 7fc551875640 1 --2- 172.21.15.116:0/4003487560 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc53006b2d0 0x7fc53006d790 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc54c06e070 tx=0x7fc53c0079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:36.250 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.249+0000 7fc542ffd640 1 -- 172.21.15.116:0/4003487560 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc548055180 con 0x7fc538005180 2023-11-28T17:11:36.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.460+0000 7fc553078640 1 -- 172.21.15.116:0/4003487560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc54c04e960 con 0x7fc538005180 2023-11-28T17:11:36.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.461+0000 7fc542ffd640 1 -- 172.21.15.116:0/4003487560 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4938 (secure 0 0 0) 0x7fc54805c360 con 0x7fc538005180 2023-11-28T17:11:36.463 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:36.463 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4790,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2474839888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2474839888},{"type":"v1","addr":"172.21.15.156:6835","nonce":2474839888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":4987,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/372407661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":372407661},{"type":"v1","addr":"172.21.15.156:6833","nonce":372407661}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:11:23.668388+0000","modified":"2023-11-28T17:11:36.019248+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5826,"mds_1":5823},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5823":{"gid":5823,"name":"a","rank":1,"incarnation":25,"state":"up:creating","state_seq":1,"addr":"172.21.15.116:6835/3251347229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3251347229},{"type":"v1","addr":"172.21.15.116:6835","nonce":3251347229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5826":{"gid":5826,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6837/1482499383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1482499383},{"type":"v1","addr":"172.21.15.116:6837","nonce":1482499383}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T17:11:36.463 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 25 2023-11-28T17:11:36.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.464+0000 7fc553078640 1 -- 172.21.15.116:0/4003487560 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc53006b2d0 msgr2=0x7fc53006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:36.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.464+0000 7fc553078640 1 --2- 172.21.15.116:0/4003487560 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc53006b2d0 0x7fc53006d790 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc54c06e070 tx=0x7fc53c0079b0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.464+0000 7fc553078640 1 -- 172.21.15.116:0/4003487560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc538005180 msgr2=0x7fc54c09bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:36.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.464+0000 7fc553078640 1 --2- 172.21.15.116:0/4003487560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc538005180 0x7fc54c09bf70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fc548011570 tx=0x7fc548011a30 comp rx=0 tx=0).stop 2023-11-28T17:11:36.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.464+0000 7fc553078640 1 -- 172.21.15.116:0/4003487560 shutdown_connections 2023-11-28T17:11:36.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.464+0000 7fc553078640 1 --2- 172.21.15.116:0/4003487560 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fc53006b2d0 0x7fc53006d790 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.464+0000 7fc553078640 1 --2- 172.21.15.116:0/4003487560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc54c143d60 0x7fc54c1441e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.464+0000 7fc553078640 1 --2- 172.21.15.116:0/4003487560 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc54c155320 0x7fc54c1413a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.464+0000 7fc553078640 1 --2- 172.21.15.116:0/4003487560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc538005180 0x7fc54c09bf70 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:36.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.464+0000 7fc553078640 1 -- 172.21.15.116:0/4003487560 >> 172.21.15.116:0/4003487560 conn(0x7fc54c0b0950 msgr2=0x7fc54c06bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:36.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.464+0000 7fc553078640 1 -- 172.21.15.116:0/4003487560 shutdown_connections 2023-11-28T17:11:36.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:36.464+0000 7fc553078640 1 -- 172.21.15.116:0/4003487560 wait complete. 2023-11-28T17:11:36.885 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T17:11:23.668388+0000', 'modified': '2023-11-28T17:11:36.019248+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 5826, 'mds_1': 5823}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5823': {'gid': 5823, 'name': 'a', 'rank': 1, 'incarnation': 25, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.116:6835/3251347229', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6834', 'nonce': 3251347229}, {'type': 'v1', 'addr': '172.21.15.116:6835', 'nonce': 3251347229}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5826': {'gid': 5826, 'name': 'c', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.116:6837/1482499383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6836', 'nonce': 1482499383}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 1482499383}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T17:11:36.886 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5823:up:creating 2023-11-28T17:11:37.887 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:11:38.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.110+0000 7f54bf53e640 1 Processor -- start 2023-11-28T17:11:38.112 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.110+0000 7f54bf53e640 1 -- start start 2023-11-28T17:11:38.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.112+0000 7f54bf53e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54b806c960 0x7f54b815a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:38.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.112+0000 7f54bf53e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f54b8140200 0x7f54b815a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:38.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.112+0000 7f54bf53e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b80aea20 con 0x7f54b813cee0 2023-11-28T17:11:38.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.112+0000 7f54bf53e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b813d2c0 con 0x7f54b8140200 2023-11-28T17:11:38.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.112+0000 7f54bf53e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b815b3a0 con 0x7f54b806c960 2023-11-28T17:11:38.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.112+0000 7f54bd2da640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f54b813cee0 0x7f54b806c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:35076/0 (socket says 172.21.15.116:35076) 2023-11-28T17:11:38.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.112+0000 7f54bd2da640 1 -- 172.21.15.116:0/2690319815 learned_addr learned my addr 172.21.15.116:0/2690319815 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:11:38.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.112+0000 7f54bdadb640 1 --2- 172.21.15.116:0/2690319815 >> v2:172.21.15.116:3300/0 conn(0x7f54b8140200 0x7f54b815a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:38.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.113+0000 7f54bcad9640 1 --2- 172.21.15.116:0/2690319815 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54b806c960 0x7f54b815a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:38.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.113+0000 7f54bdadb640 1 -- 172.21.15.116:0/2690319815 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54b806c960 msgr2=0x7f54b815a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:38.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.113+0000 7f54bdadb640 1 --2- 172.21.15.116:0/2690319815 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54b806c960 0x7f54b815a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:38.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.113+0000 7f54affff640 1 -- 172.21.15.116:0/2690319815 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152783382 0 0) 0x7f54b80aea20 con 0x7f54b813cee0 2023-11-28T17:11:38.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.113+0000 7f54bdadb640 1 -- 172.21.15.116:0/2690319815 >> v1:172.21.15.116:6789/0 conn(0x7f54b813cee0 legacy=0x7f54b806c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:38.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.113+0000 7f54bdadb640 1 -- 172.21.15.116:0/2690319815 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b815b0c0 con 0x7f54b8140200 2023-11-28T17:11:38.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.114+0000 7f54bdadb640 1 --2- 172.21.15.116:0/2690319815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54b8140200 0x7f54b815a9d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009d90 tx=0x7f54b002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e8a6ab828dae0a server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:38.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.114+0000 7f54affff640 1 -- 172.21.15.116:0/2690319815 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b002cbc0 con 0x7f54b8140200 2023-11-28T17:11:38.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.114+0000 7f54affff640 1 -- 172.21.15.116:0/2690319815 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b002cd20 con 0x7f54b8140200 2023-11-28T17:11:38.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.114+0000 7f54affff640 1 -- 172.21.15.116:0/2690319815 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0034660 con 0x7f54b8140200 2023-11-28T17:11:38.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.115+0000 7f54bf53e640 1 -- 172.21.15.116:0/2690319815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54b8140200 msgr2=0x7f54b815a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:38.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.115+0000 7f54bf53e640 1 --2- 172.21.15.116:0/2690319815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54b8140200 0x7f54b815a9d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009d90 tx=0x7f54b002be10 comp rx=0 tx=0).stop 2023-11-28T17:11:38.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.115+0000 7f54bf53e640 1 -- 172.21.15.116:0/2690319815 shutdown_connections 2023-11-28T17:11:38.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.115+0000 7f54bf53e640 1 --2- 172.21.15.116:0/2690319815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54b8140200 0x7f54b815a9d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:38.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.115+0000 7f54bf53e640 1 --2- 172.21.15.116:0/2690319815 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54b806c960 0x7f54b815a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:38.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.115+0000 7f54bf53e640 1 -- 172.21.15.116:0/2690319815 >> 172.21.15.116:0/2690319815 conn(0x7f54b806a4c0 msgr2=0x7f54b806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:38.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.115+0000 7f54bf53e640 1 -- 172.21.15.116:0/2690319815 shutdown_connections 2023-11-28T17:11:38.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.115+0000 7f54bf53e640 1 -- 172.21.15.116:0/2690319815 wait complete. 2023-11-28T17:11:38.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.117+0000 7f54bf53e640 1 Processor -- start 2023-11-28T17:11:38.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.117+0000 7f54bf53e640 1 -- start start 2023-11-28T17:11:38.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.118+0000 7f54bf53e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54b806c960 0x7f54b813d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:38.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.118+0000 7f54bf53e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54b813cee0 0x7f54b8146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:38.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.118+0000 7f54bf53e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54b8140200 0x7f54b81360e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:38.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.118+0000 7f54bf53e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b813da40 con 0x7f54b813cee0 2023-11-28T17:11:38.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.118+0000 7f54bf53e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b813dd20 con 0x7f54b806c960 2023-11-28T17:11:38.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.118+0000 7f54bf53e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b813e000 con 0x7f54b8140200 2023-11-28T17:11:38.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.118+0000 7f54bcad9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54b813cee0 0x7f54b8146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:38.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.118+0000 7f54bcad9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54b813cee0 0x7f54b8146fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53030/0 (socket says 172.21.15.116:53030) 2023-11-28T17:11:38.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.118+0000 7f54bcad9640 1 -- 172.21.15.116:0/2859417283 learned_addr learned my addr 172.21.15.116:0/2859417283 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:38.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.118+0000 7f54bcad9640 1 -- 172.21.15.116:0/2859417283 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54b8140200 msgr2=0x7f54b81360e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:38.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.118+0000 7f54bcad9640 1 --2- 172.21.15.116:0/2859417283 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54b8140200 0x7f54b81360e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:38.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.118+0000 7f54bcad9640 1 -- 172.21.15.116:0/2859417283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54b806c960 msgr2=0x7f54b813d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:38.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.118+0000 7f54bcad9640 1 --2- 172.21.15.116:0/2859417283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54b806c960 0x7f54b813d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:38.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.118+0000 7f54bcad9640 1 -- 172.21.15.116:0/2859417283 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b0009a70 con 0x7f54b813cee0 2023-11-28T17:11:38.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.119+0000 7f54bcad9640 1 --2- 172.21.15.116:0/2859417283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54b813cee0 0x7f54b8146fd0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f54a800d800 tx=0x7f54a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:38.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.119+0000 7f54adffb640 1 -- 172.21.15.116:0/2859417283 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a800b6c0 con 0x7f54b813cee0 2023-11-28T17:11:38.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.119+0000 7f54adffb640 1 -- 172.21.15.116:0/2859417283 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a800b820 con 0x7f54b813cee0 2023-11-28T17:11:38.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.119+0000 7f54adffb640 1 -- 172.21.15.116:0/2859417283 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a800bbb0 con 0x7f54b813cee0 2023-11-28T17:11:38.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.119+0000 7f54bf53e640 1 -- 172.21.15.116:0/2859417283 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b813e2e0 con 0x7f54b813cee0 2023-11-28T17:11:38.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.119+0000 7f54bf53e640 1 -- 172.21.15.116:0/2859417283 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b8147510 con 0x7f54b813cee0 2023-11-28T17:11:38.123 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.120+0000 7f54bf53e640 1 -- 172.21.15.116:0/2859417283 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b80703d0 con 0x7f54b813cee0 2023-11-28T17:11:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.121+0000 7f54adffb640 1 -- 172.21.15.116:0/2859417283 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f54a800bd10 con 0x7f54b813cee0 2023-11-28T17:11:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.122+0000 7f54adffb640 1 --2- 172.21.15.116:0/2859417283 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f549c06b2a0 0x7f549c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.122+0000 7f54adffb640 1 -- 172.21.15.116:0/2859417283 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54a808a210 con 0x7f54b813cee0 2023-11-28T17:11:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.122+0000 7f54bd2da640 1 --2- 172.21.15.116:0/2859417283 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f549c06b2a0 0x7f549c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.123+0000 7f54bd2da640 1 --2- 172.21.15.116:0/2859417283 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f549c06b2a0 0x7f549c06d760 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f54b4007fd0 tx=0x7f54b4007920 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.126+0000 7f54adffb640 1 -- 172.21.15.116:0/2859417283 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f54b80703d0 con 0x7f54b813cee0 2023-11-28T17:11:38.355 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.354+0000 7f54bf53e640 1 -- 172.21.15.116:0/2859417283 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54b80703d0 con 0x7f54b813cee0 2023-11-28T17:11:38.357 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.355+0000 7f54adffb640 1 -- 172.21.15.116:0/2859417283 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4936 (secure 0 0 0) 0x7f54b80703d0 con 0x7f54b813cee0 2023-11-28T17:11:38.357 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:11:38.357 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4790,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/2474839888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2474839888},{"type":"v1","addr":"172.21.15.156:6835","nonce":2474839888}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":4987,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/372407661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":372407661},{"type":"v1","addr":"172.21.15.156:6833","nonce":372407661}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:11:23.668388+0000","modified":"2023-11-28T17:11:36.056114+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5826,"mds_1":5823},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5823":{"gid":5823,"name":"a","rank":1,"incarnation":25,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/3251347229","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3251347229},{"type":"v1","addr":"172.21.15.116:6835","nonce":3251347229}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5826":{"gid":5826,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6837/1482499383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1482499383},{"type":"v1","addr":"172.21.15.116:6837","nonce":1482499383}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T17:11:38.357 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 26 2023-11-28T17:11:38.359 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.358+0000 7f54bf53e640 1 -- 172.21.15.116:0/2859417283 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f549c06b2a0 msgr2=0x7f549c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:38.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.358+0000 7f54bf53e640 1 --2- 172.21.15.116:0/2859417283 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f549c06b2a0 0x7f549c06d760 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f54b4007fd0 tx=0x7f54b4007920 comp rx=0 tx=0).stop 2023-11-28T17:11:38.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.358+0000 7f54bf53e640 1 -- 172.21.15.116:0/2859417283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54b813cee0 msgr2=0x7f54b8146fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:38.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.358+0000 7f54bf53e640 1 --2- 172.21.15.116:0/2859417283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54b813cee0 0x7f54b8146fd0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f54a800d800 tx=0x7f54a800dcc0 comp rx=0 tx=0).stop 2023-11-28T17:11:38.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.358+0000 7f54bf53e640 1 -- 172.21.15.116:0/2859417283 shutdown_connections 2023-11-28T17:11:38.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.358+0000 7f54bf53e640 1 --2- 172.21.15.116:0/2859417283 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f549c06b2a0 0x7f549c06d760 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:38.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.358+0000 7f54bf53e640 1 --2- 172.21.15.116:0/2859417283 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f54b8140200 0x7f54b81360e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:38.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.358+0000 7f54bf53e640 1 --2- 172.21.15.116:0/2859417283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54b813cee0 0x7f54b8146fd0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:38.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.358+0000 7f54bf53e640 1 --2- 172.21.15.116:0/2859417283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54b806c960 0x7f54b813d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:38.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.358+0000 7f54bf53e640 1 -- 172.21.15.116:0/2859417283 >> 172.21.15.116:0/2859417283 conn(0x7f54b806a4c0 msgr2=0x7f54b8139330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:38.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.358+0000 7f54bf53e640 1 -- 172.21.15.116:0/2859417283 shutdown_connections 2023-11-28T17:11:38.360 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:38.358+0000 7f54bf53e640 1 -- 172.21.15.116:0/2859417283 wait complete. 2023-11-28T17:11:38.781 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-28T17:11:23.668388+0000', 'modified': '2023-11-28T17:11:36.056114+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 5826, 'mds_1': 5823}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5823': {'gid': 5823, 'name': 'a', 'rank': 1, 'incarnation': 25, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.116:6835/3251347229', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6834', 'nonce': 3251347229}, {'type': 'v1', 'addr': '172.21.15.116:6835', 'nonce': 3251347229}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5826': {'gid': 5826, 'name': 'c', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.116:6837/1482499383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6836', 'nonce': 1482499383}, {'type': 'v1', 'addr': '172.21.15.116:6837', 'nonce': 1482499383}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-28T17:11:38.782 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2023-11-28T17:11:38.782 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-11-28T17:11:39.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.002+0000 7fa09570f640 1 Processor -- start 2023-11-28T17:11:39.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.002+0000 7fa09570f640 1 -- start start 2023-11-28T17:11:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.004+0000 7fa09570f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa09006c960 0x7fa09015a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.004+0000 7fa09570f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0900aea20 con 0x7fa09013cee0 2023-11-28T17:11:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.004+0000 7fa09570f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa09013d2c0 con 0x7fa090140200 2023-11-28T17:11:39.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.004+0000 7fa09570f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa09015b3a0 con 0x7fa09006c960 2023-11-28T17:11:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.004+0000 7fa08e7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa09006c960 0x7fa09015a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.004+0000 7fa08e7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa09006c960 0x7fa09015a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53038/0 (socket says 172.21.15.116:53038) 2023-11-28T17:11:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.004+0000 7fa08effd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa09013cee0 0x7fa09006c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:35090/0 (socket says 172.21.15.116:35090) 2023-11-28T17:11:39.006 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.004+0000 7fa08e7fc640 1 -- 172.21.15.116:0/527057024 learned_addr learned my addr 172.21.15.116:0/527057024 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.004+0000 7fa08e7fc640 1 -- 172.21.15.116:0/527057024 >> v1:172.21.15.156:6789/0 conn(0x7fa090140200 legacy=0x7fa0900cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.004+0000 7fa08e7fc640 1 -- 172.21.15.116:0/527057024 >> v1:172.21.15.116:6789/0 conn(0x7fa09013cee0 legacy=0x7fa09006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.004+0000 7fa08e7fc640 1 -- 172.21.15.116:0/527057024 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09015b0c0 con 0x7fa09006c960 2023-11-28T17:11:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.006+0000 7fa08e7fc640 1 --2- 172.21.15.116:0/527057024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa09006c960 0x7fa09015a490 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa084009870 tx=0x7fa08402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf73db39a0aede57 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:39.007 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.006+0000 7fa08dffb640 1 -- 172.21.15.116:0/527057024 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa084031870 con 0x7fa09006c960 2023-11-28T17:11:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.006+0000 7fa08dffb640 1 -- 172.21.15.116:0/527057024 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0840319d0 con 0x7fa09006c960 2023-11-28T17:11:39.008 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.006+0000 7fa08dffb640 1 -- 172.21.15.116:0/527057024 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa084031d60 con 0x7fa09006c960 2023-11-28T17:11:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.006+0000 7fa09570f640 1 -- 172.21.15.116:0/527057024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa09006c960 msgr2=0x7fa09015a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.006+0000 7fa09570f640 1 --2- 172.21.15.116:0/527057024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa09006c960 0x7fa09015a490 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa084009870 tx=0x7fa08402b910 comp rx=0 tx=0).stop 2023-11-28T17:11:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.007+0000 7fa09570f640 1 -- 172.21.15.116:0/527057024 shutdown_connections 2023-11-28T17:11:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.007+0000 7fa09570f640 1 --2- 172.21.15.116:0/527057024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa09006c960 0x7fa09015a490 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.007+0000 7fa09570f640 1 -- 172.21.15.116:0/527057024 >> 172.21.15.116:0/527057024 conn(0x7fa09006a4c0 msgr2=0x7fa09006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.007+0000 7fa09570f640 1 -- 172.21.15.116:0/527057024 shutdown_connections 2023-11-28T17:11:39.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.007+0000 7fa09570f640 1 -- 172.21.15.116:0/527057024 wait complete. 2023-11-28T17:11:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.008+0000 7fa09570f640 1 Processor -- start 2023-11-28T17:11:39.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.009+0000 7fa09570f640 1 -- start start 2023-11-28T17:11:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.009+0000 7fa09570f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa09006c960 0x7fa090136830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.010+0000 7fa09570f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa09013cee0 0x7fa090071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.010+0000 7fa09570f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa090140200 0x7fa0901f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.010+0000 7fa09570f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09014f440 con 0x7fa09006c960 2023-11-28T17:11:39.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.010+0000 7fa09570f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa090136d70 con 0x7fa090140200 2023-11-28T17:11:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.010+0000 7fa09570f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0900703f0 con 0x7fa09013cee0 2023-11-28T17:11:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.010+0000 7fa08effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa09006c960 0x7fa090136830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.010+0000 7fa08effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa09006c960 0x7fa090136830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53048/0 (socket says 172.21.15.116:53048) 2023-11-28T17:11:39.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.010+0000 7fa08effd640 1 -- 172.21.15.116:0/4082459676 learned_addr learned my addr 172.21.15.116:0/4082459676 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.010+0000 7fa08effd640 1 -- 172.21.15.116:0/4082459676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa09013cee0 msgr2=0x7fa090071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.010+0000 7fa08effd640 1 --2- 172.21.15.116:0/4082459676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa09013cee0 0x7fa090071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.010+0000 7fa08effd640 1 -- 172.21.15.116:0/4082459676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa090140200 msgr2=0x7fa0901f2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.010+0000 7fa08effd640 1 --2- 172.21.15.116:0/4082459676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa090140200 0x7fa0901f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.010+0000 7fa08effd640 1 -- 172.21.15.116:0/4082459676 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0840094d0 con 0x7fa09006c960 2023-11-28T17:11:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.011+0000 7fa08effd640 1 --2- 172.21.15.116:0/4082459676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa09006c960 0x7fa090136830 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fa08000c4d0 tx=0x7fa08000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.011+0000 7fa0777fe640 1 -- 172.21.15.116:0/4082459676 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa080007480 con 0x7fa09006c960 2023-11-28T17:11:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.011+0000 7fa0777fe640 1 -- 172.21.15.116:0/4082459676 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0800075e0 con 0x7fa09006c960 2023-11-28T17:11:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.011+0000 7fa0777fe640 1 -- 172.21.15.116:0/4082459676 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0800049e0 con 0x7fa09006c960 2023-11-28T17:11:39.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.011+0000 7fa09570f640 1 -- 172.21.15.116:0/4082459676 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09014f160 con 0x7fa09006c960 2023-11-28T17:11:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.011+0000 7fa09570f640 1 -- 172.21.15.116:0/4082459676 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa090070920 con 0x7fa09006c960 2023-11-28T17:11:39.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.013+0000 7fa0777fe640 1 -- 172.21.15.116:0/4082459676 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa080004b40 con 0x7fa09006c960 2023-11-28T17:11:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.015+0000 7fa0777fe640 1 --2- 172.21.15.116:0/4082459676 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa07c06b2a0 0x7fa07c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.015+0000 7fa0777fe640 1 -- 172.21.15.116:0/4082459676 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa080088bd0 con 0x7fa09006c960 2023-11-28T17:11:39.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.015+0000 7fa08e7fc640 1 --2- 172.21.15.116:0/4082459676 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa07c06b2a0 0x7fa07c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.015+0000 7fa09837a740 1 Processor -- start 2023-11-28T17:11:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.016+0000 7fa09837a740 1 -- start start 2023-11-28T17:11:39.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.016+0000 7fa08e7fc640 1 --2- 172.21.15.116:0/4082459676 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa07c06b2a0 0x7fa07c06d760 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa084036000 tx=0x7fa084030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.016+0000 7fa09837a740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf3fae10 0x55ecaf3fb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.016+0000 7fa09837a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3cb0c0 0x55ecaf3f8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.016+0000 7fa09837a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf3ce990 0x55ecaf3f85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.016+0000 7fa09837a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecaf32f880 con 0x55ecaf3cb0c0 2023-11-28T17:11:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.016+0000 7fa09837a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecaf33eae0 con 0x55ecaf3ce990 2023-11-28T17:11:39.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.016+0000 7fa09837a740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ecaf33c1d0 con 0x55ecaf3fae10 2023-11-28T17:11:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.017+0000 7fa08e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3cb0c0 0x55ecaf3f8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.017+0000 7fa08e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3cb0c0 0x55ecaf3f8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53052/0 (socket says 172.21.15.116:53052) 2023-11-28T17:11:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.017+0000 7fa08e7fc640 1 -- 172.21.15.116:0/2238106349 learned_addr learned my addr 172.21.15.116:0/2238106349 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.017+0000 7fa08effd640 1 --2- 172.21.15.116:0/2238106349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf3fae10 0x55ecaf3fb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.017+0000 7fa08f7fe640 1 --2- 172.21.15.116:0/2238106349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf3ce990 0x55ecaf3f85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.017+0000 7fa08e7fc640 1 -- 172.21.15.116:0/2238106349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf3fae10 msgr2=0x55ecaf3fb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.017+0000 7fa08e7fc640 1 --2- 172.21.15.116:0/2238106349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf3fae10 0x55ecaf3fb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.017+0000 7fa08e7fc640 1 -- 172.21.15.116:0/2238106349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf3ce990 msgr2=0x55ecaf3f85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.017+0000 7fa08e7fc640 1 --2- 172.21.15.116:0/2238106349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf3ce990 0x55ecaf3f85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.017+0000 7fa08e7fc640 1 -- 172.21.15.116:0/2238106349 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecaf2ff860 con 0x55ecaf3cb0c0 2023-11-28T17:11:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.017+0000 7fa08effd640 1 --2- 172.21.15.116:0/2238106349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf3fae10 0x55ecaf3fb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.018+0000 7fa08e7fc640 1 --2- 172.21.15.116:0/2238106349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3cb0c0 0x55ecaf3f8080 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fa08402bf20 tx=0x7fa084031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5df0f257f3aa7c65 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.018+0000 7fa074ff9640 1 -- 172.21.15.116:0/2238106349 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08403a970 con 0x55ecaf3cb0c0 2023-11-28T17:11:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.018+0000 7fa074ff9640 1 -- 172.21.15.116:0/2238106349 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08404a070 con 0x55ecaf3cb0c0 2023-11-28T17:11:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.018+0000 7fa074ff9640 1 -- 172.21.15.116:0/2238106349 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08403acf0 con 0x55ecaf3cb0c0 2023-11-28T17:11:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.018+0000 7fa09837a740 1 -- 172.21.15.116:0/2238106349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3cb0c0 msgr2=0x55ecaf3f8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.018+0000 7fa09837a740 1 --2- 172.21.15.116:0/2238106349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3cb0c0 0x55ecaf3f8080 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fa08402bf20 tx=0x7fa084031000 comp rx=0 tx=0).stop 2023-11-28T17:11:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.018+0000 7fa09837a740 1 -- 172.21.15.116:0/2238106349 shutdown_connections 2023-11-28T17:11:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.018+0000 7fa09837a740 1 --2- 172.21.15.116:0/2238106349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf3ce990 0x55ecaf3f85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.018+0000 7fa09837a740 1 --2- 172.21.15.116:0/2238106349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3cb0c0 0x55ecaf3f8080 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.018+0000 7fa09837a740 1 --2- 172.21.15.116:0/2238106349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf3fae10 0x55ecaf3fb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.019+0000 7fa09837a740 1 -- 172.21.15.116:0/2238106349 >> 172.21.15.116:0/2238106349 conn(0x55ecaf3cd330 msgr2=0x55ecaf415c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.019+0000 7fa09837a740 1 -- 172.21.15.116:0/2238106349 shutdown_connections 2023-11-28T17:11:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.019+0000 7fa09837a740 1 -- 172.21.15.116:0/2238106349 wait complete. 2023-11-28T17:11:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.019+0000 7fa09837a740 1 -- start start 2023-11-28T17:11:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.019+0000 7fa09837a740 1 Processor -- start 2023-11-28T17:11:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.020+0000 7fa09837a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3ce990 0x55ecaf40d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.020+0000 7fa09837a740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf3fae10 0x55ecaf40e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.020+0000 7fa09837a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf4291c0 0x55ecaf40ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.021+0000 7fa09837a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecaf337ac0 con 0x55ecaf3ce990 2023-11-28T17:11:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.021+0000 7fa09837a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecaf2d42f0 con 0x55ecaf4291c0 2023-11-28T17:11:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.021+0000 7fa09837a740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ecaf2f1c70 con 0x55ecaf3fae10 2023-11-28T17:11:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.021+0000 7fa09837a740 10 client.0 _ll_register_callbacks cb 0x55ecaf049ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:11:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.021+0000 7fa08effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3ce990 0x55ecaf40d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.021+0000 7fa08effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3ce990 0x55ecaf40d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53056/0 (socket says 172.21.15.116:53056) 2023-11-28T17:11:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.021+0000 7fa08effd640 1 -- 172.21.15.116:0/3074517632 learned_addr learned my addr 172.21.15.116:0/3074517632 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.021+0000 7fa08effd640 1 -- 172.21.15.116:0/3074517632 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf3fae10 msgr2=0x55ecaf40e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.027 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.021+0000 7fa08effd640 1 --2- 172.21.15.116:0/3074517632 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf3fae10 0x55ecaf40e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.021+0000 7fa08effd640 1 -- 172.21.15.116:0/3074517632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf4291c0 msgr2=0x55ecaf40ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.021+0000 7fa08effd640 1 --2- 172.21.15.116:0/3074517632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf4291c0 0x55ecaf40ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.021+0000 7fa08effd640 1 -- 172.21.15.116:0/3074517632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa084040070 con 0x55ecaf3ce990 2023-11-28T17:11:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.021+0000 7fa08effd640 1 --2- 172.21.15.116:0/3074517632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3ce990 0x55ecaf40d0c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa080001eb0 tx=0x7fa08005f9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3f43bdf3f68dd2 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.021+0000 7fa06dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-28T17:11:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.022+0000 7fa06dffb640 1 -- 172.21.15.116:0/3074517632 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa080056e90 con 0x55ecaf3ce990 2023-11-28T17:11:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.022+0000 7fa06dffb640 1 -- 172.21.15.116:0/3074517632 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa08005f560 con 0x55ecaf3ce990 2023-11-28T17:11:39.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.022+0000 7fa09837a740 1 -- 172.21.15.116:0/3074517632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ecaf330aa0 con 0x55ecaf3ce990 2023-11-28T17:11:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.022+0000 7fa06dffb640 1 -- 172.21.15.116:0/3074517632 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa08005f720 con 0x55ecaf3ce990 2023-11-28T17:11:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.022+0000 7fa09837a740 10 client.6036 fetch_fsmap learned FSMap version 26 2023-11-28T17:11:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.022+0000 7fa09837a740 1 -- 172.21.15.116:0/3074517632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55ecaf336140 con 0x55ecaf3ce990 2023-11-28T17:11:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.022+0000 7fa06dffb640 1 -- 172.21.15.116:0/3074517632 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ecaf330aa0 con 0x55ecaf3ce990 2023-11-28T17:11:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.022+0000 7fa06dffb640 1 -- 172.21.15.116:0/3074517632 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 26) v1 ==== 3306+0+0 (secure 0 0 0) 0x7fa080053590 con 0x55ecaf3ce990 2023-11-28T17:11:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.022+0000 7fa09837a740 10 client.6036 fetch_fsmap finished waiting for FSMap version 26 2023-11-28T17:11:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.022+0000 7fa09837a740 10 client.6036 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-28T17:11:39.029 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.022+0000 7fa09837a740 20 client.6036 populate_metadata read hostname 'smithi116' 2023-11-28T17:11:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.022+0000 7fa09837a740 1 --2- 172.21.15.116:0/3074517632 >> [v2:172.21.15.116:6834/3251347229,v1:172.21.15.116:6835/3251347229] conn(0x55ecaf429b00 0x55ecaf429ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.022+0000 7fa09837a740 4 client.6036 mds_command: new command op to 5823 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T17:11:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.022+0000 7fa09837a740 1 -- 172.21.15.116:0/3074517632 --> [v2:172.21.15.116:6834/3251347229,v1:172.21.15.116:6835/3251347229] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ecaf3307c0 con 0x55ecaf429b00 2023-11-28T17:11:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.023+0000 7fa08e7fc640 1 --2- 172.21.15.116:0/3074517632 >> [v2:172.21.15.116:6834/3251347229,v1:172.21.15.116:6835/3251347229] conn(0x55ecaf429b00 0x55ecaf429ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.023+0000 7fa08e7fc640 1 --2- 172.21.15.116:0/3074517632 >> [v2:172.21.15.116:6834/3251347229,v1:172.21.15.116:6835/3251347229] conn(0x55ecaf429b00 0x55ecaf429ee0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=963696315aa6ae6 server_cookie=e26b0c9dba69f869 in_seq=0 out_seq=0 2023-11-28T17:11:39.030 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.023+0000 7fa06dffb640 10 client.6036 ms_handle_connect on v2:172.21.15.116:6834/3251347229 2023-11-28T17:11:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.024+0000 7fa06dffb640 1 -- 172.21.15.116:0/3074517632 <== mds.1 v2:172.21.15.116:6834/3251347229 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ecaf3307c0 con 0x55ecaf429b00 2023-11-28T17:11:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.024+0000 7fa06dffb640 10 client.6036 handle_command_reply: tid=0 2023-11-28T17:11:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.024+0000 7fa09837a740 1 client.6036 shutdown 2023-11-28T17:11:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.024+0000 7fa09837a740 1 -- 172.21.15.116:0/3074517632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3ce990 msgr2=0x55ecaf40d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.024+0000 7fa09837a740 1 --2- 172.21.15.116:0/3074517632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3ce990 0x55ecaf40d0c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa080001eb0 tx=0x7fa08005f9f0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.024+0000 7fa09837a740 1 -- 172.21.15.116:0/3074517632 shutdown_connections 2023-11-28T17:11:39.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.024+0000 7fa09837a740 1 --2- 172.21.15.116:0/3074517632 >> [v2:172.21.15.116:6834/3251347229,v1:172.21.15.116:6835/3251347229] conn(0x55ecaf429b00 0x55ecaf429ee0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.024+0000 7fa09837a740 1 --2- 172.21.15.116:0/3074517632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf4291c0 0x55ecaf40ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.024+0000 7fa09837a740 1 --2- 172.21.15.116:0/3074517632 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf3fae10 0x55ecaf40e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.024+0000 7fa09837a740 1 --2- 172.21.15.116:0/3074517632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3ce990 0x55ecaf40d0c0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.024+0000 7fa09837a740 1 -- 172.21.15.116:0/3074517632 >> 172.21.15.116:0/3074517632 conn(0x55ecaf3cb0c0 msgr2=0x55ecaf419130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:39.033 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.024+0000 7fa06dffb640 0 client.6036 ms_handle_reset on v2:172.21.15.116:6834/3251347229 2023-11-28T17:11:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.024+0000 7fa09837a740 1 -- 172.21.15.116:0/3074517632 shutdown_connections 2023-11-28T17:11:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.024+0000 7fa09837a740 1 -- 172.21.15.116:0/3074517632 wait complete. 2023-11-28T17:11:39.034 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.025+0000 7fa09837a740 20 client.6036 trim_cache size 0 max 16384 2023-11-28T17:11:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.041+0000 7fa09837a740 1 Processor -- start 2023-11-28T17:11:39.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.041+0000 7fa09837a740 1 -- start start 2023-11-28T17:11:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.042+0000 7fa09837a740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf3ce990 0x55ecaf3f8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.042+0000 7fa09837a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf3fae10 0x55ecaf3ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.042+0000 7fa09837a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf4291c0 0x55ecaf3ca700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.042+0000 7fa09837a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecaf3cb8c0 con 0x55ecaf4291c0 2023-11-28T17:11:39.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.042+0000 7fa09837a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecaf09af20 con 0x55ecaf3fae10 2023-11-28T17:11:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.042+0000 7fa09837a740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ecaf3fc190 con 0x55ecaf3ce990 2023-11-28T17:11:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.043+0000 7fa08f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf4291c0 0x55ecaf3ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.043+0000 7fa08f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf4291c0 0x55ecaf3ca700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53064/0 (socket says 172.21.15.116:53064) 2023-11-28T17:11:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.043+0000 7fa08f7fe640 1 -- 172.21.15.116:0/1406289626 learned_addr learned my addr 172.21.15.116:0/1406289626 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.043+0000 7fa08f7fe640 1 -- 172.21.15.116:0/1406289626 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf3ce990 msgr2=0x55ecaf3f8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.043+0000 7fa08f7fe640 1 --2- 172.21.15.116:0/1406289626 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf3ce990 0x55ecaf3f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.043+0000 7fa08f7fe640 1 -- 172.21.15.116:0/1406289626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf3fae10 msgr2=0x55ecaf3ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.043+0000 7fa08f7fe640 1 --2- 172.21.15.116:0/1406289626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf3fae10 0x55ecaf3ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.043+0000 7fa08f7fe640 1 -- 172.21.15.116:0/1406289626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecaf2d3b80 con 0x55ecaf4291c0 2023-11-28T17:11:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.043+0000 7fa08f7fe640 1 --2- 172.21.15.116:0/1406289626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf4291c0 0x55ecaf3ca700 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fa088008ba0 tx=0x7fa08800d550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fcbc00f742dd53f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.043+0000 7fa0757fa640 1 -- 172.21.15.116:0/1406289626 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08800de60 con 0x55ecaf4291c0 2023-11-28T17:11:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.043+0000 7fa0757fa640 1 -- 172.21.15.116:0/1406289626 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa088012420 con 0x55ecaf4291c0 2023-11-28T17:11:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.043+0000 7fa0757fa640 1 -- 172.21.15.116:0/1406289626 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0880127d0 con 0x55ecaf4291c0 2023-11-28T17:11:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.044+0000 7fa09837a740 1 -- 172.21.15.116:0/1406289626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf4291c0 msgr2=0x55ecaf3ca700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.044+0000 7fa09837a740 1 --2- 172.21.15.116:0/1406289626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf4291c0 0x55ecaf3ca700 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fa088008ba0 tx=0x7fa08800d550 comp rx=0 tx=0).stop 2023-11-28T17:11:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.044+0000 7fa09837a740 1 -- 172.21.15.116:0/1406289626 shutdown_connections 2023-11-28T17:11:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.044+0000 7fa09837a740 1 --2- 172.21.15.116:0/1406289626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf4291c0 0x55ecaf3ca700 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.044+0000 7fa09837a740 1 --2- 172.21.15.116:0/1406289626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf3fae10 0x55ecaf3ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.044+0000 7fa09837a740 1 --2- 172.21.15.116:0/1406289626 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf3ce990 0x55ecaf3f8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.044+0000 7fa09837a740 1 -- 172.21.15.116:0/1406289626 >> 172.21.15.116:0/1406289626 conn(0x55ecaf3cb0c0 msgr2=0x55ecaf42ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.044+0000 7fa09837a740 1 -- 172.21.15.116:0/1406289626 shutdown_connections 2023-11-28T17:11:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.044+0000 7fa09837a740 1 -- 172.21.15.116:0/1406289626 wait complete. 2023-11-28T17:11:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.045+0000 7fa09837a740 1 -- start start 2023-11-28T17:11:39.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.045+0000 7fa09837a740 1 Processor -- start 2023-11-28T17:11:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.046+0000 7fa09837a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3ce990 0x55ecaf3ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.046+0000 7fa09837a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf3fae10 0x55ecaf3ca700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.046+0000 7fa09837a740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf4291c0 0x55ecaf424430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.046+0000 7fa09837a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecaf3f6170 con 0x55ecaf3ce990 2023-11-28T17:11:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.046+0000 7fa09837a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecaf3f75b0 con 0x55ecaf3fae10 2023-11-28T17:11:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.046+0000 7fa09837a740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ecaf4814f0 con 0x55ecaf4291c0 2023-11-28T17:11:39.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.046+0000 7fa09837a740 10 client.0 _ll_register_callbacks cb 0x7fa080053ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:11:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.046+0000 7fa08effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3ce990 0x55ecaf3ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.046+0000 7fa08effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3ce990 0x55ecaf3ca1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53070/0 (socket says 172.21.15.116:53070) 2023-11-28T17:11:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.046+0000 7fa08effd640 1 -- 172.21.15.116:0/3000484675 learned_addr learned my addr 172.21.15.116:0/3000484675 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.047+0000 7fa08effd640 1 -- 172.21.15.116:0/3000484675 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf4291c0 msgr2=0x55ecaf424430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.047+0000 7fa08effd640 1 --2- 172.21.15.116:0/3000484675 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf4291c0 0x55ecaf424430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.047+0000 7fa08e7fc640 1 --2- 172.21.15.116:0/3000484675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf3fae10 0x55ecaf3ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.047+0000 7fa08effd640 1 -- 172.21.15.116:0/3000484675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf3fae10 msgr2=0x55ecaf3ca700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.047+0000 7fa08effd640 1 --2- 172.21.15.116:0/3000484675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf3fae10 0x55ecaf3ca700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.047+0000 7fa08effd640 1 -- 172.21.15.116:0/3000484675 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa088008820 con 0x55ecaf3ce990 2023-11-28T17:11:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.047+0000 7fa08effd640 1 --2- 172.21.15.116:0/3000484675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3ce990 0x55ecaf3ca1c0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa08005fa30 tx=0x7fa080001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d9b41752161837e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.047+0000 7fa06dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-28T17:11:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.047+0000 7fa06dffb640 1 -- 172.21.15.116:0/3000484675 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08000e040 con 0x55ecaf3ce990 2023-11-28T17:11:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.047+0000 7fa06dffb640 1 -- 172.21.15.116:0/3000484675 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa080056700 con 0x55ecaf3ce990 2023-11-28T17:11:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.047+0000 7fa09837a740 1 -- 172.21.15.116:0/3000484675 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa08404a070 con 0x55ecaf3ce990 2023-11-28T17:11:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.047+0000 7fa06dffb640 1 -- 172.21.15.116:0/3000484675 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0800577a0 con 0x55ecaf3ce990 2023-11-28T17:11:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.047+0000 7fa09837a740 10 client.6042 fetch_fsmap learned FSMap version 26 2023-11-28T17:11:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.047+0000 7fa09837a740 1 -- 172.21.15.116:0/3000484675 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55ecaf2f9170 con 0x55ecaf3ce990 2023-11-28T17:11:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.048+0000 7fa06dffb640 1 -- 172.21.15.116:0/3000484675 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08007b020 con 0x55ecaf3ce990 2023-11-28T17:11:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.048+0000 7fa06dffb640 1 -- 172.21.15.116:0/3000484675 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 26) v1 ==== 3306+0+0 (secure 0 0 0) 0x7fa0800568c0 con 0x55ecaf3ce990 2023-11-28T17:11:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.048+0000 7fa09837a740 10 client.6042 fetch_fsmap finished waiting for FSMap version 26 2023-11-28T17:11:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.048+0000 7fa09837a740 10 client.6042 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-28T17:11:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.048+0000 7fa09837a740 20 client.6042 populate_metadata read hostname 'smithi116' 2023-11-28T17:11:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.048+0000 7fa09837a740 1 --2- 172.21.15.116:0/3000484675 >> [v2:172.21.15.116:6834/3251347229,v1:172.21.15.116:6835/3251347229] conn(0x55ecaf4806a0 0x55ecaf424970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.048+0000 7fa09837a740 4 client.6042 mds_command: new command op to 5823 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T17:11:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.048+0000 7fa09837a740 1 -- 172.21.15.116:0/3000484675 --> [v2:172.21.15.116:6834/3251347229,v1:172.21.15.116:6835/3251347229] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ecaf0386c0 con 0x55ecaf4806a0 2023-11-28T17:11:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.048+0000 7fa08e7fc640 1 --2- 172.21.15.116:0/3000484675 >> [v2:172.21.15.116:6834/3251347229,v1:172.21.15.116:6835/3251347229] conn(0x55ecaf4806a0 0x55ecaf424970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.049+0000 7fa06dffb640 10 client.6042 ms_handle_connect on v2:172.21.15.116:6834/3251347229 2023-11-28T17:11:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.049+0000 7fa08e7fc640 1 --2- 172.21.15.116:0/3000484675 >> [v2:172.21.15.116:6834/3251347229,v1:172.21.15.116:6835/3251347229] conn(0x55ecaf4806a0 0x55ecaf424970 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ed4bb728d6a0b265 server_cookie=2cd56a712b2948f3 in_seq=0 out_seq=0 2023-11-28T17:11:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.049+0000 7fa06dffb640 1 -- 172.21.15.116:0/3000484675 <== mds.1 v2:172.21.15.116:6834/3251347229 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55ecaf0386c0 con 0x55ecaf4806a0 2023-11-28T17:11:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.049+0000 7fa06dffb640 10 client.6042 handle_command_reply: tid=0 2023-11-28T17:11:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.049+0000 7fa09837a740 1 client.6042 shutdown 2023-11-28T17:11:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.049+0000 7fa09837a740 1 -- 172.21.15.116:0/3000484675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3ce990 msgr2=0x55ecaf3ca1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.049+0000 7fa09837a740 1 --2- 172.21.15.116:0/3000484675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3ce990 0x55ecaf3ca1c0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa08005fa30 tx=0x7fa080001120 comp rx=0 tx=0).stop 2023-11-28T17:11:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.049+0000 7fa09837a740 1 -- 172.21.15.116:0/3000484675 shutdown_connections 2023-11-28T17:11:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.049+0000 7fa09837a740 1 --2- 172.21.15.116:0/3000484675 >> [v2:172.21.15.116:6834/3251347229,v1:172.21.15.116:6835/3251347229] conn(0x55ecaf4806a0 0x55ecaf424970 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.049+0000 7fa09837a740 1 --2- 172.21.15.116:0/3000484675 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x55ecaf4291c0 0x55ecaf424430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.049+0000 7fa09837a740 1 --2- 172.21.15.116:0/3000484675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ecaf3fae10 0x55ecaf3ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.049+0000 7fa09837a740 1 --2- 172.21.15.116:0/3000484675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecaf3ce990 0x55ecaf3ca1c0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.049+0000 7fa06dffb640 0 client.6042 ms_handle_reset on v2:172.21.15.116:6834/3251347229 2023-11-28T17:11:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.049+0000 7fa09837a740 1 -- 172.21.15.116:0/3000484675 >> 172.21.15.116:0/3000484675 conn(0x55ecaf3cb0c0 msgr2=0x55ecaf3d0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.050+0000 7fa09837a740 1 -- 172.21.15.116:0/3000484675 shutdown_connections 2023-11-28T17:11:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.050+0000 7fa09837a740 1 -- 172.21.15.116:0/3000484675 wait complete. 2023-11-28T17:11:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.050+0000 7fa09837a740 20 client.6042 trim_cache size 0 max 16384 2023-11-28T17:11:39.057 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-28T17:11:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a5f2fbcf-fde4-4c38-94da-08178784d9fc", 2023-11-28T17:11:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 1, 2023-11-28T17:11:39.057 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 5823, 2023-11-28T17:11:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2023-11-28T17:11:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2023-11-28T17:11:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2023-11-28T17:11:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 3.0252861439999998, 2023-11-28T17:11:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 26, 2023-11-28T17:11:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 48, 2023-11-28T17:11:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 48, 2023-11-28T17:11:39.058 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 27.780491692999998 2023-11-28T17:11:39.058 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-28T17:11:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.050+0000 7fa09570f640 1 -- 172.21.15.116:0/4082459676 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa07c06b2a0 msgr2=0x7fa07c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.050+0000 7fa09570f640 1 --2- 172.21.15.116:0/4082459676 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa07c06b2a0 0x7fa07c06d760 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa084036000 tx=0x7fa084030000 comp rx=0 tx=0).stop 2023-11-28T17:11:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.050+0000 7fa09570f640 1 -- 172.21.15.116:0/4082459676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa09006c960 msgr2=0x7fa090136830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.050+0000 7fa09570f640 1 --2- 172.21.15.116:0/4082459676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa09006c960 0x7fa090136830 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fa08000c4d0 tx=0x7fa08000c990 comp rx=0 tx=0).stop 2023-11-28T17:11:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.050+0000 7fa09570f640 1 -- 172.21.15.116:0/4082459676 shutdown_connections 2023-11-28T17:11:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.050+0000 7fa09570f640 1 --2- 172.21.15.116:0/4082459676 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa07c06b2a0 0x7fa07c06d760 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.059 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.050+0000 7fa09570f640 1 --2- 172.21.15.116:0/4082459676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa090140200 0x7fa0901f2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.050+0000 7fa09570f640 1 --2- 172.21.15.116:0/4082459676 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa09013cee0 0x7fa090071120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.050+0000 7fa09570f640 1 --2- 172.21.15.116:0/4082459676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa09006c960 0x7fa090136830 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.051+0000 7fa09570f640 1 -- 172.21.15.116:0/4082459676 >> 172.21.15.116:0/4082459676 conn(0x7fa09006a4c0 msgr2=0x7fa09006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:39.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.051+0000 7fa09570f640 1 -- 172.21.15.116:0/4082459676 shutdown_connections 2023-11-28T17:11:39.060 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.051+0000 7fa09570f640 1 -- 172.21.15.116:0/4082459676 wait complete. 2023-11-28T17:11:39.070 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-11-28T17:11:39.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.287+0000 7ff4bdf61640 1 Processor -- start 2023-11-28T17:11:39.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.287+0000 7ff4bdf61640 1 -- start start 2023-11-28T17:11:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.289+0000 7ff4bdf61640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4b815f7b0 0x7ff4b8071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.289+0000 7ff4bdf61640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff4b80721b0 0x7ff4b806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.289+0000 7ff4bdf61640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b815fc10 con 0x7ff4b8144430 2023-11-28T17:11:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.289+0000 7ff4bdf61640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4b8144810 con 0x7ff4b80721b0 2023-11-28T17:11:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.289+0000 7ff4bdf61640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b8070bd0 con 0x7ff4b815f7b0 2023-11-28T17:11:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.289+0000 7ff4b7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff4b80721b0 0x7ff4b806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.289+0000 7ff4b7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff4b80721b0 0x7ff4b806c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53082/0 (socket says 172.21.15.116:53082) 2023-11-28T17:11:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.289+0000 7ff4b7fff640 1 -- 172.21.15.116:0/4131487943 learned_addr learned my addr 172.21.15.116:0/4131487943 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.290+0000 7ff4b7fff640 1 -- 172.21.15.116:0/4131487943 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4b815f7b0 msgr2=0x7ff4b8071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.290+0000 7ff4b6ffd640 1 --2- 172.21.15.116:0/4131487943 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4b815f7b0 0x7ff4b8071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.290+0000 7ff4b7fff640 1 --2- 172.21.15.116:0/4131487943 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4b815f7b0 0x7ff4b8071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.290+0000 7ff4b7fff640 1 -- 172.21.15.116:0/4131487943 >> v1:172.21.15.116:6789/0 conn(0x7ff4b8144430 legacy=0x7ff4b80703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.290+0000 7ff4b7fff640 1 -- 172.21.15.116:0/4131487943 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b8070eb0 con 0x7ff4b80721b0 2023-11-28T17:11:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.290+0000 7ff4b6ffd640 1 --2- 172.21.15.116:0/4131487943 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4b815f7b0 0x7ff4b8071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.291+0000 7ff4b7fff640 1 --2- 172.21.15.116:0/4131487943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4b80721b0 0x7ff4b806c250 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0009e50 tx=0x7ff4b002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4b6e277b2cc0b4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.291+0000 7ff4b67fc640 1 -- 172.21.15.116:0/4131487943 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b002cb00 con 0x7ff4b80721b0 2023-11-28T17:11:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.291+0000 7ff4b67fc640 1 -- 172.21.15.116:0/4131487943 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b002cc60 con 0x7ff4b80721b0 2023-11-28T17:11:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.291+0000 7ff4b67fc640 1 -- 172.21.15.116:0/4131487943 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b00343f0 con 0x7ff4b80721b0 2023-11-28T17:11:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.292+0000 7ff4bdf61640 1 -- 172.21.15.116:0/4131487943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4b80721b0 msgr2=0x7ff4b806c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.292+0000 7ff4bdf61640 1 --2- 172.21.15.116:0/4131487943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4b80721b0 0x7ff4b806c250 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0009e50 tx=0x7ff4b002bf60 comp rx=0 tx=0).stop 2023-11-28T17:11:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.292+0000 7ff4bdf61640 1 -- 172.21.15.116:0/4131487943 shutdown_connections 2023-11-28T17:11:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.292+0000 7ff4bdf61640 1 --2- 172.21.15.116:0/4131487943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4b80721b0 0x7ff4b806c250 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.292+0000 7ff4bdf61640 1 --2- 172.21.15.116:0/4131487943 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4b815f7b0 0x7ff4b8071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.292+0000 7ff4bdf61640 1 -- 172.21.15.116:0/4131487943 >> 172.21.15.116:0/4131487943 conn(0x7ff4b806a4c0 msgr2=0x7ff4b806a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:39.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.292+0000 7ff4bdf61640 1 -- 172.21.15.116:0/4131487943 shutdown_connections 2023-11-28T17:11:39.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.292+0000 7ff4bdf61640 1 -- 172.21.15.116:0/4131487943 wait complete. 2023-11-28T17:11:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.294+0000 7ff4bdf61640 1 Processor -- start 2023-11-28T17:11:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.294+0000 7ff4bdf61640 1 -- start start 2023-11-28T17:11:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.295+0000 7ff4bdf61640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4b80721b0 0x7ff4b815be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.295+0000 7ff4bdf61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4b8144430 0x7ff4b81409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.295+0000 7ff4bdf61640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4b815f7b0 0x7ff4b80a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.295+0000 7ff4bdf61640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b812fa10 con 0x7ff4b8144430 2023-11-28T17:11:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.295+0000 7ff4bdf61640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b812fcf0 con 0x7ff4b815f7b0 2023-11-28T17:11:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.295+0000 7ff4bdf61640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b812ffd0 con 0x7ff4b80721b0 2023-11-28T17:11:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.295+0000 7ff4b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4b8144430 0x7ff4b81409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.295+0000 7ff4b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4b8144430 0x7ff4b81409f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53094/0 (socket says 172.21.15.116:53094) 2023-11-28T17:11:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.295+0000 7ff4b6ffd640 1 -- 172.21.15.116:0/4151850493 learned_addr learned my addr 172.21.15.116:0/4151850493 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.295+0000 7ff4b6ffd640 1 -- 172.21.15.116:0/4151850493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4b80721b0 msgr2=0x7ff4b815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.295+0000 7ff4b6ffd640 1 --2- 172.21.15.116:0/4151850493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4b80721b0 0x7ff4b815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.295+0000 7ff4b6ffd640 1 -- 172.21.15.116:0/4151850493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4b815f7b0 msgr2=0x7ff4b80a8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.295+0000 7ff4b6ffd640 1 --2- 172.21.15.116:0/4151850493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4b815f7b0 0x7ff4b80a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.295+0000 7ff4b6ffd640 1 -- 172.21.15.116:0/4151850493 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b0009b30 con 0x7ff4b8144430 2023-11-28T17:11:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.296+0000 7ff4b6ffd640 1 --2- 172.21.15.116:0/4151850493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4b8144430 0x7ff4b81409f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac00c7f0 tx=0x7ff4ac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.296+0000 7ff49ffff640 1 -- 172.21.15.116:0/4151850493 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac007820 con 0x7ff4b8144430 2023-11-28T17:11:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.296+0000 7ff49ffff640 1 -- 172.21.15.116:0/4151850493 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4ac007980 con 0x7ff4b8144430 2023-11-28T17:11:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.296+0000 7ff49ffff640 1 -- 172.21.15.116:0/4151850493 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac007cf0 con 0x7ff4b8144430 2023-11-28T17:11:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.296+0000 7ff4bdf61640 1 -- 172.21.15.116:0/4151850493 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b81302b0 con 0x7ff4b8144430 2023-11-28T17:11:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.296+0000 7ff4bdf61640 1 -- 172.21.15.116:0/4151850493 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b8140f30 con 0x7ff4b8144430 2023-11-28T17:11:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.298+0000 7ff49ffff640 1 -- 172.21.15.116:0/4151850493 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff4ac0113f0 con 0x7ff4b8144430 2023-11-28T17:11:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.299+0000 7ff49ffff640 1 --2- 172.21.15.116:0/4151850493 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff4a406b2d0 0x7ff4a406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.299+0000 7ff49ffff640 1 -- 172.21.15.116:0/4151850493 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4ac089fe0 con 0x7ff4b8144430 2023-11-28T17:11:39.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.299+0000 7ff4b77fe640 1 --2- 172.21.15.116:0/4151850493 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff4a406b2d0 0x7ff4a406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.300+0000 7ff4c0bd0740 1 Processor -- start 2023-11-28T17:11:39.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.300+0000 7ff4c0bd0740 1 -- start start 2023-11-28T17:11:39.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.300+0000 7ff4b77fe640 1 --2- 172.21.15.116:0/4151850493 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff4a406b2d0 0x7ff4a406d790 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8000f60 tx=0x7ff4a80079b0 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:39.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.301+0000 7ff4c0bd0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fbb4e10 0x56267fbb51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.301+0000 7ff4c0bd0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fb850c0 0x56267fbb2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.301+0000 7ff4c0bd0740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fb88990 0x56267fbb25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.301+0000 7ff4c0bd0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56267fae9880 con 0x56267fbb4e10 2023-11-28T17:11:39.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.301+0000 7ff4c0bd0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56267faf8ae0 con 0x56267fb850c0 2023-11-28T17:11:39.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.301+0000 7ff4c0bd0740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x56267faf61d0 con 0x56267fb88990 2023-11-28T17:11:39.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.301+0000 7ff4b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fbb4e10 0x56267fbb51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.301+0000 7ff4b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fbb4e10 0x56267fbb51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53096/0 (socket says 172.21.15.116:53096) 2023-11-28T17:11:39.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.301+0000 7ff4b77fe640 1 -- 172.21.15.116:0/3604854183 learned_addr learned my addr 172.21.15.116:0/3604854183 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:39.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.301+0000 7ff4b77fe640 1 -- 172.21.15.116:0/3604854183 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fb88990 msgr2=0x56267fbb25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.301+0000 7ff4b77fe640 1 --2- 172.21.15.116:0/3604854183 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fb88990 0x56267fbb25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.301+0000 7ff4b77fe640 1 -- 172.21.15.116:0/3604854183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fb850c0 msgr2=0x56267fbb2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.301+0000 7ff4b77fe640 1 --2- 172.21.15.116:0/3604854183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fb850c0 0x56267fbb2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.301+0000 7ff4b77fe640 1 -- 172.21.15.116:0/3604854183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56267fab9860 con 0x56267fbb4e10 2023-11-28T17:11:39.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.302+0000 7ff4b77fe640 1 --2- 172.21.15.116:0/3604854183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fbb4e10 0x56267fbb51f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff4a80166c0 tx=0x7ff4a800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0fb31db932374dc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:39.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.302+0000 7ff49d7fa640 1 -- 172.21.15.116:0/3604854183 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a801b070 con 0x56267fbb4e10 2023-11-28T17:11:39.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.302+0000 7ff49d7fa640 1 -- 172.21.15.116:0/3604854183 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a8005110 con 0x56267fbb4e10 2023-11-28T17:11:39.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.302+0000 7ff49d7fa640 1 -- 172.21.15.116:0/3604854183 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a80163a0 con 0x56267fbb4e10 2023-11-28T17:11:39.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.303+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/3604854183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fbb4e10 msgr2=0x56267fbb51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.303+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/3604854183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fbb4e10 0x56267fbb51f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff4a80166c0 tx=0x7ff4a800f000 comp rx=0 tx=0).stop 2023-11-28T17:11:39.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.303+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/3604854183 shutdown_connections 2023-11-28T17:11:39.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.303+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/3604854183 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fb88990 0x56267fbb25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.303+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/3604854183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fb850c0 0x56267fbb2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.303+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/3604854183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fbb4e10 0x56267fbb51f0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.303+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/3604854183 >> 172.21.15.116:0/3604854183 conn(0x56267fb87330 msgr2=0x56267fbcfc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:39.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.303+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/3604854183 shutdown_connections 2023-11-28T17:11:39.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.303+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/3604854183 wait complete. 2023-11-28T17:11:39.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.303+0000 7ff4c0bd0740 1 -- start start 2023-11-28T17:11:39.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.304+0000 7ff4c0bd0740 1 Processor -- start 2023-11-28T17:11:39.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4c0bd0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fb9f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4c0bd0740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fbb4e10 0x56267fbe3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4c0bd0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fbb72d0 0x56267fbb7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4c0bd0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56267faf1ac0 con 0x56267fb88990 2023-11-28T17:11:39.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4c0bd0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56267fa8e2f0 con 0x56267fbb72d0 2023-11-28T17:11:39.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4c0bd0740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x56267faabc70 con 0x56267fbb4e10 2023-11-28T17:11:39.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4c0bd0740 10 client.0 _ll_register_callbacks cb 0x56267f803ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:11:39.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fb9f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fb9f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53106/0 (socket says 172.21.15.116:53106) 2023-11-28T17:11:39.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4b77fe640 1 -- 172.21.15.116:0/1837762929 learned_addr learned my addr 172.21.15.116:0/1837762929 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:39.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4b77fe640 1 -- 172.21.15.116:0/1837762929 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fbb4e10 msgr2=0x56267fbe3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4b77fe640 1 --2- 172.21.15.116:0/1837762929 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fbb4e10 0x56267fbe3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4b77fe640 1 -- 172.21.15.116:0/1837762929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fbb72d0 msgr2=0x56267fbb7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4b77fe640 1 --2- 172.21.15.116:0/1837762929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fbb72d0 0x56267fbb7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.305+0000 7ff4b77fe640 1 -- 172.21.15.116:0/1837762929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b815fc10 con 0x56267fb88990 2023-11-28T17:11:39.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.306+0000 7ff4b77fe640 1 --2- 172.21.15.116:0/1837762929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fb9f5c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8003950 tx=0x7ff4a8001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6541910052289a1b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:39.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.306+0000 7ff4967fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-28T17:11:39.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.306+0000 7ff4967fc640 1 -- 172.21.15.116:0/1837762929 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8025070 con 0x56267fb88990 2023-11-28T17:11:39.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.306+0000 7ff4967fc640 1 -- 172.21.15.116:0/1837762929 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4a8003cc0 con 0x56267fb88990 2023-11-28T17:11:39.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.306+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/1837762929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56267faeaaa0 con 0x56267fb88990 2023-11-28T17:11:39.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.306+0000 7ff4967fc640 1 -- 172.21.15.116:0/1837762929 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4a8003e80 con 0x56267fb88990 2023-11-28T17:11:39.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.306+0000 7ff4c0bd0740 10 client.6057 fetch_fsmap learned FSMap version 26 2023-11-28T17:11:39.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.306+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/1837762929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x56267faf0140 con 0x56267fb88990 2023-11-28T17:11:39.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.306+0000 7ff4967fc640 1 -- 172.21.15.116:0/1837762929 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56267faeaaa0 con 0x56267fb88990 2023-11-28T17:11:39.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.307+0000 7ff4967fc640 1 -- 172.21.15.116:0/1837762929 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 26) v1 ==== 3306+0+0 (secure 0 0 0) 0x7ff4a800c570 con 0x56267fb88990 2023-11-28T17:11:39.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.307+0000 7ff4c0bd0740 10 client.6057 fetch_fsmap finished waiting for FSMap version 26 2023-11-28T17:11:39.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.307+0000 7ff4c0bd0740 10 client.6057 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T17:11:39.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.307+0000 7ff4c0bd0740 20 client.6057 populate_metadata read hostname 'smithi116' 2023-11-28T17:11:39.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.307+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/1837762929 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x56267fbaccb0 0x56267fbad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.307+0000 7ff4c0bd0740 4 client.6057 mds_command: new command op to 5826 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-28T17:11:39.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.307+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/1837762929 --> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56267faea7c0 con 0x56267fbaccb0 2023-11-28T17:11:39.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.307+0000 7ff4b6ffd640 1 --2- 172.21.15.116:0/1837762929 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x56267fbaccb0 0x56267fbad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.307+0000 7ff4b6ffd640 1 --2- 172.21.15.116:0/1837762929 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x56267fbaccb0 0x56267fbad090 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9de94618044ef875 server_cookie=cbb8c3e3efa9ad5d in_seq=0 out_seq=0 2023-11-28T17:11:39.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.307+0000 7ff4967fc640 10 client.6057 ms_handle_connect on v2:172.21.15.116:6836/1482499383 2023-11-28T17:11:39.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.308+0000 7ff4967fc640 1 -- 172.21.15.116:0/1837762929 <== mds.0 v2:172.21.15.116:6836/1482499383 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56267faea7c0 con 0x56267fbaccb0 2023-11-28T17:11:39.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.308+0000 7ff4967fc640 10 client.6057 handle_command_reply: tid=0 2023-11-28T17:11:39.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.308+0000 7ff4c0bd0740 1 client.6057 shutdown 2023-11-28T17:11:39.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.308+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/1837762929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 msgr2=0x56267fb9f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.308+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/1837762929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fb9f5c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8003950 tx=0x7ff4a8001190 comp rx=0 tx=0).stop 2023-11-28T17:11:39.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.308+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/1837762929 shutdown_connections 2023-11-28T17:11:39.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.308+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/1837762929 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x56267fbaccb0 0x56267fbad090 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.308+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/1837762929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fbb72d0 0x56267fbb7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.308+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/1837762929 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fbb4e10 0x56267fbe3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.308+0000 7ff4967fc640 0 client.6057 ms_handle_reset on v2:172.21.15.116:6836/1482499383 2023-11-28T17:11:39.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.308+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/1837762929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fb9f5c0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.308+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/1837762929 >> 172.21.15.116:0/1837762929 conn(0x56267fb850c0 msgr2=0x56267fbb8990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:39.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.309+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/1837762929 shutdown_connections 2023-11-28T17:11:39.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.309+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/1837762929 wait complete. 2023-11-28T17:11:39.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.309+0000 7ff4c0bd0740 20 client.6057 trim_cache size 0 max 16384 2023-11-28T17:11:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.326+0000 7ff4c0bd0740 1 Processor -- start 2023-11-28T17:11:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.326+0000 7ff4c0bd0740 1 -- start start 2023-11-28T17:11:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.327+0000 7ff4c0bd0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fbe22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.327+0000 7ff4c0bd0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fbb4e10 0x56267fbe2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.327+0000 7ff4c0bd0740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fbb72d0 0x56267fbb8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.327+0000 7ff4c0bd0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56267fb8b5a0 con 0x56267fb88990 2023-11-28T17:11:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.327+0000 7ff4c0bd0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56267f854f20 con 0x56267fbb4e10 2023-11-28T17:11:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.327+0000 7ff4c0bd0740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x56267fbe2d40 con 0x56267fbb72d0 2023-11-28T17:11:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.328+0000 7ff4b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fbe22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.328+0000 7ff4b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fbe22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53110/0 (socket says 172.21.15.116:53110) 2023-11-28T17:11:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.328+0000 7ff4b77fe640 1 -- 172.21.15.116:0/3331823118 learned_addr learned my addr 172.21.15.116:0/3331823118 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.328+0000 7ff4b6ffd640 1 --2- 172.21.15.116:0/3331823118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fbb4e10 0x56267fbe2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.328+0000 7ff4b77fe640 1 -- 172.21.15.116:0/3331823118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fbb72d0 msgr2=0x56267fbb8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.328+0000 7ff4b7fff640 1 --2- 172.21.15.116:0/3331823118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fbb72d0 0x56267fbb8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.328+0000 7ff4b77fe640 1 --2- 172.21.15.116:0/3331823118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fbb72d0 0x56267fbb8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.328+0000 7ff4b77fe640 1 -- 172.21.15.116:0/3331823118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fbb4e10 msgr2=0x56267fbe2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.328+0000 7ff4b77fe640 1 --2- 172.21.15.116:0/3331823118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fbb4e10 0x56267fbe2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.328+0000 7ff4b77fe640 1 -- 172.21.15.116:0/3331823118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56267fa8db80 con 0x56267fb88990 2023-11-28T17:11:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.328+0000 7ff4b6ffd640 1 --2- 172.21.15.116:0/3331823118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fbb4e10 0x56267fbe2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:11:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.328+0000 7ff4b77fe640 1 --2- 172.21.15.116:0/3331823118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fbe22c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8007bf0 tx=0x7ff4a8025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f02eca2162afb8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.329+0000 7ff49dffb640 1 -- 172.21.15.116:0/3331823118 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a800c0e0 con 0x56267fb88990 2023-11-28T17:11:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.329+0000 7ff49dffb640 1 -- 172.21.15.116:0/3331823118 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a800cd90 con 0x56267fb88990 2023-11-28T17:11:39.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.329+0000 7ff49dffb640 1 -- 172.21.15.116:0/3331823118 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8014420 con 0x56267fb88990 2023-11-28T17:11:39.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.329+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/3331823118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 msgr2=0x56267fbe22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.329+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/3331823118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fbe22c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8007bf0 tx=0x7ff4a8025000 comp rx=0 tx=0).stop 2023-11-28T17:11:39.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.329+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/3331823118 shutdown_connections 2023-11-28T17:11:39.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.329+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/3331823118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fbb72d0 0x56267fbb8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.329+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/3331823118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fbb4e10 0x56267fbe2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.329+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/3331823118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fbe22c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.329+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/3331823118 >> 172.21.15.116:0/3331823118 conn(0x56267fb850c0 msgr2=0x56267fb85530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:39.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.329+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/3331823118 shutdown_connections 2023-11-28T17:11:39.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.330+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/3331823118 wait complete. 2023-11-28T17:11:39.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.330+0000 7ff4c0bd0740 1 -- start start 2023-11-28T17:11:39.334 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.330+0000 7ff4c0bd0740 1 Processor -- start 2023-11-28T17:11:39.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.331+0000 7ff4c0bd0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fc397b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.331+0000 7ff4c0bd0740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fbb4e10 0x56267fc3db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.331+0000 7ff4c0bd0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fbb72d0 0x56267fbb0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.331+0000 7ff4c0bd0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x56267fbae450 con 0x56267fb88990 2023-11-28T17:11:39.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.331+0000 7ff4c0bd0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56267fbb1100 con 0x56267fbb72d0 2023-11-28T17:11:39.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.331+0000 7ff4c0bd0740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x56267fc3d350 con 0x56267fbb4e10 2023-11-28T17:11:39.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.332+0000 7ff4c0bd0740 10 client.0 _ll_register_callbacks cb 0x7ff4a8016590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-28T17:11:39.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.332+0000 7ff4b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fc397b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.332+0000 7ff4b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fc397b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53114/0 (socket says 172.21.15.116:53114) 2023-11-28T17:11:39.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.332+0000 7ff4b77fe640 1 -- 172.21.15.116:0/845011725 learned_addr learned my addr 172.21.15.116:0/845011725 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:11:39.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.332+0000 7ff4b77fe640 1 -- 172.21.15.116:0/845011725 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fbb4e10 msgr2=0x56267fc3db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.332+0000 7ff4b77fe640 1 --2- 172.21.15.116:0/845011725 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fbb4e10 0x56267fc3db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.332+0000 7ff4b77fe640 1 -- 172.21.15.116:0/845011725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fbb72d0 msgr2=0x56267fbb0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:11:39.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.332+0000 7ff4b77fe640 1 --2- 172.21.15.116:0/845011725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fbb72d0 0x56267fbb0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.332+0000 7ff4b77fe640 1 -- 172.21.15.116:0/845011725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56267faf0140 con 0x56267fb88990 2023-11-28T17:11:39.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.332+0000 7ff4b77fe640 1 --2- 172.21.15.116:0/845011725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fc397b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8001eb0 tx=0x7ff4a8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938fd1d3bff48630 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:11:39.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.332+0000 7ff4967fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2023-11-28T17:11:39.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.333+0000 7ff4967fc640 1 -- 172.21.15.116:0/845011725 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8025040 con 0x56267fb88990 2023-11-28T17:11:39.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.333+0000 7ff4967fc640 1 -- 172.21.15.116:0/845011725 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4a800c0e0 con 0x56267fb88990 2023-11-28T17:11:39.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.333+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/845011725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4a8005110 con 0x56267fb88990 2023-11-28T17:11:39.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.333+0000 7ff4967fc640 1 -- 172.21.15.116:0/845011725 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4a8014ab0 con 0x56267fb88990 2023-11-28T17:11:39.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.333+0000 7ff4c0bd0740 10 client.6063 fetch_fsmap learned FSMap version 26 2023-11-28T17:11:39.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.333+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/845011725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x56267fab3170 con 0x56267fb88990 2023-11-28T17:11:39.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.333+0000 7ff4967fc640 1 -- 172.21.15.116:0/845011725 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a8005110 con 0x56267fb88990 2023-11-28T17:11:39.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.333+0000 7ff4967fc640 1 -- 172.21.15.116:0/845011725 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 26) v1 ==== 3306+0+0 (secure 0 0 0) 0x7ff4a8014c10 con 0x56267fb88990 2023-11-28T17:11:39.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.333+0000 7ff4c0bd0740 10 client.6063 fetch_fsmap finished waiting for FSMap version 26 2023-11-28T17:11:39.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.333+0000 7ff4c0bd0740 10 client.6063 resolve_mds: resolved name 'c' to daemon mds.c 2023-11-28T17:11:39.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.333+0000 7ff4c0bd0740 20 client.6063 populate_metadata read hostname 'smithi116' 2023-11-28T17:11:39.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.333+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/845011725 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x56267fb8a6b0 0x56267fbb8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:11:39.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.333+0000 7ff4c0bd0740 4 client.6063 mds_command: new command op to 5826 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-28T17:11:39.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.333+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/845011725 --> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56267f7f26c0 con 0x56267fb8a6b0 2023-11-28T17:11:39.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.334+0000 7ff4b6ffd640 1 --2- 172.21.15.116:0/845011725 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x56267fb8a6b0 0x56267fbb8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:11:39.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.334+0000 7ff4b6ffd640 1 --2- 172.21.15.116:0/845011725 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x56267fb8a6b0 0x56267fbb8520 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fad0f49ced96dfcb server_cookie=58bc84af9e65cc7f in_seq=0 out_seq=0 2023-11-28T17:11:39.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.334+0000 7ff4967fc640 10 client.6063 ms_handle_connect on v2:172.21.15.116:6836/1482499383 2023-11-28T17:11:39.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.334+0000 7ff4967fc640 1 -- 172.21.15.116:0/845011725 <== mds.0 v2:172.21.15.116:6836/1482499383 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x56267f7f26c0 con 0x56267fb8a6b0 2023-11-28T17:11:39.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.334+0000 7ff4967fc640 10 client.6063 handle_command_reply: tid=0 2023-11-28T17:11:39.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.334+0000 7ff4c0bd0740 1 client.6063 shutdown 2023-11-28T17:11:39.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.334+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/845011725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 msgr2=0x56267fc397b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.334+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/845011725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fc397b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8001eb0 tx=0x7ff4a8012000 comp rx=0 tx=0).stop 2023-11-28T17:11:39.341 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.335+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/845011725 shutdown_connections 2023-11-28T17:11:39.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.335+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/845011725 >> [v2:172.21.15.116:6836/1482499383,v1:172.21.15.116:6837/1482499383] conn(0x56267fb8a6b0 0x56267fbb8520 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.335+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/845011725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56267fbb72d0 0x56267fbb0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.335+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/845011725 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56267fbb4e10 0x56267fc3db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.335+0000 7ff4c0bd0740 1 --2- 172.21.15.116:0/845011725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56267fb88990 0x56267fc397b0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.335+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/845011725 >> 172.21.15.116:0/845011725 conn(0x56267fb850c0 msgr2=0x56267fbd2270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:39.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.335+0000 7ff4967fc640 0 client.6063 ms_handle_reset on v2:172.21.15.116:6836/1482499383 2023-11-28T17:11:39.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.335+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/845011725 shutdown_connections 2023-11-28T17:11:39.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.335+0000 7ff4c0bd0740 1 -- 172.21.15.116:0/845011725 wait complete. 2023-11-28T17:11:39.342 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.335+0000 7ff4c0bd0740 20 client.6063 trim_cache size 0 max 16384 2023-11-28T17:11:39.342 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2023-11-28T17:11:39.343 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "a5f2fbcf-fde4-4c38-94da-08178784d9fc", 2023-11-28T17:11:39.343 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2023-11-28T17:11:39.343 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 5826, 2023-11-28T17:11:39.343 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2023-11-28T17:11:39.343 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2023-11-28T17:11:39.343 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2023-11-28T17:11:39.343 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 15.140014999, 2023-11-28T17:11:39.343 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 26, 2023-11-28T17:11:39.343 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 48, 2023-11-28T17:11:39.343 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 48, 2023-11-28T17:11:39.343 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 28.059921886000001 2023-11-28T17:11:39.344 INFO:teuthology.orchestra.run.smithi116.stdout:} 2023-11-28T17:11:39.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.335+0000 7ff4bdf61640 1 -- 172.21.15.116:0/4151850493 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff4a406b2d0 msgr2=0x7ff4a406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.335+0000 7ff4bdf61640 1 --2- 172.21.15.116:0/4151850493 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff4a406b2d0 0x7ff4a406d790 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8000f60 tx=0x7ff4a80079b0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.335+0000 7ff4bdf61640 1 -- 172.21.15.116:0/4151850493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4b8144430 msgr2=0x7ff4b81409f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:11:39.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.335+0000 7ff4bdf61640 1 --2- 172.21.15.116:0/4151850493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4b8144430 0x7ff4b81409f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac00c7f0 tx=0x7ff4ac00ccb0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.336+0000 7ff4bdf61640 1 -- 172.21.15.116:0/4151850493 shutdown_connections 2023-11-28T17:11:39.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.336+0000 7ff4bdf61640 1 --2- 172.21.15.116:0/4151850493 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff4a406b2d0 0x7ff4a406d790 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.336+0000 7ff4bdf61640 1 --2- 172.21.15.116:0/4151850493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4b815f7b0 0x7ff4b80a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.336+0000 7ff4bdf61640 1 --2- 172.21.15.116:0/4151850493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4b8144430 0x7ff4b81409f0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.336+0000 7ff4bdf61640 1 --2- 172.21.15.116:0/4151850493 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4b80721b0 0x7ff4b815be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:11:39.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.336+0000 7ff4bdf61640 1 -- 172.21.15.116:0/4151850493 >> 172.21.15.116:0/4151850493 conn(0x7ff4b806a4c0 msgr2=0x7ff4b80b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:11:39.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.336+0000 7ff4bdf61640 1 -- 172.21.15.116:0/4151850493 shutdown_connections 2023-11-28T17:11:39.346 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:11:39.336+0000 7ff4bdf61640 1 -- 172.21.15.116:0/4151850493 wait complete. 2023-11-28T17:11:39.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:39.354 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'mkdir {d0,d1} && touch {d0,d1}/file') 2023-11-28T17:11:39.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:39.425 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 d0) 2023-11-28T17:11:39.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:39.501 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 d1) 2023-11-28T17:11:39.557 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2023-11-28T17:11:39.783 INFO:teuthology.orchestra.run.smithi116.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"9","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":2,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2023-11-28T17:11:39.784 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 2, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "9" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2023-11-28T17:11:39.784 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d0', 0), ('/d1', 1)] 2023-11-28T17:11:41.786 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2023-11-28T17:11:42.024 INFO:teuthology.orchestra.run.smithi116.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"13","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2023-11-28T17:11:42.024 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2023-11-28T17:11:42.024 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d0', 0), ('/d1', 1)] 2023-11-28T17:11:44.026 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2023-11-28T17:11:44.271 INFO:teuthology.orchestra.run.smithi116.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"13","version":"13","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2023-11-28T17:11:44.272 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "13", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "13" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2023-11-28T17:11:44.272 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/d0', 0), ('/d1', 1)] 2023-11-28T17:11:46.273 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2023-11-28T17:11:46.502 INFO:teuthology.orchestra.run.smithi116.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"18","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/d0","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2023-11-28T17:11:46.502 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "18" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/d0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false } ] 2023-11-28T17:11:46.502 INFO:tasks.cephfs.cephfs_test_case:[('/d0', 0)] =?= [('/d0', 0), ('/d1', 1)] 2023-11-28T17:11:48.503 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2023-11-28T17:11:48.740 INFO:teuthology.orchestra.run.smithi116.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"18","version":"18","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/d0","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/d1","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2023-11-28T17:11:48.740 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "18", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "18" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/d0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000001", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/d1", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2023-11-28T17:11:48.740 INFO:tasks.cephfs.cephfs_test_case:[('/d0', 0), ('/d1', 1)] =?= [('/d0', 0), ('/d1', 1)] 2023-11-28T17:11:48.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:48.741 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch d0/f0) 2023-11-28T17:11:48.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:11:48.800 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch d1/f0) 2023-11-28T17:11:48.860 INFO:tasks.ceph.mds.a.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7fa6685cd640 time 2023-11-28T17:11:48.860023+0000 2023-11-28T17:11:48.861 INFO:tasks.ceph.mds.a.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:11:48.862 INFO:tasks.ceph.mds.a.smithi116.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:11:48.863 INFO:tasks.ceph.mds.a.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fa66e16eef8] 2023-11-28T17:11:48.863 INFO:tasks.ceph.mds.a.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa66e16f0b6] 2023-11-28T17:11:48.864 INFO:tasks.ceph.mds.a.smithi116.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55f8e3f2d329] 2023-11-28T17:11:48.865 INFO:tasks.ceph.mds.a.smithi116.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55f8e3f225b5] 2023-11-28T17:11:48.865 INFO:tasks.ceph.mds.a.smithi116.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55f8e3ed185b] 2023-11-28T17:11:48.865 INFO:tasks.ceph.mds.a.smithi116.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55f8e3e0b521] 2023-11-28T17:11:48.865 INFO:tasks.ceph.mds.a.smithi116.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55f8e3e12508] 2023-11-28T17:11:48.866 INFO:tasks.ceph.mds.a.smithi116.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55f8e3e07edd] 2023-11-28T17:11:48.866 INFO:tasks.ceph.mds.a.smithi116.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55f8e3e087bc] 2023-11-28T17:11:48.866 INFO:tasks.ceph.mds.a.smithi116.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8e3df4467] 2023-11-28T17:11:48.866 INFO:tasks.ceph.mds.a.smithi116.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55f8e3d7848b] 2023-11-28T17:11:48.866 INFO:tasks.ceph.mds.a.smithi116.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8e3d7897c] 2023-11-28T17:11:48.866 INFO:tasks.ceph.mds.a.smithi116.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55f8e3d624b5] 2023-11-28T17:11:48.867 INFO:tasks.ceph.mds.a.smithi116.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7fa66e36040a] 2023-11-28T17:11:48.867 INFO:tasks.ceph.mds.a.smithi116.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fa66e3f24a1] 2023-11-28T17:11:48.867 INFO:tasks.ceph.mds.a.smithi116.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7fa66d89f802] 2023-11-28T17:11:48.867 INFO:tasks.ceph.mds.a.smithi116.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7fa66d83f450] 2023-11-28T17:11:48.867 INFO:tasks.ceph.mds.a.smithi116.stderr:2023-11-28T17:11:48.861+0000 7fa6685cd640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7fa6685cd640 time 2023-11-28T17:11:48.860023+0000 2023-11-28T17:11:48.867 INFO:tasks.ceph.mds.a.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:11:48.867 INFO:tasks.ceph.mds.a.smithi116.stderr: 2023-11-28T17:11:48.867 INFO:tasks.ceph.mds.a.smithi116.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:11:48.867 INFO:tasks.ceph.mds.a.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fa66e16eef8] 2023-11-28T17:11:48.867 INFO:tasks.ceph.mds.a.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa66e16f0b6] 2023-11-28T17:11:48.867 INFO:tasks.ceph.mds.a.smithi116.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55f8e3f2d329] 2023-11-28T17:11:48.867 INFO:tasks.ceph.mds.a.smithi116.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55f8e3f225b5] 2023-11-28T17:11:48.867 INFO:tasks.ceph.mds.a.smithi116.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55f8e3ed185b] 2023-11-28T17:11:48.868 INFO:tasks.ceph.mds.a.smithi116.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55f8e3e0b521] 2023-11-28T17:11:48.868 INFO:tasks.ceph.mds.a.smithi116.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55f8e3e12508] 2023-11-28T17:11:48.868 INFO:tasks.ceph.mds.a.smithi116.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55f8e3e07edd] 2023-11-28T17:11:48.868 INFO:tasks.ceph.mds.a.smithi116.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55f8e3e087bc] 2023-11-28T17:11:48.868 INFO:tasks.ceph.mds.a.smithi116.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8e3df4467] 2023-11-28T17:11:48.868 INFO:tasks.ceph.mds.a.smithi116.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55f8e3d7848b] 2023-11-28T17:11:48.868 INFO:tasks.ceph.mds.a.smithi116.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8e3d7897c] 2023-11-28T17:11:48.868 INFO:tasks.ceph.mds.a.smithi116.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55f8e3d624b5] 2023-11-28T17:11:48.868 INFO:tasks.ceph.mds.a.smithi116.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7fa66e36040a] 2023-11-28T17:11:48.868 INFO:tasks.ceph.mds.a.smithi116.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fa66e3f24a1] 2023-11-28T17:11:48.868 INFO:tasks.ceph.mds.a.smithi116.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7fa66d89f802] 2023-11-28T17:11:48.868 INFO:tasks.ceph.mds.a.smithi116.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7fa66d83f450] 2023-11-28T17:11:48.868 INFO:tasks.ceph.mds.a.smithi116.stderr: 2023-11-28T17:11:48.869 INFO:tasks.ceph.mds.a.smithi116.stderr:*** Caught signal (Aborted) ** 2023-11-28T17:11:48.869 INFO:tasks.ceph.mds.a.smithi116.stderr: in thread 7fa6685cd640 thread_name:ms_dispatch 2023-11-28T17:11:48.869 INFO:tasks.ceph.mds.a.smithi116.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:11:48.869 INFO:tasks.ceph.mds.a.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fa66d854db0] 2023-11-28T17:11:48.870 INFO:tasks.ceph.mds.a.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fa66d8a154c] 2023-11-28T17:11:48.870 INFO:tasks.ceph.mds.a.smithi116.stderr: 3: raise() 2023-11-28T17:11:48.870 INFO:tasks.ceph.mds.a.smithi116.stderr: 4: abort() 2023-11-28T17:11:48.870 INFO:tasks.ceph.mds.a.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa66e16ef52] 2023-11-28T17:11:48.870 INFO:tasks.ceph.mds.a.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa66e16f0b6] 2023-11-28T17:11:48.870 INFO:tasks.ceph.mds.a.smithi116.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55f8e3f2d329] 2023-11-28T17:11:48.870 INFO:tasks.ceph.mds.a.smithi116.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55f8e3f225b5] 2023-11-28T17:11:48.870 INFO:tasks.ceph.mds.a.smithi116.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55f8e3ed185b] 2023-11-28T17:11:48.870 INFO:tasks.ceph.mds.a.smithi116.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55f8e3e0b521] 2023-11-28T17:11:48.870 INFO:tasks.ceph.mds.a.smithi116.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55f8e3e12508] 2023-11-28T17:11:48.870 INFO:tasks.ceph.mds.a.smithi116.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55f8e3e07edd] 2023-11-28T17:11:48.870 INFO:tasks.ceph.mds.a.smithi116.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55f8e3e087bc] 2023-11-28T17:11:48.871 INFO:tasks.ceph.mds.a.smithi116.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8e3df4467] 2023-11-28T17:11:48.871 INFO:tasks.ceph.mds.a.smithi116.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55f8e3d7848b] 2023-11-28T17:11:48.871 INFO:tasks.ceph.mds.a.smithi116.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8e3d7897c] 2023-11-28T17:11:48.871 INFO:tasks.ceph.mds.a.smithi116.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55f8e3d624b5] 2023-11-28T17:11:48.871 INFO:tasks.ceph.mds.a.smithi116.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7fa66e36040a] 2023-11-28T17:11:48.871 INFO:tasks.ceph.mds.a.smithi116.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fa66e3f24a1] 2023-11-28T17:11:48.871 INFO:tasks.ceph.mds.a.smithi116.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7fa66d89f802] 2023-11-28T17:11:48.871 INFO:tasks.ceph.mds.a.smithi116.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7fa66d83f450] 2023-11-28T17:11:48.871 INFO:tasks.ceph.mds.a.smithi116.stderr:2023-11-28T17:11:48.864+0000 7fa6685cd640 -1 *** Caught signal (Aborted) ** 2023-11-28T17:11:48.871 INFO:tasks.ceph.mds.a.smithi116.stderr: in thread 7fa6685cd640 thread_name:ms_dispatch 2023-11-28T17:11:48.871 INFO:tasks.ceph.mds.a.smithi116.stderr: 2023-11-28T17:11:48.871 INFO:tasks.ceph.mds.a.smithi116.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:11:48.871 INFO:tasks.ceph.mds.a.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fa66d854db0] 2023-11-28T17:11:48.871 INFO:tasks.ceph.mds.a.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fa66d8a154c] 2023-11-28T17:11:48.872 INFO:tasks.ceph.mds.a.smithi116.stderr: 3: raise() 2023-11-28T17:11:48.872 INFO:tasks.ceph.mds.a.smithi116.stderr: 4: abort() 2023-11-28T17:11:48.872 INFO:tasks.ceph.mds.a.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa66e16ef52] 2023-11-28T17:11:48.872 INFO:tasks.ceph.mds.a.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa66e16f0b6] 2023-11-28T17:11:48.872 INFO:tasks.ceph.mds.a.smithi116.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55f8e3f2d329] 2023-11-28T17:11:48.872 INFO:tasks.ceph.mds.a.smithi116.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55f8e3f225b5] 2023-11-28T17:11:48.872 INFO:tasks.ceph.mds.a.smithi116.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55f8e3ed185b] 2023-11-28T17:11:48.872 INFO:tasks.ceph.mds.a.smithi116.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55f8e3e0b521] 2023-11-28T17:11:48.872 INFO:tasks.ceph.mds.a.smithi116.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55f8e3e12508] 2023-11-28T17:11:48.873 INFO:tasks.ceph.mds.a.smithi116.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55f8e3e07edd] 2023-11-28T17:11:48.873 INFO:tasks.ceph.mds.a.smithi116.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55f8e3e087bc] 2023-11-28T17:11:48.873 INFO:tasks.ceph.mds.a.smithi116.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8e3df4467] 2023-11-28T17:11:48.873 INFO:tasks.ceph.mds.a.smithi116.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55f8e3d7848b] 2023-11-28T17:11:48.873 INFO:tasks.ceph.mds.a.smithi116.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8e3d7897c] 2023-11-28T17:11:48.873 INFO:tasks.ceph.mds.a.smithi116.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55f8e3d624b5] 2023-11-28T17:11:48.873 INFO:tasks.ceph.mds.a.smithi116.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7fa66e36040a] 2023-11-28T17:11:48.873 INFO:tasks.ceph.mds.a.smithi116.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fa66e3f24a1] 2023-11-28T17:11:48.873 INFO:tasks.ceph.mds.a.smithi116.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7fa66d89f802] 2023-11-28T17:11:48.873 INFO:tasks.ceph.mds.a.smithi116.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7fa66d83f450] 2023-11-28T17:11:48.873 INFO:tasks.ceph.mds.a.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T17:11:48.873 INFO:tasks.ceph.mds.a.smithi116.stderr: 2023-11-28T17:11:48.887 INFO:tasks.ceph.mds.a.smithi116.stderr: -1> 2023-11-28T17:11:48.861+0000 7fa6685cd640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7fa6685cd640 time 2023-11-28T17:11:48.860023+0000 2023-11-28T17:11:48.887 INFO:tasks.ceph.mds.a.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:11:48.888 INFO:tasks.ceph.mds.a.smithi116.stderr: 2023-11-28T17:11:48.888 INFO:tasks.ceph.mds.a.smithi116.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:11:48.888 INFO:tasks.ceph.mds.a.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fa66e16eef8] 2023-11-28T17:11:48.888 INFO:tasks.ceph.mds.a.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa66e16f0b6] 2023-11-28T17:11:48.888 INFO:tasks.ceph.mds.a.smithi116.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55f8e3f2d329] 2023-11-28T17:11:48.888 INFO:tasks.ceph.mds.a.smithi116.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55f8e3f225b5] 2023-11-28T17:11:48.889 INFO:tasks.ceph.mds.a.smithi116.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55f8e3ed185b] 2023-11-28T17:11:48.889 INFO:tasks.ceph.mds.a.smithi116.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55f8e3e0b521] 2023-11-28T17:11:48.889 INFO:tasks.ceph.mds.a.smithi116.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55f8e3e12508] 2023-11-28T17:11:48.889 INFO:tasks.ceph.mds.a.smithi116.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55f8e3e07edd] 2023-11-28T17:11:48.889 INFO:tasks.ceph.mds.a.smithi116.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55f8e3e087bc] 2023-11-28T17:11:48.889 INFO:tasks.ceph.mds.a.smithi116.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8e3df4467] 2023-11-28T17:11:48.889 INFO:tasks.ceph.mds.a.smithi116.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55f8e3d7848b] 2023-11-28T17:11:48.889 INFO:tasks.ceph.mds.a.smithi116.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8e3d7897c] 2023-11-28T17:11:48.890 INFO:tasks.ceph.mds.a.smithi116.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55f8e3d624b5] 2023-11-28T17:11:48.890 INFO:tasks.ceph.mds.a.smithi116.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7fa66e36040a] 2023-11-28T17:11:48.890 INFO:tasks.ceph.mds.a.smithi116.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fa66e3f24a1] 2023-11-28T17:11:48.890 INFO:tasks.ceph.mds.a.smithi116.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7fa66d89f802] 2023-11-28T17:11:48.890 INFO:tasks.ceph.mds.a.smithi116.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7fa66d83f450] 2023-11-28T17:11:48.890 INFO:tasks.ceph.mds.a.smithi116.stderr: 2023-11-28T17:11:48.891 INFO:tasks.ceph.mds.a.smithi116.stderr: 0> 2023-11-28T17:11:48.864+0000 7fa6685cd640 -1 *** Caught signal (Aborted) ** 2023-11-28T17:11:48.891 INFO:tasks.ceph.mds.a.smithi116.stderr: in thread 7fa6685cd640 thread_name:ms_dispatch 2023-11-28T17:11:48.891 INFO:tasks.ceph.mds.a.smithi116.stderr: 2023-11-28T17:11:48.891 INFO:tasks.ceph.mds.a.smithi116.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:11:48.891 INFO:tasks.ceph.mds.a.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fa66d854db0] 2023-11-28T17:11:48.891 INFO:tasks.ceph.mds.a.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fa66d8a154c] 2023-11-28T17:11:48.892 INFO:tasks.ceph.mds.a.smithi116.stderr: 3: raise() 2023-11-28T17:11:48.892 INFO:tasks.ceph.mds.a.smithi116.stderr: 4: abort() 2023-11-28T17:11:48.892 INFO:tasks.ceph.mds.a.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa66e16ef52] 2023-11-28T17:11:48.892 INFO:tasks.ceph.mds.a.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa66e16f0b6] 2023-11-28T17:11:48.892 INFO:tasks.ceph.mds.a.smithi116.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55f8e3f2d329] 2023-11-28T17:11:48.892 INFO:tasks.ceph.mds.a.smithi116.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55f8e3f225b5] 2023-11-28T17:11:48.892 INFO:tasks.ceph.mds.a.smithi116.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55f8e3ed185b] 2023-11-28T17:11:48.893 INFO:tasks.ceph.mds.a.smithi116.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55f8e3e0b521] 2023-11-28T17:11:48.893 INFO:tasks.ceph.mds.a.smithi116.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55f8e3e12508] 2023-11-28T17:11:48.893 INFO:tasks.ceph.mds.a.smithi116.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55f8e3e07edd] 2023-11-28T17:11:48.893 INFO:tasks.ceph.mds.a.smithi116.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55f8e3e087bc] 2023-11-28T17:11:48.893 INFO:tasks.ceph.mds.a.smithi116.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8e3df4467] 2023-11-28T17:11:48.893 INFO:tasks.ceph.mds.a.smithi116.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55f8e3d7848b] 2023-11-28T17:11:48.893 INFO:tasks.ceph.mds.a.smithi116.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8e3d7897c] 2023-11-28T17:11:48.893 INFO:tasks.ceph.mds.a.smithi116.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55f8e3d624b5] 2023-11-28T17:11:48.894 INFO:tasks.ceph.mds.a.smithi116.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7fa66e36040a] 2023-11-28T17:11:48.894 INFO:tasks.ceph.mds.a.smithi116.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fa66e3f24a1] 2023-11-28T17:11:48.894 INFO:tasks.ceph.mds.a.smithi116.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7fa66d89f802] 2023-11-28T17:11:48.894 INFO:tasks.ceph.mds.a.smithi116.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7fa66d83f450] 2023-11-28T17:11:48.894 INFO:tasks.ceph.mds.a.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T17:11:48.894 INFO:tasks.ceph.mds.a.smithi116.stderr: 2023-11-28T17:11:48.902 INFO:tasks.ceph.mds.a.smithi116.stderr: -1> 2023-11-28T17:11:48.861+0000 7fa6685cd640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7fa6685cd640 time 2023-11-28T17:11:48.860023+0000 2023-11-28T17:11:48.903 INFO:tasks.ceph.mds.a.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:11:48.903 INFO:tasks.ceph.mds.a.smithi116.stderr: 2023-11-28T17:11:48.903 INFO:tasks.ceph.mds.a.smithi116.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:11:48.903 INFO:tasks.ceph.mds.a.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fa66e16eef8] 2023-11-28T17:11:48.903 INFO:tasks.ceph.mds.a.smithi116.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa66e16f0b6] 2023-11-28T17:11:48.903 INFO:tasks.ceph.mds.a.smithi116.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55f8e3f2d329] 2023-11-28T17:11:48.903 INFO:tasks.ceph.mds.a.smithi116.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55f8e3f225b5] 2023-11-28T17:11:48.903 INFO:tasks.ceph.mds.a.smithi116.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55f8e3ed185b] 2023-11-28T17:11:48.904 INFO:tasks.ceph.mds.a.smithi116.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55f8e3e0b521] 2023-11-28T17:11:48.904 INFO:tasks.ceph.mds.a.smithi116.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55f8e3e12508] 2023-11-28T17:11:48.904 INFO:tasks.ceph.mds.a.smithi116.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55f8e3e07edd] 2023-11-28T17:11:48.904 INFO:tasks.ceph.mds.a.smithi116.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55f8e3e087bc] 2023-11-28T17:11:48.904 INFO:tasks.ceph.mds.a.smithi116.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8e3df4467] 2023-11-28T17:11:48.904 INFO:tasks.ceph.mds.a.smithi116.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55f8e3d7848b] 2023-11-28T17:11:48.905 INFO:tasks.ceph.mds.a.smithi116.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8e3d7897c] 2023-11-28T17:11:48.905 INFO:tasks.ceph.mds.a.smithi116.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55f8e3d624b5] 2023-11-28T17:11:48.905 INFO:tasks.ceph.mds.a.smithi116.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7fa66e36040a] 2023-11-28T17:11:48.905 INFO:tasks.ceph.mds.a.smithi116.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fa66e3f24a1] 2023-11-28T17:11:48.905 INFO:tasks.ceph.mds.a.smithi116.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7fa66d89f802] 2023-11-28T17:11:48.905 INFO:tasks.ceph.mds.a.smithi116.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7fa66d83f450] 2023-11-28T17:11:48.905 INFO:tasks.ceph.mds.a.smithi116.stderr: 2023-11-28T17:11:48.905 INFO:tasks.ceph.mds.a.smithi116.stderr: 0> 2023-11-28T17:11:48.864+0000 7fa6685cd640 -1 *** Caught signal (Aborted) ** 2023-11-28T17:11:48.906 INFO:tasks.ceph.mds.a.smithi116.stderr: in thread 7fa6685cd640 thread_name:ms_dispatch 2023-11-28T17:11:48.906 INFO:tasks.ceph.mds.a.smithi116.stderr: 2023-11-28T17:11:48.906 INFO:tasks.ceph.mds.a.smithi116.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:11:48.906 INFO:tasks.ceph.mds.a.smithi116.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fa66d854db0] 2023-11-28T17:11:48.906 INFO:tasks.ceph.mds.a.smithi116.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fa66d8a154c] 2023-11-28T17:11:48.906 INFO:tasks.ceph.mds.a.smithi116.stderr: 3: raise() 2023-11-28T17:11:48.906 INFO:tasks.ceph.mds.a.smithi116.stderr: 4: abort() 2023-11-28T17:11:48.906 INFO:tasks.ceph.mds.a.smithi116.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa66e16ef52] 2023-11-28T17:11:48.906 INFO:tasks.ceph.mds.a.smithi116.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa66e16f0b6] 2023-11-28T17:11:48.907 INFO:tasks.ceph.mds.a.smithi116.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55f8e3f2d329] 2023-11-28T17:11:48.907 INFO:tasks.ceph.mds.a.smithi116.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55f8e3f225b5] 2023-11-28T17:11:48.907 INFO:tasks.ceph.mds.a.smithi116.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55f8e3ed185b] 2023-11-28T17:11:48.907 INFO:tasks.ceph.mds.a.smithi116.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55f8e3e0b521] 2023-11-28T17:11:48.907 INFO:tasks.ceph.mds.a.smithi116.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55f8e3e12508] 2023-11-28T17:11:48.907 INFO:tasks.ceph.mds.a.smithi116.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55f8e3e07edd] 2023-11-28T17:11:48.907 INFO:tasks.ceph.mds.a.smithi116.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55f8e3e087bc] 2023-11-28T17:11:48.907 INFO:tasks.ceph.mds.a.smithi116.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55f8e3df4467] 2023-11-28T17:11:48.908 INFO:tasks.ceph.mds.a.smithi116.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55f8e3d7848b] 2023-11-28T17:11:48.908 INFO:tasks.ceph.mds.a.smithi116.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55f8e3d7897c] 2023-11-28T17:11:48.908 INFO:tasks.ceph.mds.a.smithi116.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55f8e3d624b5] 2023-11-28T17:11:48.908 INFO:tasks.ceph.mds.a.smithi116.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7fa66e36040a] 2023-11-28T17:11:48.908 INFO:tasks.ceph.mds.a.smithi116.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7fa66e3f24a1] 2023-11-28T17:11:48.908 INFO:tasks.ceph.mds.a.smithi116.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7fa66d89f802] 2023-11-28T17:11:48.909 INFO:tasks.ceph.mds.a.smithi116.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7fa66d83f450] 2023-11-28T17:11:48.909 INFO:tasks.ceph.mds.a.smithi116.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T17:11:48.909 INFO:tasks.ceph.mds.a.smithi116.stderr: 2023-11-28T17:11:49.086 INFO:tasks.ceph.mds.a.smithi116.stderr:daemon-helper: command crashed with signal 6 2023-11-28T17:11:53.107 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:11:53.118 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:11:55.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2023-11-28T17:12:02.185 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~7s 2023-11-28T17:12:08.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~13s 2023-11-28T17:12:10.107 INFO:tasks.ceph.mds.b.smithi156.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f46015f7640 time 2023-11-28T17:12:10.106834+0000 2023-11-28T17:12:10.107 INFO:tasks.ceph.mds.b.smithi156.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:12:10.109 INFO:tasks.ceph.mds.b.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:10.109 INFO:tasks.ceph.mds.b.smithi156.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f460716eef8] 2023-11-28T17:12:10.109 INFO:tasks.ceph.mds.b.smithi156.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f460716f0b6] 2023-11-28T17:12:10.109 INFO:tasks.ceph.mds.b.smithi156.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5567567d2329] 2023-11-28T17:12:10.109 INFO:tasks.ceph.mds.b.smithi156.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5567567c75b5] 2023-11-28T17:12:10.109 INFO:tasks.ceph.mds.b.smithi156.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55675677685b] 2023-11-28T17:12:10.109 INFO:tasks.ceph.mds.b.smithi156.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5567566b0521] 2023-11-28T17:12:10.109 INFO:tasks.ceph.mds.b.smithi156.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5567566b7508] 2023-11-28T17:12:10.109 INFO:tasks.ceph.mds.b.smithi156.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5567566acedd] 2023-11-28T17:12:10.110 INFO:tasks.ceph.mds.b.smithi156.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5567566ad7bc] 2023-11-28T17:12:10.110 INFO:tasks.ceph.mds.b.smithi156.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556756699467] 2023-11-28T17:12:10.110 INFO:tasks.ceph.mds.b.smithi156.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55675661d48b] 2023-11-28T17:12:10.110 INFO:tasks.ceph.mds.b.smithi156.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55675661d97c] 2023-11-28T17:12:10.110 INFO:tasks.ceph.mds.b.smithi156.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5567566074b5] 2023-11-28T17:12:10.110 INFO:tasks.ceph.mds.b.smithi156.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7f460736040a] 2023-11-28T17:12:10.110 INFO:tasks.ceph.mds.b.smithi156.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f46073f24a1] 2023-11-28T17:12:10.110 INFO:tasks.ceph.mds.b.smithi156.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7f460689f802] 2023-11-28T17:12:10.110 INFO:tasks.ceph.mds.b.smithi156.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7f460683f450] 2023-11-28T17:12:10.110 INFO:tasks.ceph.mds.b.smithi156.stderr:2023-11-28T17:12:10.107+0000 7f46015f7640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f46015f7640 time 2023-11-28T17:12:10.106834+0000 2023-11-28T17:12:10.110 INFO:tasks.ceph.mds.b.smithi156.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:12:10.111 INFO:tasks.ceph.mds.b.smithi156.stderr: 2023-11-28T17:12:10.111 INFO:tasks.ceph.mds.b.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:10.111 INFO:tasks.ceph.mds.b.smithi156.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f460716eef8] 2023-11-28T17:12:10.111 INFO:tasks.ceph.mds.b.smithi156.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f460716f0b6] 2023-11-28T17:12:10.111 INFO:tasks.ceph.mds.b.smithi156.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5567567d2329] 2023-11-28T17:12:10.111 INFO:tasks.ceph.mds.b.smithi156.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5567567c75b5] 2023-11-28T17:12:10.111 INFO:tasks.ceph.mds.b.smithi156.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55675677685b] 2023-11-28T17:12:10.111 INFO:tasks.ceph.mds.b.smithi156.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5567566b0521] 2023-11-28T17:12:10.111 INFO:tasks.ceph.mds.b.smithi156.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5567566b7508] 2023-11-28T17:12:10.111 INFO:tasks.ceph.mds.b.smithi156.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5567566acedd] 2023-11-28T17:12:10.111 INFO:tasks.ceph.mds.b.smithi156.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5567566ad7bc] 2023-11-28T17:12:10.111 INFO:tasks.ceph.mds.b.smithi156.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556756699467] 2023-11-28T17:12:10.112 INFO:tasks.ceph.mds.b.smithi156.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55675661d48b] 2023-11-28T17:12:10.112 INFO:tasks.ceph.mds.b.smithi156.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55675661d97c] 2023-11-28T17:12:10.112 INFO:tasks.ceph.mds.b.smithi156.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5567566074b5] 2023-11-28T17:12:10.112 INFO:tasks.ceph.mds.b.smithi156.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7f460736040a] 2023-11-28T17:12:10.112 INFO:tasks.ceph.mds.b.smithi156.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f46073f24a1] 2023-11-28T17:12:10.112 INFO:tasks.ceph.mds.b.smithi156.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7f460689f802] 2023-11-28T17:12:10.112 INFO:tasks.ceph.mds.b.smithi156.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7f460683f450] 2023-11-28T17:12:10.112 INFO:tasks.ceph.mds.b.smithi156.stderr: 2023-11-28T17:12:10.112 INFO:tasks.ceph.mds.b.smithi156.stderr:*** Caught signal (Aborted) ** 2023-11-28T17:12:10.112 INFO:tasks.ceph.mds.b.smithi156.stderr: in thread 7f46015f7640 thread_name:ms_dispatch 2023-11-28T17:12:10.113 INFO:tasks.ceph.mds.b.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:10.113 INFO:tasks.ceph.mds.b.smithi156.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f4606854db0] 2023-11-28T17:12:10.113 INFO:tasks.ceph.mds.b.smithi156.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f46068a154c] 2023-11-28T17:12:10.113 INFO:tasks.ceph.mds.b.smithi156.stderr: 3: raise() 2023-11-28T17:12:10.113 INFO:tasks.ceph.mds.b.smithi156.stderr: 4: abort() 2023-11-28T17:12:10.113 INFO:tasks.ceph.mds.b.smithi156.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f460716ef52] 2023-11-28T17:12:10.113 INFO:tasks.ceph.mds.b.smithi156.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f460716f0b6] 2023-11-28T17:12:10.113 INFO:tasks.ceph.mds.b.smithi156.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5567567d2329] 2023-11-28T17:12:10.113 INFO:tasks.ceph.mds.b.smithi156.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5567567c75b5] 2023-11-28T17:12:10.114 INFO:tasks.ceph.mds.b.smithi156.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55675677685b] 2023-11-28T17:12:10.114 INFO:tasks.ceph.mds.b.smithi156.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5567566b0521] 2023-11-28T17:12:10.114 INFO:tasks.ceph.mds.b.smithi156.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5567566b7508] 2023-11-28T17:12:10.114 INFO:tasks.ceph.mds.b.smithi156.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5567566acedd] 2023-11-28T17:12:10.114 INFO:tasks.ceph.mds.b.smithi156.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5567566ad7bc] 2023-11-28T17:12:10.114 INFO:tasks.ceph.mds.b.smithi156.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556756699467] 2023-11-28T17:12:10.114 INFO:tasks.ceph.mds.b.smithi156.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55675661d48b] 2023-11-28T17:12:10.114 INFO:tasks.ceph.mds.b.smithi156.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55675661d97c] 2023-11-28T17:12:10.114 INFO:tasks.ceph.mds.b.smithi156.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5567566074b5] 2023-11-28T17:12:10.114 INFO:tasks.ceph.mds.b.smithi156.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7f460736040a] 2023-11-28T17:12:10.114 INFO:tasks.ceph.mds.b.smithi156.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f46073f24a1] 2023-11-28T17:12:10.114 INFO:tasks.ceph.mds.b.smithi156.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7f460689f802] 2023-11-28T17:12:10.115 INFO:tasks.ceph.mds.b.smithi156.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7f460683f450] 2023-11-28T17:12:10.115 INFO:tasks.ceph.mds.b.smithi156.stderr:2023-11-28T17:12:10.110+0000 7f46015f7640 -1 *** Caught signal (Aborted) ** 2023-11-28T17:12:10.115 INFO:tasks.ceph.mds.b.smithi156.stderr: in thread 7f46015f7640 thread_name:ms_dispatch 2023-11-28T17:12:10.115 INFO:tasks.ceph.mds.b.smithi156.stderr: 2023-11-28T17:12:10.115 INFO:tasks.ceph.mds.b.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:10.115 INFO:tasks.ceph.mds.b.smithi156.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f4606854db0] 2023-11-28T17:12:10.115 INFO:tasks.ceph.mds.b.smithi156.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f46068a154c] 2023-11-28T17:12:10.115 INFO:tasks.ceph.mds.b.smithi156.stderr: 3: raise() 2023-11-28T17:12:10.115 INFO:tasks.ceph.mds.b.smithi156.stderr: 4: abort() 2023-11-28T17:12:10.115 INFO:tasks.ceph.mds.b.smithi156.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f460716ef52] 2023-11-28T17:12:10.115 INFO:tasks.ceph.mds.b.smithi156.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f460716f0b6] 2023-11-28T17:12:10.115 INFO:tasks.ceph.mds.b.smithi156.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5567567d2329] 2023-11-28T17:12:10.115 INFO:tasks.ceph.mds.b.smithi156.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5567567c75b5] 2023-11-28T17:12:10.116 INFO:tasks.ceph.mds.b.smithi156.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55675677685b] 2023-11-28T17:12:10.116 INFO:tasks.ceph.mds.b.smithi156.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5567566b0521] 2023-11-28T17:12:10.116 INFO:tasks.ceph.mds.b.smithi156.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5567566b7508] 2023-11-28T17:12:10.116 INFO:tasks.ceph.mds.b.smithi156.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5567566acedd] 2023-11-28T17:12:10.116 INFO:tasks.ceph.mds.b.smithi156.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5567566ad7bc] 2023-11-28T17:12:10.116 INFO:tasks.ceph.mds.b.smithi156.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556756699467] 2023-11-28T17:12:10.116 INFO:tasks.ceph.mds.b.smithi156.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55675661d48b] 2023-11-28T17:12:10.116 INFO:tasks.ceph.mds.b.smithi156.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55675661d97c] 2023-11-28T17:12:10.116 INFO:tasks.ceph.mds.b.smithi156.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5567566074b5] 2023-11-28T17:12:10.116 INFO:tasks.ceph.mds.b.smithi156.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7f460736040a] 2023-11-28T17:12:10.116 INFO:tasks.ceph.mds.b.smithi156.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f46073f24a1] 2023-11-28T17:12:10.116 INFO:tasks.ceph.mds.b.smithi156.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7f460689f802] 2023-11-28T17:12:10.117 INFO:tasks.ceph.mds.b.smithi156.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7f460683f450] 2023-11-28T17:12:10.117 INFO:tasks.ceph.mds.b.smithi156.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T17:12:10.117 INFO:tasks.ceph.mds.b.smithi156.stderr: 2023-11-28T17:12:10.132 INFO:tasks.ceph.mds.b.smithi156.stderr: -1> 2023-11-28T17:12:10.107+0000 7f46015f7640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f46015f7640 time 2023-11-28T17:12:10.106834+0000 2023-11-28T17:12:10.133 INFO:tasks.ceph.mds.b.smithi156.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:12:10.133 INFO:tasks.ceph.mds.b.smithi156.stderr: 2023-11-28T17:12:10.133 INFO:tasks.ceph.mds.b.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:10.133 INFO:tasks.ceph.mds.b.smithi156.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f460716eef8] 2023-11-28T17:12:10.133 INFO:tasks.ceph.mds.b.smithi156.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f460716f0b6] 2023-11-28T17:12:10.133 INFO:tasks.ceph.mds.b.smithi156.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5567567d2329] 2023-11-28T17:12:10.133 INFO:tasks.ceph.mds.b.smithi156.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5567567c75b5] 2023-11-28T17:12:10.133 INFO:tasks.ceph.mds.b.smithi156.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55675677685b] 2023-11-28T17:12:10.133 INFO:tasks.ceph.mds.b.smithi156.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5567566b0521] 2023-11-28T17:12:10.133 INFO:tasks.ceph.mds.b.smithi156.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5567566b7508] 2023-11-28T17:12:10.134 INFO:tasks.ceph.mds.b.smithi156.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5567566acedd] 2023-11-28T17:12:10.134 INFO:tasks.ceph.mds.b.smithi156.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5567566ad7bc] 2023-11-28T17:12:10.134 INFO:tasks.ceph.mds.b.smithi156.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556756699467] 2023-11-28T17:12:10.134 INFO:tasks.ceph.mds.b.smithi156.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55675661d48b] 2023-11-28T17:12:10.134 INFO:tasks.ceph.mds.b.smithi156.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55675661d97c] 2023-11-28T17:12:10.134 INFO:tasks.ceph.mds.b.smithi156.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5567566074b5] 2023-11-28T17:12:10.134 INFO:tasks.ceph.mds.b.smithi156.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7f460736040a] 2023-11-28T17:12:10.134 INFO:tasks.ceph.mds.b.smithi156.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f46073f24a1] 2023-11-28T17:12:10.134 INFO:tasks.ceph.mds.b.smithi156.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7f460689f802] 2023-11-28T17:12:10.134 INFO:tasks.ceph.mds.b.smithi156.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7f460683f450] 2023-11-28T17:12:10.134 INFO:tasks.ceph.mds.b.smithi156.stderr: 2023-11-28T17:12:10.134 INFO:tasks.ceph.mds.b.smithi156.stderr: 0> 2023-11-28T17:12:10.110+0000 7f46015f7640 -1 *** Caught signal (Aborted) ** 2023-11-28T17:12:10.135 INFO:tasks.ceph.mds.b.smithi156.stderr: in thread 7f46015f7640 thread_name:ms_dispatch 2023-11-28T17:12:10.135 INFO:tasks.ceph.mds.b.smithi156.stderr: 2023-11-28T17:12:10.135 INFO:tasks.ceph.mds.b.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:10.135 INFO:tasks.ceph.mds.b.smithi156.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f4606854db0] 2023-11-28T17:12:10.135 INFO:tasks.ceph.mds.b.smithi156.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f46068a154c] 2023-11-28T17:12:10.135 INFO:tasks.ceph.mds.b.smithi156.stderr: 3: raise() 2023-11-28T17:12:10.135 INFO:tasks.ceph.mds.b.smithi156.stderr: 4: abort() 2023-11-28T17:12:10.135 INFO:tasks.ceph.mds.b.smithi156.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f460716ef52] 2023-11-28T17:12:10.135 INFO:tasks.ceph.mds.b.smithi156.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f460716f0b6] 2023-11-28T17:12:10.135 INFO:tasks.ceph.mds.b.smithi156.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5567567d2329] 2023-11-28T17:12:10.135 INFO:tasks.ceph.mds.b.smithi156.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5567567c75b5] 2023-11-28T17:12:10.135 INFO:tasks.ceph.mds.b.smithi156.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55675677685b] 2023-11-28T17:12:10.135 INFO:tasks.ceph.mds.b.smithi156.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5567566b0521] 2023-11-28T17:12:10.136 INFO:tasks.ceph.mds.b.smithi156.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5567566b7508] 2023-11-28T17:12:10.136 INFO:tasks.ceph.mds.b.smithi156.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5567566acedd] 2023-11-28T17:12:10.136 INFO:tasks.ceph.mds.b.smithi156.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5567566ad7bc] 2023-11-28T17:12:10.136 INFO:tasks.ceph.mds.b.smithi156.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556756699467] 2023-11-28T17:12:10.136 INFO:tasks.ceph.mds.b.smithi156.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55675661d48b] 2023-11-28T17:12:10.136 INFO:tasks.ceph.mds.b.smithi156.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55675661d97c] 2023-11-28T17:12:10.136 INFO:tasks.ceph.mds.b.smithi156.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5567566074b5] 2023-11-28T17:12:10.136 INFO:tasks.ceph.mds.b.smithi156.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7f460736040a] 2023-11-28T17:12:10.136 INFO:tasks.ceph.mds.b.smithi156.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f46073f24a1] 2023-11-28T17:12:10.136 INFO:tasks.ceph.mds.b.smithi156.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7f460689f802] 2023-11-28T17:12:10.136 INFO:tasks.ceph.mds.b.smithi156.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7f460683f450] 2023-11-28T17:12:10.136 INFO:tasks.ceph.mds.b.smithi156.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T17:12:10.136 INFO:tasks.ceph.mds.b.smithi156.stderr: 2023-11-28T17:12:10.146 INFO:tasks.ceph.mds.b.smithi156.stderr: -1> 2023-11-28T17:12:10.107+0000 7f46015f7640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f46015f7640 time 2023-11-28T17:12:10.106834+0000 2023-11-28T17:12:10.146 INFO:tasks.ceph.mds.b.smithi156.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:12:10.147 INFO:tasks.ceph.mds.b.smithi156.stderr: 2023-11-28T17:12:10.147 INFO:tasks.ceph.mds.b.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:10.147 INFO:tasks.ceph.mds.b.smithi156.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f460716eef8] 2023-11-28T17:12:10.147 INFO:tasks.ceph.mds.b.smithi156.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f460716f0b6] 2023-11-28T17:12:10.147 INFO:tasks.ceph.mds.b.smithi156.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5567567d2329] 2023-11-28T17:12:10.147 INFO:tasks.ceph.mds.b.smithi156.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5567567c75b5] 2023-11-28T17:12:10.147 INFO:tasks.ceph.mds.b.smithi156.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55675677685b] 2023-11-28T17:12:10.147 INFO:tasks.ceph.mds.b.smithi156.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5567566b0521] 2023-11-28T17:12:10.147 INFO:tasks.ceph.mds.b.smithi156.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5567566b7508] 2023-11-28T17:12:10.147 INFO:tasks.ceph.mds.b.smithi156.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5567566acedd] 2023-11-28T17:12:10.147 INFO:tasks.ceph.mds.b.smithi156.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5567566ad7bc] 2023-11-28T17:12:10.147 INFO:tasks.ceph.mds.b.smithi156.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556756699467] 2023-11-28T17:12:10.148 INFO:tasks.ceph.mds.b.smithi156.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55675661d48b] 2023-11-28T17:12:10.148 INFO:tasks.ceph.mds.b.smithi156.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55675661d97c] 2023-11-28T17:12:10.148 INFO:tasks.ceph.mds.b.smithi156.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5567566074b5] 2023-11-28T17:12:10.148 INFO:tasks.ceph.mds.b.smithi156.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7f460736040a] 2023-11-28T17:12:10.148 INFO:tasks.ceph.mds.b.smithi156.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f46073f24a1] 2023-11-28T17:12:10.148 INFO:tasks.ceph.mds.b.smithi156.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7f460689f802] 2023-11-28T17:12:10.148 INFO:tasks.ceph.mds.b.smithi156.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7f460683f450] 2023-11-28T17:12:10.148 INFO:tasks.ceph.mds.b.smithi156.stderr: 2023-11-28T17:12:10.148 INFO:tasks.ceph.mds.b.smithi156.stderr: 0> 2023-11-28T17:12:10.110+0000 7f46015f7640 -1 *** Caught signal (Aborted) ** 2023-11-28T17:12:10.148 INFO:tasks.ceph.mds.b.smithi156.stderr: in thread 7f46015f7640 thread_name:ms_dispatch 2023-11-28T17:12:10.148 INFO:tasks.ceph.mds.b.smithi156.stderr: 2023-11-28T17:12:10.148 INFO:tasks.ceph.mds.b.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:10.149 INFO:tasks.ceph.mds.b.smithi156.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f4606854db0] 2023-11-28T17:12:10.149 INFO:tasks.ceph.mds.b.smithi156.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f46068a154c] 2023-11-28T17:12:10.149 INFO:tasks.ceph.mds.b.smithi156.stderr: 3: raise() 2023-11-28T17:12:10.149 INFO:tasks.ceph.mds.b.smithi156.stderr: 4: abort() 2023-11-28T17:12:10.149 INFO:tasks.ceph.mds.b.smithi156.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f460716ef52] 2023-11-28T17:12:10.149 INFO:tasks.ceph.mds.b.smithi156.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f460716f0b6] 2023-11-28T17:12:10.149 INFO:tasks.ceph.mds.b.smithi156.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x5567567d2329] 2023-11-28T17:12:10.149 INFO:tasks.ceph.mds.b.smithi156.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x5567567c75b5] 2023-11-28T17:12:10.149 INFO:tasks.ceph.mds.b.smithi156.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55675677685b] 2023-11-28T17:12:10.149 INFO:tasks.ceph.mds.b.smithi156.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x5567566b0521] 2023-11-28T17:12:10.149 INFO:tasks.ceph.mds.b.smithi156.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x5567566b7508] 2023-11-28T17:12:10.149 INFO:tasks.ceph.mds.b.smithi156.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x5567566acedd] 2023-11-28T17:12:10.150 INFO:tasks.ceph.mds.b.smithi156.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x5567566ad7bc] 2023-11-28T17:12:10.150 INFO:tasks.ceph.mds.b.smithi156.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556756699467] 2023-11-28T17:12:10.150 INFO:tasks.ceph.mds.b.smithi156.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55675661d48b] 2023-11-28T17:12:10.150 INFO:tasks.ceph.mds.b.smithi156.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55675661d97c] 2023-11-28T17:12:10.150 INFO:tasks.ceph.mds.b.smithi156.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5567566074b5] 2023-11-28T17:12:10.150 INFO:tasks.ceph.mds.b.smithi156.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7f460736040a] 2023-11-28T17:12:10.150 INFO:tasks.ceph.mds.b.smithi156.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f46073f24a1] 2023-11-28T17:12:10.150 INFO:tasks.ceph.mds.b.smithi156.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7f460689f802] 2023-11-28T17:12:10.150 INFO:tasks.ceph.mds.b.smithi156.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7f460683f450] 2023-11-28T17:12:10.150 INFO:tasks.ceph.mds.b.smithi156.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T17:12:10.150 INFO:tasks.ceph.mds.b.smithi156.stderr: 2023-11-28T17:12:10.342 INFO:tasks.ceph.mds.b.smithi156.stderr:daemon-helper: command crashed with signal 6 2023-11-28T17:12:15.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~20s 2023-11-28T17:12:15.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2023-11-28T17:12:21.822 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~26s 2023-11-28T17:12:21.823 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~7s 2023-11-28T17:12:23.263 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:12:23.274 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:12:28.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~33s 2023-11-28T17:12:28.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~13s 2023-11-28T17:12:30.116 INFO:tasks.ceph.mds.d.smithi156.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f7474190640 time 2023-11-28T17:12:30.115592+0000 2023-11-28T17:12:30.116 INFO:tasks.ceph.mds.d.smithi156.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:12:30.117 INFO:tasks.ceph.mds.d.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:30.117 INFO:tasks.ceph.mds.d.smithi156.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f7479d6eef8] 2023-11-28T17:12:30.117 INFO:tasks.ceph.mds.d.smithi156.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7479d6f0b6] 2023-11-28T17:12:30.118 INFO:tasks.ceph.mds.d.smithi156.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55ff4cd85329] 2023-11-28T17:12:30.118 INFO:tasks.ceph.mds.d.smithi156.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55ff4cd7a5b5] 2023-11-28T17:12:30.118 INFO:tasks.ceph.mds.d.smithi156.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55ff4cd2985b] 2023-11-28T17:12:30.118 INFO:tasks.ceph.mds.d.smithi156.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55ff4cc63521] 2023-11-28T17:12:30.118 INFO:tasks.ceph.mds.d.smithi156.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55ff4cc6a508] 2023-11-28T17:12:30.118 INFO:tasks.ceph.mds.d.smithi156.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55ff4cc5fedd] 2023-11-28T17:12:30.118 INFO:tasks.ceph.mds.d.smithi156.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55ff4cc607bc] 2023-11-28T17:12:30.118 INFO:tasks.ceph.mds.d.smithi156.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4cc4c467] 2023-11-28T17:12:30.118 INFO:tasks.ceph.mds.d.smithi156.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55ff4cbd048b] 2023-11-28T17:12:30.118 INFO:tasks.ceph.mds.d.smithi156.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4cbd097c] 2023-11-28T17:12:30.118 INFO:tasks.ceph.mds.d.smithi156.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55ff4cbba4b5] 2023-11-28T17:12:30.119 INFO:tasks.ceph.mds.d.smithi156.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7f7479f6040a] 2023-11-28T17:12:30.119 INFO:tasks.ceph.mds.d.smithi156.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f7479ff24a1] 2023-11-28T17:12:30.119 INFO:tasks.ceph.mds.d.smithi156.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7f747949f802] 2023-11-28T17:12:30.119 INFO:tasks.ceph.mds.d.smithi156.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7f747943f450] 2023-11-28T17:12:30.119 INFO:tasks.ceph.mds.d.smithi156.stderr:2023-11-28T17:12:30.115+0000 7f7474190640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f7474190640 time 2023-11-28T17:12:30.115592+0000 2023-11-28T17:12:30.120 INFO:tasks.ceph.mds.d.smithi156.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:12:30.120 INFO:tasks.ceph.mds.d.smithi156.stderr: 2023-11-28T17:12:30.120 INFO:tasks.ceph.mds.d.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:30.120 INFO:tasks.ceph.mds.d.smithi156.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f7479d6eef8] 2023-11-28T17:12:30.120 INFO:tasks.ceph.mds.d.smithi156.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7479d6f0b6] 2023-11-28T17:12:30.121 INFO:tasks.ceph.mds.d.smithi156.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55ff4cd85329] 2023-11-28T17:12:30.121 INFO:tasks.ceph.mds.d.smithi156.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55ff4cd7a5b5] 2023-11-28T17:12:30.121 INFO:tasks.ceph.mds.d.smithi156.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55ff4cd2985b] 2023-11-28T17:12:30.121 INFO:tasks.ceph.mds.d.smithi156.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55ff4cc63521] 2023-11-28T17:12:30.122 INFO:tasks.ceph.mds.d.smithi156.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55ff4cc6a508] 2023-11-28T17:12:30.122 INFO:tasks.ceph.mds.d.smithi156.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55ff4cc5fedd] 2023-11-28T17:12:30.122 INFO:tasks.ceph.mds.d.smithi156.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55ff4cc607bc] 2023-11-28T17:12:30.122 INFO:tasks.ceph.mds.d.smithi156.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4cc4c467] 2023-11-28T17:12:30.122 INFO:tasks.ceph.mds.d.smithi156.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55ff4cbd048b] 2023-11-28T17:12:30.123 INFO:tasks.ceph.mds.d.smithi156.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4cbd097c] 2023-11-28T17:12:30.123 INFO:tasks.ceph.mds.d.smithi156.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55ff4cbba4b5] 2023-11-28T17:12:30.123 INFO:tasks.ceph.mds.d.smithi156.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7f7479f6040a] 2023-11-28T17:12:30.123 INFO:tasks.ceph.mds.d.smithi156.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f7479ff24a1] 2023-11-28T17:12:30.123 INFO:tasks.ceph.mds.d.smithi156.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7f747949f802] 2023-11-28T17:12:30.124 INFO:tasks.ceph.mds.d.smithi156.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7f747943f450] 2023-11-28T17:12:30.124 INFO:tasks.ceph.mds.d.smithi156.stderr: 2023-11-28T17:12:30.124 INFO:tasks.ceph.mds.d.smithi156.stderr:*** Caught signal (Aborted) ** 2023-11-28T17:12:30.124 INFO:tasks.ceph.mds.d.smithi156.stderr: in thread 7f7474190640 thread_name:ms_dispatch 2023-11-28T17:12:30.125 INFO:tasks.ceph.mds.d.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:30.125 INFO:tasks.ceph.mds.d.smithi156.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f7479454db0] 2023-11-28T17:12:30.126 INFO:tasks.ceph.mds.d.smithi156.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f74794a154c] 2023-11-28T17:12:30.126 INFO:tasks.ceph.mds.d.smithi156.stderr: 3: raise() 2023-11-28T17:12:30.126 INFO:tasks.ceph.mds.d.smithi156.stderr: 4: abort() 2023-11-28T17:12:30.126 INFO:tasks.ceph.mds.d.smithi156.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f7479d6ef52] 2023-11-28T17:12:30.127 INFO:tasks.ceph.mds.d.smithi156.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7479d6f0b6] 2023-11-28T17:12:30.127 INFO:tasks.ceph.mds.d.smithi156.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55ff4cd85329] 2023-11-28T17:12:30.127 INFO:tasks.ceph.mds.d.smithi156.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55ff4cd7a5b5] 2023-11-28T17:12:30.127 INFO:tasks.ceph.mds.d.smithi156.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55ff4cd2985b] 2023-11-28T17:12:30.127 INFO:tasks.ceph.mds.d.smithi156.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55ff4cc63521] 2023-11-28T17:12:30.127 INFO:tasks.ceph.mds.d.smithi156.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55ff4cc6a508] 2023-11-28T17:12:30.127 INFO:tasks.ceph.mds.d.smithi156.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55ff4cc5fedd] 2023-11-28T17:12:30.127 INFO:tasks.ceph.mds.d.smithi156.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55ff4cc607bc] 2023-11-28T17:12:30.127 INFO:tasks.ceph.mds.d.smithi156.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4cc4c467] 2023-11-28T17:12:30.127 INFO:tasks.ceph.mds.d.smithi156.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55ff4cbd048b] 2023-11-28T17:12:30.127 INFO:tasks.ceph.mds.d.smithi156.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4cbd097c] 2023-11-28T17:12:30.128 INFO:tasks.ceph.mds.d.smithi156.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55ff4cbba4b5] 2023-11-28T17:12:30.128 INFO:tasks.ceph.mds.d.smithi156.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7f7479f6040a] 2023-11-28T17:12:30.128 INFO:tasks.ceph.mds.d.smithi156.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f7479ff24a1] 2023-11-28T17:12:30.128 INFO:tasks.ceph.mds.d.smithi156.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7f747949f802] 2023-11-28T17:12:30.128 INFO:tasks.ceph.mds.d.smithi156.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7f747943f450] 2023-11-28T17:12:30.128 INFO:tasks.ceph.mds.d.smithi156.stderr:2023-11-28T17:12:30.119+0000 7f7474190640 -1 *** Caught signal (Aborted) ** 2023-11-28T17:12:30.128 INFO:tasks.ceph.mds.d.smithi156.stderr: in thread 7f7474190640 thread_name:ms_dispatch 2023-11-28T17:12:30.128 INFO:tasks.ceph.mds.d.smithi156.stderr: 2023-11-28T17:12:30.128 INFO:tasks.ceph.mds.d.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:30.128 INFO:tasks.ceph.mds.d.smithi156.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f7479454db0] 2023-11-28T17:12:30.128 INFO:tasks.ceph.mds.d.smithi156.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f74794a154c] 2023-11-28T17:12:30.128 INFO:tasks.ceph.mds.d.smithi156.stderr: 3: raise() 2023-11-28T17:12:30.129 INFO:tasks.ceph.mds.d.smithi156.stderr: 4: abort() 2023-11-28T17:12:30.129 INFO:tasks.ceph.mds.d.smithi156.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f7479d6ef52] 2023-11-28T17:12:30.129 INFO:tasks.ceph.mds.d.smithi156.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7479d6f0b6] 2023-11-28T17:12:30.129 INFO:tasks.ceph.mds.d.smithi156.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55ff4cd85329] 2023-11-28T17:12:30.129 INFO:tasks.ceph.mds.d.smithi156.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55ff4cd7a5b5] 2023-11-28T17:12:30.129 INFO:tasks.ceph.mds.d.smithi156.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55ff4cd2985b] 2023-11-28T17:12:30.129 INFO:tasks.ceph.mds.d.smithi156.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55ff4cc63521] 2023-11-28T17:12:30.129 INFO:tasks.ceph.mds.d.smithi156.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55ff4cc6a508] 2023-11-28T17:12:30.129 INFO:tasks.ceph.mds.d.smithi156.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55ff4cc5fedd] 2023-11-28T17:12:30.129 INFO:tasks.ceph.mds.d.smithi156.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55ff4cc607bc] 2023-11-28T17:12:30.129 INFO:tasks.ceph.mds.d.smithi156.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4cc4c467] 2023-11-28T17:12:30.130 INFO:tasks.ceph.mds.d.smithi156.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55ff4cbd048b] 2023-11-28T17:12:30.130 INFO:tasks.ceph.mds.d.smithi156.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4cbd097c] 2023-11-28T17:12:30.130 INFO:tasks.ceph.mds.d.smithi156.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55ff4cbba4b5] 2023-11-28T17:12:30.130 INFO:tasks.ceph.mds.d.smithi156.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7f7479f6040a] 2023-11-28T17:12:30.130 INFO:tasks.ceph.mds.d.smithi156.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f7479ff24a1] 2023-11-28T17:12:30.130 INFO:tasks.ceph.mds.d.smithi156.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7f747949f802] 2023-11-28T17:12:30.130 INFO:tasks.ceph.mds.d.smithi156.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7f747943f450] 2023-11-28T17:12:30.130 INFO:tasks.ceph.mds.d.smithi156.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T17:12:30.130 INFO:tasks.ceph.mds.d.smithi156.stderr: 2023-11-28T17:12:30.141 INFO:tasks.ceph.mds.d.smithi156.stderr: -1> 2023-11-28T17:12:30.115+0000 7f7474190640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f7474190640 time 2023-11-28T17:12:30.115592+0000 2023-11-28T17:12:30.141 INFO:tasks.ceph.mds.d.smithi156.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:12:30.141 INFO:tasks.ceph.mds.d.smithi156.stderr: 2023-11-28T17:12:30.142 INFO:tasks.ceph.mds.d.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:30.142 INFO:tasks.ceph.mds.d.smithi156.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f7479d6eef8] 2023-11-28T17:12:30.142 INFO:tasks.ceph.mds.d.smithi156.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7479d6f0b6] 2023-11-28T17:12:30.142 INFO:tasks.ceph.mds.d.smithi156.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55ff4cd85329] 2023-11-28T17:12:30.142 INFO:tasks.ceph.mds.d.smithi156.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55ff4cd7a5b5] 2023-11-28T17:12:30.142 INFO:tasks.ceph.mds.d.smithi156.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55ff4cd2985b] 2023-11-28T17:12:30.142 INFO:tasks.ceph.mds.d.smithi156.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55ff4cc63521] 2023-11-28T17:12:30.142 INFO:tasks.ceph.mds.d.smithi156.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55ff4cc6a508] 2023-11-28T17:12:30.142 INFO:tasks.ceph.mds.d.smithi156.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55ff4cc5fedd] 2023-11-28T17:12:30.142 INFO:tasks.ceph.mds.d.smithi156.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55ff4cc607bc] 2023-11-28T17:12:30.142 INFO:tasks.ceph.mds.d.smithi156.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4cc4c467] 2023-11-28T17:12:30.142 INFO:tasks.ceph.mds.d.smithi156.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55ff4cbd048b] 2023-11-28T17:12:30.143 INFO:tasks.ceph.mds.d.smithi156.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4cbd097c] 2023-11-28T17:12:30.143 INFO:tasks.ceph.mds.d.smithi156.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55ff4cbba4b5] 2023-11-28T17:12:30.143 INFO:tasks.ceph.mds.d.smithi156.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7f7479f6040a] 2023-11-28T17:12:30.143 INFO:tasks.ceph.mds.d.smithi156.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f7479ff24a1] 2023-11-28T17:12:30.143 INFO:tasks.ceph.mds.d.smithi156.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7f747949f802] 2023-11-28T17:12:30.143 INFO:tasks.ceph.mds.d.smithi156.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7f747943f450] 2023-11-28T17:12:30.143 INFO:tasks.ceph.mds.d.smithi156.stderr: 2023-11-28T17:12:30.143 INFO:tasks.ceph.mds.d.smithi156.stderr: 0> 2023-11-28T17:12:30.119+0000 7f7474190640 -1 *** Caught signal (Aborted) ** 2023-11-28T17:12:30.143 INFO:tasks.ceph.mds.d.smithi156.stderr: in thread 7f7474190640 thread_name:ms_dispatch 2023-11-28T17:12:30.143 INFO:tasks.ceph.mds.d.smithi156.stderr: 2023-11-28T17:12:30.143 INFO:tasks.ceph.mds.d.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:30.144 INFO:tasks.ceph.mds.d.smithi156.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f7479454db0] 2023-11-28T17:12:30.144 INFO:tasks.ceph.mds.d.smithi156.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f74794a154c] 2023-11-28T17:12:30.144 INFO:tasks.ceph.mds.d.smithi156.stderr: 3: raise() 2023-11-28T17:12:30.144 INFO:tasks.ceph.mds.d.smithi156.stderr: 4: abort() 2023-11-28T17:12:30.144 INFO:tasks.ceph.mds.d.smithi156.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f7479d6ef52] 2023-11-28T17:12:30.145 INFO:tasks.ceph.mds.d.smithi156.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7479d6f0b6] 2023-11-28T17:12:30.145 INFO:tasks.ceph.mds.d.smithi156.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55ff4cd85329] 2023-11-28T17:12:30.145 INFO:tasks.ceph.mds.d.smithi156.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55ff4cd7a5b5] 2023-11-28T17:12:30.145 INFO:tasks.ceph.mds.d.smithi156.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55ff4cd2985b] 2023-11-28T17:12:30.145 INFO:tasks.ceph.mds.d.smithi156.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55ff4cc63521] 2023-11-28T17:12:30.145 INFO:tasks.ceph.mds.d.smithi156.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55ff4cc6a508] 2023-11-28T17:12:30.145 INFO:tasks.ceph.mds.d.smithi156.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55ff4cc5fedd] 2023-11-28T17:12:30.145 INFO:tasks.ceph.mds.d.smithi156.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55ff4cc607bc] 2023-11-28T17:12:30.146 INFO:tasks.ceph.mds.d.smithi156.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4cc4c467] 2023-11-28T17:12:30.146 INFO:tasks.ceph.mds.d.smithi156.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55ff4cbd048b] 2023-11-28T17:12:30.146 INFO:tasks.ceph.mds.d.smithi156.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4cbd097c] 2023-11-28T17:12:30.146 INFO:tasks.ceph.mds.d.smithi156.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55ff4cbba4b5] 2023-11-28T17:12:30.146 INFO:tasks.ceph.mds.d.smithi156.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7f7479f6040a] 2023-11-28T17:12:30.146 INFO:tasks.ceph.mds.d.smithi156.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f7479ff24a1] 2023-11-28T17:12:30.146 INFO:tasks.ceph.mds.d.smithi156.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7f747949f802] 2023-11-28T17:12:30.146 INFO:tasks.ceph.mds.d.smithi156.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7f747943f450] 2023-11-28T17:12:30.146 INFO:tasks.ceph.mds.d.smithi156.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T17:12:30.146 INFO:tasks.ceph.mds.d.smithi156.stderr: 2023-11-28T17:12:30.156 INFO:tasks.ceph.mds.d.smithi156.stderr: -1> 2023-11-28T17:12:30.115+0000 7f7474190640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: In function 'bool Locker::xlock_start(SimpleLock*, MDRequestRef&)' thread 7f7474190640 time 2023-11-28T17:12:30.115592+0000 2023-11-28T17:12:30.156 INFO:tasks.ceph.mds.d.smithi156.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7531-g47c1c547/rpm/el9/BUILD/ceph-18.0.0-7531-g47c1c547/src/mds/Locker.cc: 1963: FAILED ceph_assert(lock->get_sm()->can_remote_xlock) 2023-11-28T17:12:30.156 INFO:tasks.ceph.mds.d.smithi156.stderr: 2023-11-28T17:12:30.157 INFO:tasks.ceph.mds.d.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:30.157 INFO:tasks.ceph.mds.d.smithi156.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f7479d6eef8] 2023-11-28T17:12:30.157 INFO:tasks.ceph.mds.d.smithi156.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7479d6f0b6] 2023-11-28T17:12:30.157 INFO:tasks.ceph.mds.d.smithi156.stderr: 3: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55ff4cd85329] 2023-11-28T17:12:30.157 INFO:tasks.ceph.mds.d.smithi156.stderr: 4: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55ff4cd7a5b5] 2023-11-28T17:12:30.157 INFO:tasks.ceph.mds.d.smithi156.stderr: 5: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55ff4cd2985b] 2023-11-28T17:12:30.157 INFO:tasks.ceph.mds.d.smithi156.stderr: 6: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55ff4cc63521] 2023-11-28T17:12:30.158 INFO:tasks.ceph.mds.d.smithi156.stderr: 7: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55ff4cc6a508] 2023-11-28T17:12:30.158 INFO:tasks.ceph.mds.d.smithi156.stderr: 8: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55ff4cc5fedd] 2023-11-28T17:12:30.158 INFO:tasks.ceph.mds.d.smithi156.stderr: 9: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55ff4cc607bc] 2023-11-28T17:12:30.158 INFO:tasks.ceph.mds.d.smithi156.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4cc4c467] 2023-11-28T17:12:30.158 INFO:tasks.ceph.mds.d.smithi156.stderr: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55ff4cbd048b] 2023-11-28T17:12:30.158 INFO:tasks.ceph.mds.d.smithi156.stderr: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4cbd097c] 2023-11-28T17:12:30.158 INFO:tasks.ceph.mds.d.smithi156.stderr: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55ff4cbba4b5] 2023-11-28T17:12:30.158 INFO:tasks.ceph.mds.d.smithi156.stderr: 14: (DispatchQueue::entry()+0x53a) [0x7f7479f6040a] 2023-11-28T17:12:30.158 INFO:tasks.ceph.mds.d.smithi156.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f7479ff24a1] 2023-11-28T17:12:30.158 INFO:tasks.ceph.mds.d.smithi156.stderr: 16: /lib64/libc.so.6(+0x9f802) [0x7f747949f802] 2023-11-28T17:12:30.158 INFO:tasks.ceph.mds.d.smithi156.stderr: 17: /lib64/libc.so.6(+0x3f450) [0x7f747943f450] 2023-11-28T17:12:30.159 INFO:tasks.ceph.mds.d.smithi156.stderr: 2023-11-28T17:12:30.159 INFO:tasks.ceph.mds.d.smithi156.stderr: 0> 2023-11-28T17:12:30.119+0000 7f7474190640 -1 *** Caught signal (Aborted) ** 2023-11-28T17:12:30.159 INFO:tasks.ceph.mds.d.smithi156.stderr: in thread 7f7474190640 thread_name:ms_dispatch 2023-11-28T17:12:30.159 INFO:tasks.ceph.mds.d.smithi156.stderr: 2023-11-28T17:12:30.159 INFO:tasks.ceph.mds.d.smithi156.stderr: ceph version 18.0.0-7531-g47c1c547 (47c1c547372ed5bff7984647dc200bac0e019caf) reef (dev) 2023-11-28T17:12:30.159 INFO:tasks.ceph.mds.d.smithi156.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f7479454db0] 2023-11-28T17:12:30.159 INFO:tasks.ceph.mds.d.smithi156.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f74794a154c] 2023-11-28T17:12:30.159 INFO:tasks.ceph.mds.d.smithi156.stderr: 3: raise() 2023-11-28T17:12:30.159 INFO:tasks.ceph.mds.d.smithi156.stderr: 4: abort() 2023-11-28T17:12:30.159 INFO:tasks.ceph.mds.d.smithi156.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f7479d6ef52] 2023-11-28T17:12:30.159 INFO:tasks.ceph.mds.d.smithi156.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7479d6f0b6] 2023-11-28T17:12:30.160 INFO:tasks.ceph.mds.d.smithi156.stderr: 7: (Locker::xlock_start(SimpleLock*, boost::intrusive_ptr&)+0x939) [0x55ff4cd85329] 2023-11-28T17:12:30.160 INFO:tasks.ceph.mds.d.smithi156.stderr: 8: (Locker::acquire_locks(boost::intrusive_ptr&, MutationImpl::LockOpVec&, CInode*, bool)+0x20f5) [0x55ff4cd7a5b5] 2023-11-28T17:12:30.160 INFO:tasks.ceph.mds.d.smithi156.stderr: 9: (MDCache::path_traverse(boost::intrusive_ptr&, ContextFactory&, filepath const&, int, std::vector >*, CInode**)+0x79b) [0x55ff4cd2985b] 2023-11-28T17:12:30.160 INFO:tasks.ceph.mds.d.smithi156.stderr: 10: (Server::rdlock_path_xlock_dentry(boost::intrusive_ptr&, bool, bool, bool, bool)+0x371) [0x55ff4cc63521] 2023-11-28T17:12:30.160 INFO:tasks.ceph.mds.d.smithi156.stderr: 11: (Server::handle_client_openc(boost::intrusive_ptr&)+0xd8) [0x55ff4cc6a508] 2023-11-28T17:12:30.160 INFO:tasks.ceph.mds.d.smithi156.stderr: 12: (Server::handle_peer_auth_pin_ack(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x83d) [0x55ff4cc5fedd] 2023-11-28T17:12:30.160 INFO:tasks.ceph.mds.d.smithi156.stderr: 13: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6bc) [0x55ff4cc607bc] 2023-11-28T17:12:30.160 INFO:tasks.ceph.mds.d.smithi156.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ff4cc4c467] 2023-11-28T17:12:30.160 INFO:tasks.ceph.mds.d.smithi156.stderr: 15: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55ff4cbd048b] 2023-11-28T17:12:30.160 INFO:tasks.ceph.mds.d.smithi156.stderr: 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ff4cbd097c] 2023-11-28T17:12:30.161 INFO:tasks.ceph.mds.d.smithi156.stderr: 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x55ff4cbba4b5] 2023-11-28T17:12:30.161 INFO:tasks.ceph.mds.d.smithi156.stderr: 18: (DispatchQueue::entry()+0x53a) [0x7f7479f6040a] 2023-11-28T17:12:30.161 INFO:tasks.ceph.mds.d.smithi156.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x3f24a1) [0x7f7479ff24a1] 2023-11-28T17:12:30.161 INFO:tasks.ceph.mds.d.smithi156.stderr: 20: /lib64/libc.so.6(+0x9f802) [0x7f747949f802] 2023-11-28T17:12:30.161 INFO:tasks.ceph.mds.d.smithi156.stderr: 21: /lib64/libc.so.6(+0x3f450) [0x7f747943f450] 2023-11-28T17:12:30.161 INFO:tasks.ceph.mds.d.smithi156.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-28T17:12:30.161 INFO:tasks.ceph.mds.d.smithi156.stderr: 2023-11-28T17:12:30.332 INFO:tasks.ceph.mds.d.smithi156.stderr:daemon-helper: command crashed with signal 6 2023-11-28T17:12:34.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~39s 2023-11-28T17:12:34.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~20s 2023-11-28T17:12:34.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-11-28T17:12:41.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~46s 2023-11-28T17:12:41.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~26s 2023-11-28T17:12:41.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~6s 2023-11-28T17:12:47.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~52s 2023-11-28T17:12:47.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~32s 2023-11-28T17:12:47.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~13s 2023-11-28T17:12:53.428 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:12:53.438 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:12:53.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~59s 2023-11-28T17:12:53.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~39s 2023-11-28T17:12:53.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~19s 2023-11-28T17:13:00.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~65s 2023-11-28T17:13:00.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~45s 2023-11-28T17:13:00.397 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~26s 2023-11-28T17:13:06.808 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~71s 2023-11-28T17:13:06.808 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~52s 2023-11-28T17:13:06.808 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~32s 2023-11-28T17:13:13.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~78s 2023-11-28T17:13:13.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~58s 2023-11-28T17:13:13.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~38s 2023-11-28T17:13:19.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~84s 2023-11-28T17:13:19.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~64s 2023-11-28T17:13:19.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~45s 2023-11-28T17:13:23.574 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:13:23.584 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:13:26.045 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~91s 2023-11-28T17:13:26.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~71s 2023-11-28T17:13:26.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~51s 2023-11-28T17:13:32.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~97s 2023-11-28T17:13:32.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~77s 2023-11-28T17:13:32.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~58s 2023-11-28T17:13:38.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~104s 2023-11-28T17:13:38.868 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~84s 2023-11-28T17:13:38.868 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~64s 2023-11-28T17:13:45.276 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~110s 2023-11-28T17:13:45.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~90s 2023-11-28T17:13:45.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~71s 2023-11-28T17:13:51.685 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~116s 2023-11-28T17:13:51.686 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~96s 2023-11-28T17:13:51.686 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~77s 2023-11-28T17:13:53.722 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:13:53.732 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:13:58.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~123s 2023-11-28T17:13:58.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~103s 2023-11-28T17:13:58.098 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~83s 2023-11-28T17:14:04.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~129s 2023-11-28T17:14:04.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~109s 2023-11-28T17:14:04.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~90s 2023-11-28T17:14:10.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~136s 2023-11-28T17:14:10.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~116s 2023-11-28T17:14:10.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~96s 2023-11-28T17:14:17.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~142s 2023-11-28T17:14:17.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~122s 2023-11-28T17:14:17.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~103s 2023-11-28T17:14:23.747 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~148s 2023-11-28T17:14:23.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~129s 2023-11-28T17:14:23.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~109s 2023-11-28T17:14:23.876 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:14:23.886 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:14:30.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~155s 2023-11-28T17:14:30.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~135s 2023-11-28T17:14:30.159 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~115s 2023-11-28T17:14:36.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~161s 2023-11-28T17:14:36.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~141s 2023-11-28T17:14:36.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~122s 2023-11-28T17:14:42.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~168s 2023-11-28T17:14:42.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~148s 2023-11-28T17:14:42.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~128s 2023-11-28T17:14:49.394 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~174s 2023-11-28T17:14:49.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~154s 2023-11-28T17:14:49.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~135s 2023-11-28T17:14:54.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:14:54.041 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:14:55.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~180s 2023-11-28T17:14:55.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~161s 2023-11-28T17:14:55.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~141s 2023-11-28T17:15:02.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~187s 2023-11-28T17:15:02.218 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~167s 2023-11-28T17:15:02.218 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~147s 2023-11-28T17:15:08.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~193s 2023-11-28T17:15:08.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~173s 2023-11-28T17:15:08.630 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~154s 2023-11-28T17:15:15.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~200s 2023-11-28T17:15:15.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~180s 2023-11-28T17:15:15.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~160s 2023-11-28T17:15:21.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~206s 2023-11-28T17:15:21.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~186s 2023-11-28T17:15:21.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~167s 2023-11-28T17:15:24.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:15:24.464 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:15:27.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~212s 2023-11-28T17:15:27.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~193s 2023-11-28T17:15:27.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~173s 2023-11-28T17:15:34.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~219s 2023-11-28T17:15:34.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~199s 2023-11-28T17:15:34.277 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~180s 2023-11-28T17:15:40.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~225s 2023-11-28T17:15:40.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~205s 2023-11-28T17:15:40.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~186s 2023-11-28T17:15:47.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~232s 2023-11-28T17:15:47.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~212s 2023-11-28T17:15:47.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~192s 2023-11-28T17:15:53.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~238s 2023-11-28T17:15:53.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~218s 2023-11-28T17:15:53.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~199s 2023-11-28T17:15:54.693 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:15:54.702 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:15:59.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~245s 2023-11-28T17:15:59.923 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~225s 2023-11-28T17:15:59.923 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~205s 2023-11-28T17:16:06.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~251s 2023-11-28T17:16:06.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~231s 2023-11-28T17:16:06.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~212s 2023-11-28T17:16:12.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~257s 2023-11-28T17:16:12.748 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~238s 2023-11-28T17:16:12.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~218s 2023-11-28T17:16:19.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~264s 2023-11-28T17:16:19.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~244s 2023-11-28T17:16:19.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~224s 2023-11-28T17:16:24.864 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:16:24.873 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:16:25.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~270s 2023-11-28T17:16:25.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~250s 2023-11-28T17:16:25.697 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~231s 2023-11-28T17:16:32.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~277s 2023-11-28T17:16:32.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~257s 2023-11-28T17:16:32.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~237s 2023-11-28T17:16:38.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~283s 2023-11-28T17:16:38.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~263s 2023-11-28T17:16:38.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~244s 2023-11-28T17:16:44.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~290s 2023-11-28T17:16:44.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~270s 2023-11-28T17:16:44.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~250s 2023-11-28T17:16:48.812 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2023-11-28T17:16:48.813 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2023-11-28T17:16:51.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~296s 2023-11-28T17:16:51.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~276s 2023-11-28T17:16:51.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~257s 2023-11-28T17:16:55.057 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:16:55.066 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:16:57.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~302s 2023-11-28T17:16:57.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~283s 2023-11-28T17:16:57.745 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~263s 2023-11-28T17:16:57.745 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-11-28T17:16:57.745 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:16:57.745 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:16:57.806 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:16:57.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:16:57.813 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-28T17:16:57.813 DEBUG:teuthology.orchestra.run.smithi116:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T17:16:57.917 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-28T17:16:58.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.002+0000 7fa4f3a6b640 1 Processor -- start 2023-11-28T17:16:58.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.002+0000 7fa4f3a6b640 1 -- start start 2023-11-28T17:16:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.003+0000 7fa4f3a6b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa4ec144430 0x7fa4ec15f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.003+0000 7fa4f3a6b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0703f0 0x7fa4ec071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.003+0000 7fa4f3a6b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4ec070bd0 con 0x7fa4ec0721b0 2023-11-28T17:16:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.003+0000 7fa4f3a6b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ec144810 con 0x7fa4ec144430 2023-11-28T17:16:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.003+0000 7fa4f3a6b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec070eb0 con 0x7fa4ec0703f0 2023-11-28T17:16:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.004+0000 7fa4f2008640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa4ec0721b0 0x7fa4ec06c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:32898/0 (socket says 172.21.15.116:32898) 2023-11-28T17:16:58.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.004+0000 7fa4f2008640 1 -- 172.21.15.116:0/2502566672 learned_addr learned my addr 172.21.15.116:0/2502566672 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:16:58.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.004+0000 7fa4f1807640 1 --2- 172.21.15.116:0/2502566672 >> v2:172.21.15.156:3300/0 conn(0x7fa4ec144430 0x7fa4ec15f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.004+0000 7fa4f1006640 1 --2- 172.21.15.116:0/2502566672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0703f0 0x7fa4ec071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.004+0000 7fa4f0805640 1 -- 172.21.15.116:0/2502566672 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058824218 0 0) 0x7fa4ec070bd0 con 0x7fa4ec0721b0 2023-11-28T17:16:58.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.004+0000 7fa4f0805640 1 -- 172.21.15.116:0/2502566672 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d8003520 con 0x7fa4ec0721b0 2023-11-28T17:16:58.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f0805640 1 -- 172.21.15.116:0/2502566672 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3297283842 0 0) 0x7fa4d8003520 con 0x7fa4ec0721b0 2023-11-28T17:16:58.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f0805640 1 -- 172.21.15.116:0/2502566672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0703f0 msgr2=0x7fa4ec071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:58.009 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f0805640 1 --2- 172.21.15.116:0/2502566672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0703f0 0x7fa4ec071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f0805640 1 -- 172.21.15.116:0/2502566672 >> v2:172.21.15.156:3300/0 conn(0x7fa4ec144430 msgr2=0x7fa4ec15f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:58.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f0805640 1 --2- 172.21.15.116:0/2502566672 >> v2:172.21.15.156:3300/0 conn(0x7fa4ec144430 0x7fa4ec15f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f0805640 1 -- 172.21.15.116:0/2502566672 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec071190 con 0x7fa4ec0721b0 2023-11-28T17:16:58.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f0805640 1 -- 172.21.15.116:0/2502566672 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fa4e40019f0 con 0x7fa4ec0721b0 2023-11-28T17:16:58.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f0805640 1 -- 172.21.15.116:0/2502566672 >> v1:172.21.15.116:6789/0 conn(0x7fa4ec0721b0 legacy=0x7fa4ec06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f0805640 1 --2- 172.21.15.116:0/2502566672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0703f0 0x7fa4d8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f0805640 1 --2- 172.21.15.116:0/2502566672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4d8003ef0 0x7fa4d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f0805640 1 --2- 172.21.15.116:0/2502566672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4d8004d00 0x7fa4d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.010 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f0805640 1 -- 172.21.15.116:0/2502566672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec144810 con 0x7fa4d8004d00 2023-11-28T17:16:58.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f0805640 1 -- 172.21.15.116:0/2502566672 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec070eb0 con 0x7fa4d8003ef0 2023-11-28T17:16:58.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f0805640 1 -- 172.21.15.116:0/2502566672 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d800b980 con 0x7fa4ec0703f0 2023-11-28T17:16:58.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f2008640 1 --2- 172.21.15.116:0/2502566672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4d8004d00 0x7fa4d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f1807640 1 --2- 172.21.15.116:0/2502566672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0703f0 0x7fa4d8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f2008640 1 -- 172.21.15.116:0/2502566672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0703f0 msgr2=0x7fa4d8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:58.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f2008640 1 --2- 172.21.15.116:0/2502566672 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0703f0 0x7fa4d8003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f2008640 1 -- 172.21.15.116:0/2502566672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4d8003ef0 msgr2=0x7fa4d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:58.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f2008640 1 --2- 172.21.15.116:0/2502566672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4d8003ef0 0x7fa4d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.011 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.005+0000 7fa4f2008640 1 -- 172.21.15.116:0/2502566672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d8003520 con 0x7fa4d8004d00 2023-11-28T17:16:58.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.006+0000 7fa4f2008640 1 --2- 172.21.15.116:0/2502566672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4d8004d00 0x7fa4d8005180 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400a830 tx=0x7fa4d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce97fcebfe5f449f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.007+0000 7fa4f0805640 1 -- 172.21.15.116:0/2502566672 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e402cea0 con 0x7fa4d8004d00 2023-11-28T17:16:58.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.007+0000 7fa4f0805640 1 -- 172.21.15.116:0/2502566672 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e402f070 con 0x7fa4d8004d00 2023-11-28T17:16:58.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.007+0000 7fa4f0805640 1 -- 172.21.15.116:0/2502566672 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e40348d0 con 0x7fa4d8004d00 2023-11-28T17:16:58.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.007+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2502566672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4d8004d00 msgr2=0x7fa4d8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.007+0000 7fa4f3a6b640 1 --2- 172.21.15.116:0/2502566672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4d8004d00 0x7fa4d8005180 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400a830 tx=0x7fa4d80057f0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.007+0000 7fa4f2008640 1 -- 172.21.15.116:0/2502566672 reap_dead start 2023-11-28T17:16:58.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.007+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2502566672 shutdown_connections 2023-11-28T17:16:58.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.007+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2502566672 >> 172.21.15.116:0/2502566672 conn(0x7fa4ec06a4c0 msgr2=0x7fa4ec06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:58.012 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.007+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2502566672 shutdown_connections 2023-11-28T17:16:58.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.007+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2502566672 wait complete. 2023-11-28T17:16:58.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.009+0000 7fa4f3a6b640 1 Processor -- start 2023-11-28T17:16:58.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.009+0000 7fa4f3a6b640 1 -- start start 2023-11-28T17:16:58.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.010+0000 7fa4f3a6b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4d8004d00 0x7fa4ec13d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.010+0000 7fa4f3a6b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec15a840 0x7fa4ec15acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.010+0000 7fa4f3a6b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0c65e0 0x7fa4ec0c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.010+0000 7fa4f3a6b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec0c5930 con 0x7fa4ec15a840 2023-11-28T17:16:58.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.010+0000 7fa4f3a6b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec0c54a0 con 0x7fa4d8004d00 2023-11-28T17:16:58.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.010+0000 7fa4f3a6b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec15b200 con 0x7fa4ec0c65e0 2023-11-28T17:16:58.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.010+0000 7fa4f1006640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec15a840 0x7fa4ec15acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.011+0000 7fa4f1006640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec15a840 0x7fa4ec15acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53474/0 (socket says 172.21.15.116:53474) 2023-11-28T17:16:58.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.011+0000 7fa4f1006640 1 -- 172.21.15.116:0/2623956119 learned_addr learned my addr 172.21.15.116:0/2623956119 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:58.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.011+0000 7fa4f2008640 1 --2- 172.21.15.116:0/2623956119 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0c65e0 0x7fa4ec0c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.011+0000 7fa4f1807640 1 --2- 172.21.15.116:0/2623956119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4d8004d00 0x7fa4ec13d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.011+0000 7fa4f1006640 1 -- 172.21.15.116:0/2623956119 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0c65e0 msgr2=0x7fa4ec0c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.011+0000 7fa4f1006640 1 --2- 172.21.15.116:0/2623956119 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0c65e0 0x7fa4ec0c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.011+0000 7fa4f1006640 1 -- 172.21.15.116:0/2623956119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4d8004d00 msgr2=0x7fa4ec13d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.011+0000 7fa4f1006640 1 --2- 172.21.15.116:0/2623956119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4d8004d00 0x7fa4ec13d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.011+0000 7fa4f1006640 1 -- 172.21.15.116:0/2623956119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec0c4890 con 0x7fa4ec15a840 2023-11-28T17:16:58.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.011+0000 7fa4f1006640 1 --2- 172.21.15.116:0/2623956119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec15a840 0x7fa4ec15acc0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc00e7c0 tx=0x7fa4dc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.011+0000 7fa4e27fc640 1 -- 172.21.15.116:0/2623956119 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc00eed0 con 0x7fa4ec15a840 2023-11-28T17:16:58.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.011+0000 7fa4e27fc640 1 -- 172.21.15.116:0/2623956119 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4dc00c8c0 con 0x7fa4ec15a840 2023-11-28T17:16:58.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.011+0000 7fa4e27fc640 1 -- 172.21.15.116:0/2623956119 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc00cc70 con 0x7fa4ec15a840 2023-11-28T17:16:58.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.012+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2623956119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ec0c4b70 con 0x7fa4ec15a840 2023-11-28T17:16:58.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.012+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2623956119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ec0c5130 con 0x7fa4ec15a840 2023-11-28T17:16:58.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.013+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2623956119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ec0c5630 con 0x7fa4ec15a840 2023-11-28T17:16:58.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.014+0000 7fa4e27fc640 1 -- 172.21.15.116:0/2623956119 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fa4dc013470 con 0x7fa4ec15a840 2023-11-28T17:16:58.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.015+0000 7fa4e27fc640 1 --2- 172.21.15.116:0/2623956119 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa4d006b2a0 0x7fa4d006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.015+0000 7fa4e27fc640 1 -- 172.21.15.116:0/2623956119 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fa4dc089e80 con 0x7fa4ec15a840 2023-11-28T17:16:58.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.015+0000 7fa4f1807640 1 --2- 172.21.15.116:0/2623956119 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa4d006b2a0 0x7fa4d006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.016+0000 7fa4f1807640 1 --2- 172.21.15.116:0/2623956119 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa4d006b2a0 0x7fa4d006d760 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8009990 tx=0x7fa4e8008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.021 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.020+0000 7fa4e27fc640 1 -- 172.21.15.116:0/2623956119 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4dc054b10 con 0x7fa4ec15a840 2023-11-28T17:16:58.040 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-28T17:16:58.040 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:16:58.040 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:16:58.083 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:16:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.215+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2623956119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4ec0693d0 con 0x7fa4ec15a840 2023-11-28T17:16:58.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.217+0000 7fa4e27fc640 1 -- 172.21.15.116:0/2623956119 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+17073 (secure 0 0 0) 0x7fa4dc017090 con 0x7fa4ec15a840 2023-11-28T17:16:58.218 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:16:58.219 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":52,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:12:26.038218+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:11:21.518078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T17:11:22.672451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":0}]},{"pgid":"7.8","mappings":[{"from":5,"to":7}]},{"pgid":"7.1b","mappings":[{"from":5,"to":7}]},{"pgid":"7.30","mappings":[{"from":5,"to":7}]},{"pgid":"7.31","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6833/372407661":"2023-11-29T17:12:26.038071+0000","172.21.15.156:6832/372407661":"2023-11-29T17:12:26.038071+0000","172.21.15.116:6835/3251347229":"2023-11-29T17:12:06.030424+0000","172.21.15.116:6834/3251347229":"2023-11-29T17:12:06.030424+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:16:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.219+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2623956119 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa4d006b2a0 msgr2=0x7fa4d006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.219+0000 7fa4f3a6b640 1 --2- 172.21.15.116:0/2623956119 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa4d006b2a0 0x7fa4d006d760 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8009990 tx=0x7fa4e8008040 comp rx=0 tx=0).stop 2023-11-28T17:16:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.219+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2623956119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec15a840 msgr2=0x7fa4ec15acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.219+0000 7fa4f3a6b640 1 --2- 172.21.15.116:0/2623956119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec15a840 0x7fa4ec15acc0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc00e7c0 tx=0x7fa4dc00ec80 comp rx=0 tx=0).stop 2023-11-28T17:16:58.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.219+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2623956119 shutdown_connections 2023-11-28T17:16:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.219+0000 7fa4f3a6b640 1 --2- 172.21.15.116:0/2623956119 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fa4d006b2a0 0x7fa4d006d760 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.219+0000 7fa4f3a6b640 1 --2- 172.21.15.116:0/2623956119 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa4ec0c65e0 0x7fa4ec0c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.219+0000 7fa4f3a6b640 1 --2- 172.21.15.116:0/2623956119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec15a840 0x7fa4ec15acc0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.219+0000 7fa4f3a6b640 1 --2- 172.21.15.116:0/2623956119 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4d8004d00 0x7fa4ec13d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.219+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2623956119 >> 172.21.15.116:0/2623956119 conn(0x7fa4ec06a4c0 msgr2=0x7fa4ec06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.220+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2623956119 shutdown_connections 2023-11-28T17:16:58.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.220+0000 7fa4f3a6b640 1 -- 172.21.15.116:0/2623956119 wait complete. 2023-11-28T17:16:58.240 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:11:21.518078+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:11:22.672451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:16:58.241 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:16:58.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.325+0000 7f47b661e640 1 Processor -- start 2023-11-28T17:16:58.327 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.325+0000 7f47b661e640 1 -- start start 2023-11-28T17:16:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.326+0000 7f47b661e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f47b00d1780 0x7f47b00d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.326+0000 7f47b661e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47b00b20a0 0x7f47b00d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47b661e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f47b00b2ad0 0x7f47b00ce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47b661e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b00cdcb0 con 0x7f47b00b2ad0 2023-11-28T17:16:58.329 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47b661e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b00d20a0 con 0x7f47b00d1780 2023-11-28T17:16:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47b661e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b00d39e0 con 0x7f47b00b20a0 2023-11-28T17:16:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47b4bbb640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f47b00b2ad0 0x7f47b00ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47b4bbb640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f47b00b2ad0 0x7f47b00ce750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53476/0 (socket says 172.21.15.116:53476) 2023-11-28T17:16:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47b4bbb640 1 -- 172.21.15.116:0/4196221712 learned_addr learned my addr 172.21.15.116:0/4196221712 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47affff640 1 --2- 172.21.15.116:0/4196221712 >> v2:172.21.15.156:3300/0 conn(0x7f47b00d1780 0x7f47b00d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47af7fe640 1 --2- 172.21.15.116:0/4196221712 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47b00b20a0 0x7f47b00d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.330 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47b4bbb640 1 -- 172.21.15.116:0/4196221712 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47b00b20a0 msgr2=0x7f47b00d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47b4bbb640 1 --2- 172.21.15.116:0/4196221712 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47b00b20a0 0x7f47b00d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47b4bbb640 1 -- 172.21.15.116:0/4196221712 >> v2:172.21.15.156:3300/0 conn(0x7f47b00d1780 msgr2=0x7f47b00d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47b4bbb640 1 --2- 172.21.15.116:0/4196221712 >> v2:172.21.15.156:3300/0 conn(0x7f47b00d1780 0x7f47b00d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47b4bbb640 1 -- 172.21.15.116:0/4196221712 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b00d3cc0 con 0x7f47b00b2ad0 2023-11-28T17:16:58.331 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.327+0000 7f47affff640 1 --2- 172.21.15.116:0/4196221712 >> v2:172.21.15.156:3300/0 conn(0x7f47b00d1780 0x7f47b00d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:16:58.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.329+0000 7f47b4bbb640 1 --2- 172.21.15.116:0/4196221712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b00b2ad0 0x7f47b00ce750 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f47b008d370 tx=0x7f47a802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac0912045f1892 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.329+0000 7f47aeffd640 1 -- 172.21.15.116:0/4196221712 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a802c9e0 con 0x7f47b00b2ad0 2023-11-28T17:16:58.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.329+0000 7f47aeffd640 1 -- 172.21.15.116:0/4196221712 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a802cb40 con 0x7f47b00b2ad0 2023-11-28T17:16:58.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.329+0000 7f47aeffd640 1 -- 172.21.15.116:0/4196221712 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a80343f0 con 0x7f47b00b2ad0 2023-11-28T17:16:58.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.330+0000 7f47b661e640 1 -- 172.21.15.116:0/4196221712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b00b2ad0 msgr2=0x7f47b00ce750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.330+0000 7f47b661e640 1 --2- 172.21.15.116:0/4196221712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b00b2ad0 0x7f47b00ce750 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f47b008d370 tx=0x7f47a802be40 comp rx=0 tx=0).stop 2023-11-28T17:16:58.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.330+0000 7f47b661e640 1 -- 172.21.15.116:0/4196221712 shutdown_connections 2023-11-28T17:16:58.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.330+0000 7f47b661e640 1 --2- 172.21.15.116:0/4196221712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b00b2ad0 0x7f47b00ce750 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.330+0000 7f47b661e640 1 --2- 172.21.15.116:0/4196221712 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47b00b20a0 0x7f47b00d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.330+0000 7f47b661e640 1 --2- 172.21.15.116:0/4196221712 >> v2:172.21.15.156:3300/0 conn(0x7f47b00d1780 0x7f47b00d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.330+0000 7f47b661e640 1 -- 172.21.15.116:0/4196221712 >> 172.21.15.116:0/4196221712 conn(0x7f47b009c4e0 msgr2=0x7f47b00d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:58.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.330+0000 7f47b661e640 1 -- 172.21.15.116:0/4196221712 shutdown_connections 2023-11-28T17:16:58.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.330+0000 7f47b661e640 1 -- 172.21.15.116:0/4196221712 wait complete. 2023-11-28T17:16:58.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.331+0000 7f47b661e640 1 Processor -- start 2023-11-28T17:16:58.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.332+0000 7f47b661e640 1 -- start start 2023-11-28T17:16:58.335 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47b661e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47b00b20a0 0x7f47b00b6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47b661e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47b00b2ad0 0x7f47b00b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47b661e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b00d1780 0x7f47b00b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47b661e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b01f2c40 con 0x7f47b00d1780 2023-11-28T17:16:58.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47b661e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b00ccc90 con 0x7f47b00b20a0 2023-11-28T17:16:58.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47b661e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b00ccf30 con 0x7f47b00b2ad0 2023-11-28T17:16:58.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47b4bbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b00d1780 0x7f47b00b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47b4bbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b00d1780 0x7f47b00b0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53486/0 (socket says 172.21.15.116:53486) 2023-11-28T17:16:58.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47b4bbb640 1 -- 172.21.15.116:0/828704029 learned_addr learned my addr 172.21.15.116:0/828704029 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:58.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47affff640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47b00b20a0 0x7f47b00b6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47af7fe640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47b00b2ad0 0x7f47b00b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47b4bbb640 1 -- 172.21.15.116:0/828704029 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47b00b2ad0 msgr2=0x7f47b00b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47b4bbb640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47b00b2ad0 0x7f47b00b0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47b4bbb640 1 -- 172.21.15.116:0/828704029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47b00b20a0 msgr2=0x7f47b00b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47b4bbb640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47b00b20a0 0x7f47b00b6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.333+0000 7f47b4bbb640 1 -- 172.21.15.116:0/828704029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a8009b30 con 0x7f47b00d1780 2023-11-28T17:16:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.334+0000 7f47af7fe640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47b00b2ad0 0x7f47b00b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:16:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.334+0000 7f47b4bbb640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b00d1780 0x7f47b00b0ed0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f47a8036040 tx=0x7f47a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.334+0000 7f47acff9640 1 -- 172.21.15.116:0/828704029 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a8009510 con 0x7f47b00d1780 2023-11-28T17:16:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.334+0000 7f47acff9640 1 -- 172.21.15.116:0/828704029 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a803e450 con 0x7f47b00d1780 2023-11-28T17:16:58.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.334+0000 7f47acff9640 1 -- 172.21.15.116:0/828704029 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a803e800 con 0x7f47b00d1780 2023-11-28T17:16:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.334+0000 7f47b661e640 1 -- 172.21.15.116:0/828704029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b00cd210 con 0x7f47b00d1780 2023-11-28T17:16:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.334+0000 7f47b661e640 1 -- 172.21.15.116:0/828704029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b00cd7d0 con 0x7f47b00d1780 2023-11-28T17:16:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.335+0000 7f47b661e640 1 -- 172.21.15.116:0/828704029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b00693d0 con 0x7f47b00d1780 2023-11-28T17:16:58.338 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.336+0000 7f47acff9640 1 -- 172.21.15.116:0/828704029 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f47a803e960 con 0x7f47b00d1780 2023-11-28T17:16:58.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.338+0000 7f47acff9640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f479c06b2a0 0x7f479c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.338+0000 7f47acff9640 1 -- 172.21.15.116:0/828704029 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f47a80acaf0 con 0x7f47b00d1780 2023-11-28T17:16:58.339 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.338+0000 7f47affff640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f479c06b2a0 0x7f479c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.340 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.339+0000 7f47affff640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f479c06b2a0 0x7f479c06d760 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f47a000a990 tx=0x7f47a0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.343 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.342+0000 7f47acff9640 1 -- 172.21.15.116:0/828704029 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47a8077810 con 0x7f47b00d1780 2023-11-28T17:16:58.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.455+0000 7f7a87c83640 1 Processor -- start 2023-11-28T17:16:58.456 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.455+0000 7f7a87c83640 1 -- start start 2023-11-28T17:16:58.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.457+0000 7f7a87c83640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a800b20a0 0x7f7a800d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.457+0000 7f7a87c83640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a800cdcb0 con 0x7f7a800d1780 2023-11-28T17:16:58.458 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.457+0000 7f7a87c83640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a800d39e0 con 0x7f7a800b2ad0 2023-11-28T17:16:58.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.457+0000 7f7a87c83640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a800d3cc0 con 0x7f7a800b20a0 2023-11-28T17:16:58.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.457+0000 7f7a85a1f640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7a800d1780 0x7f7a800d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:32904/0 (socket says 172.21.15.116:32904) 2023-11-28T17:16:58.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.457+0000 7f7a85a1f640 1 -- 172.21.15.116:0/4240589654 learned_addr learned my addr 172.21.15.116:0/4240589654 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:16:58.459 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.457+0000 7f7a8521e640 1 --2- 172.21.15.116:0/4240589654 >> v2:172.21.15.116:3300/0 conn(0x7f7a800b20a0 0x7f7a800d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.457+0000 7f7a84a1d640 1 -- 172.21.15.116:0/4240589654 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680284097 0 0) 0x7f7a800cdcb0 con 0x7f7a800d1780 2023-11-28T17:16:58.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.457+0000 7f7a84a1d640 1 -- 172.21.15.116:0/4240589654 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a6c003520 con 0x7f7a800d1780 2023-11-28T17:16:58.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.458+0000 7f7a8521e640 1 -- 172.21.15.116:0/4240589654 >> v1:172.21.15.156:6789/0 conn(0x7f7a800b2ad0 legacy=0x7f7a800ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:58.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.458+0000 7f7a8521e640 1 -- 172.21.15.116:0/4240589654 >> v1:172.21.15.116:6789/0 conn(0x7f7a800d1780 legacy=0x7f7a800d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.460 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.458+0000 7f7a8521e640 1 -- 172.21.15.116:0/4240589654 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a800cdf50 con 0x7f7a800b20a0 2023-11-28T17:16:58.461 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.459+0000 7f7a8521e640 1 --2- 172.21.15.116:0/4240589654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a800b20a0 0x7f7a800d01e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7a700097a0 tx=0x7f7a7002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efbe58ed348ac38f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.459+0000 7f7a84a1d640 1 -- 172.21.15.116:0/4240589654 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a70031830 con 0x7f7a800b20a0 2023-11-28T17:16:58.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.459+0000 7f7a84a1d640 1 -- 172.21.15.116:0/4240589654 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a70031990 con 0x7f7a800b20a0 2023-11-28T17:16:58.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.459+0000 7f7a84a1d640 1 -- 172.21.15.116:0/4240589654 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a70031d20 con 0x7f7a800b20a0 2023-11-28T17:16:58.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.460+0000 7f7a87c83640 1 -- 172.21.15.116:0/4240589654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a800b20a0 msgr2=0x7f7a800d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.460+0000 7f7a87c83640 1 --2- 172.21.15.116:0/4240589654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a800b20a0 0x7f7a800d01e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f7a700097a0 tx=0x7f7a7002b8d0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.460+0000 7f7a87c83640 1 -- 172.21.15.116:0/4240589654 shutdown_connections 2023-11-28T17:16:58.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.460+0000 7f7a87c83640 1 --2- 172.21.15.116:0/4240589654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a800b20a0 0x7f7a800d01e0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.462 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.460+0000 7f7a87c83640 1 -- 172.21.15.116:0/4240589654 >> 172.21.15.116:0/4240589654 conn(0x7f7a8009c4e0 msgr2=0x7f7a800d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:58.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.460+0000 7f7a87c83640 1 -- 172.21.15.116:0/4240589654 shutdown_connections 2023-11-28T17:16:58.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.460+0000 7f7a87c83640 1 -- 172.21.15.116:0/4240589654 wait complete. 2023-11-28T17:16:58.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.462+0000 7f7a87c83640 1 Processor -- start 2023-11-28T17:16:58.463 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.462+0000 7f7a87c83640 1 -- start start 2023-11-28T17:16:58.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.463+0000 7f7a87c83640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a800b20a0 0x7f7a8009d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.465 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.463+0000 7f7a87c83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a800b2ad0 0x7f7a8014d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.463+0000 7f7a87c83640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a800d1780 0x7f7a800a09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.463+0000 7f7a87c83640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8014da90 con 0x7f7a800b2ad0 2023-11-28T17:16:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.463+0000 7f7a87c83640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a800cb910 con 0x7f7a800b20a0 2023-11-28T17:16:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.463+0000 7f7a87c83640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a800a0f30 con 0x7f7a800d1780 2023-11-28T17:16:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.463+0000 7f7a8521e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a800b2ad0 0x7f7a8014d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.463+0000 7f7a8521e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a800b2ad0 0x7f7a8014d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53506/0 (socket says 172.21.15.116:53506) 2023-11-28T17:16:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.463+0000 7f7a8521e640 1 -- 172.21.15.116:0/3164348598 learned_addr learned my addr 172.21.15.116:0/3164348598 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.464+0000 7f7a85a1f640 1 --2- 172.21.15.116:0/3164348598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a800b20a0 0x7f7a8009d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.466 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.464+0000 7f7a86220640 1 --2- 172.21.15.116:0/3164348598 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a800d1780 0x7f7a800a09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.464+0000 7f7a8521e640 1 -- 172.21.15.116:0/3164348598 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a800d1780 msgr2=0x7f7a800a09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.464+0000 7f7a8521e640 1 --2- 172.21.15.116:0/3164348598 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a800d1780 0x7f7a800a09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.464+0000 7f7a8521e640 1 -- 172.21.15.116:0/3164348598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a800b20a0 msgr2=0x7f7a8009d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.464+0000 7f7a8521e640 1 --2- 172.21.15.116:0/3164348598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a800b20a0 0x7f7a8009d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.464+0000 7f7a8521e640 1 -- 172.21.15.116:0/3164348598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a70009400 con 0x7f7a800b2ad0 2023-11-28T17:16:58.467 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.464+0000 7f7a8521e640 1 --2- 172.21.15.116:0/3164348598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a800b2ad0 0x7f7a8014d550 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7a7002bee0 tx=0x7f7a700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.465+0000 7f7a767fc640 1 -- 172.21.15.116:0/3164348598 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7003a6f0 con 0x7f7a800b2ad0 2023-11-28T17:16:58.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.465+0000 7f7a767fc640 1 -- 172.21.15.116:0/3164348598 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7003a870 con 0x7f7a800b2ad0 2023-11-28T17:16:58.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.465+0000 7f7a767fc640 1 -- 172.21.15.116:0/3164348598 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7003ac20 con 0x7f7a800b2ad0 2023-11-28T17:16:58.468 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.465+0000 7f7a87c83640 1 -- 172.21.15.116:0/3164348598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8009df40 con 0x7f7a800b2ad0 2023-11-28T17:16:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.465+0000 7f7a87c83640 1 -- 172.21.15.116:0/3164348598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a800cbbb0 con 0x7f7a800b2ad0 2023-11-28T17:16:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.466+0000 7f7a87c83640 1 -- 172.21.15.116:0/3164348598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a80069360 con 0x7f7a800b2ad0 2023-11-28T17:16:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.467+0000 7f7a767fc640 1 -- 172.21.15.116:0/3164348598 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f7a70030380 con 0x7f7a800b2ad0 2023-11-28T17:16:58.469 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.468+0000 7f7a767fc640 1 --2- 172.21.15.116:0/3164348598 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7a6406b2a0 0x7f7a6406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.468+0000 7f7a767fc640 1 -- 172.21.15.116:0/3164348598 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f7a700ab720 con 0x7f7a800b2ad0 2023-11-28T17:16:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.468+0000 7f7a85a1f640 1 --2- 172.21.15.116:0/3164348598 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7a6406b2a0 0x7f7a6406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.470 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.469+0000 7f7a85a1f640 1 --2- 172.21.15.116:0/3164348598 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7a6406b2a0 0x7f7a6406d760 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0011e0 tx=0x7f7a7c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.472+0000 7f7a767fc640 1 -- 172.21.15.116:0/3164348598 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a700763b0 con 0x7f7a800b2ad0 2023-11-28T17:16:58.555 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.554+0000 7f47b661e640 1 -- 172.21.15.116:0/828704029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47b00d03f0 con 0x7f47b00d1780 2023-11-28T17:16:58.558 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.557+0000 7f47acff9640 1 -- 172.21.15.116:0/828704029 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+17073 (secure 0 0 0) 0x7f47a807ae30 con 0x7f47b00d1780 2023-11-28T17:16:58.559 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:16:58.559 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":52,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:12:26.038218+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:11:21.518078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T17:11:22.672451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":0}]},{"pgid":"7.8","mappings":[{"from":5,"to":7}]},{"pgid":"7.1b","mappings":[{"from":5,"to":7}]},{"pgid":"7.30","mappings":[{"from":5,"to":7}]},{"pgid":"7.31","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6833/372407661":"2023-11-29T17:12:26.038071+0000","172.21.15.156:6832/372407661":"2023-11-29T17:12:26.038071+0000","172.21.15.116:6835/3251347229":"2023-11-29T17:12:06.030424+0000","172.21.15.116:6834/3251347229":"2023-11-29T17:12:06.030424+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:16:58.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.559+0000 7f47b661e640 1 -- 172.21.15.116:0/828704029 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f479c06b2a0 msgr2=0x7f479c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.559+0000 7f47b661e640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f479c06b2a0 0x7f479c06d760 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f47a000a990 tx=0x7f47a0008040 comp rx=0 tx=0).stop 2023-11-28T17:16:58.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.559+0000 7f47b661e640 1 -- 172.21.15.116:0/828704029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b00d1780 msgr2=0x7f47b00b0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.559+0000 7f47b661e640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b00d1780 0x7f47b00b0ed0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f47a8036040 tx=0x7f47a8004a40 comp rx=0 tx=0).stop 2023-11-28T17:16:58.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.559+0000 7f47b661e640 1 -- 172.21.15.116:0/828704029 shutdown_connections 2023-11-28T17:16:58.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.559+0000 7f47b661e640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f479c06b2a0 0x7f479c06d760 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.559+0000 7f47b661e640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47b00d1780 0x7f47b00b0ed0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.559+0000 7f47b661e640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f47b00b2ad0 0x7f47b00b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.559+0000 7f47b661e640 1 --2- 172.21.15.116:0/828704029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47b00b20a0 0x7f47b00b6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.559+0000 7f47b661e640 1 -- 172.21.15.116:0/828704029 >> 172.21.15.116:0/828704029 conn(0x7f47b009c4e0 msgr2=0x7f47b006c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:58.561 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.560+0000 7f47b661e640 1 -- 172.21.15.116:0/828704029 shutdown_connections 2023-11-28T17:16:58.562 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.560+0000 7f47b661e640 1 -- 172.21.15.116:0/828704029 wait complete. 2023-11-28T17:16:58.580 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:11:21.518078+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:11:22.672451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:16:58.581 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:16:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.665+0000 7f7a87c83640 1 -- 172.21.15.116:0/3164348598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7a800c9b00 con 0x7f7a800b2ad0 2023-11-28T17:16:58.667 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.666+0000 7f7a767fc640 1 -- 172.21.15.116:0/3164348598 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7a700799d0 con 0x7f7a800b2ad0 2023-11-28T17:16:58.667 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:16:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.669+0000 7f7a87c83640 1 -- 172.21.15.116:0/3164348598 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7a6406b2a0 msgr2=0x7f7a6406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.669+0000 7f7a87c83640 1 --2- 172.21.15.116:0/3164348598 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7a6406b2a0 0x7f7a6406d760 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0011e0 tx=0x7f7a7c009040 comp rx=0 tx=0).stop 2023-11-28T17:16:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.669+0000 7f7a87c83640 1 -- 172.21.15.116:0/3164348598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a800b2ad0 msgr2=0x7f7a8014d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.669+0000 7f7a87c83640 1 --2- 172.21.15.116:0/3164348598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a800b2ad0 0x7f7a8014d550 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7a7002bee0 tx=0x7f7a700046c0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.671 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.669+0000 7f7a87c83640 1 -- 172.21.15.116:0/3164348598 shutdown_connections 2023-11-28T17:16:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.669+0000 7f7a87c83640 1 --2- 172.21.15.116:0/3164348598 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f7a6406b2a0 0x7f7a6406d760 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.669+0000 7f7a87c83640 1 --2- 172.21.15.116:0/3164348598 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7a800d1780 0x7f7a800a09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.669+0000 7f7a87c83640 1 --2- 172.21.15.116:0/3164348598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a800b2ad0 0x7f7a8014d550 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.669+0000 7f7a87c83640 1 --2- 172.21.15.116:0/3164348598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7a800b20a0 0x7f7a8009d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.669+0000 7f7a87c83640 1 -- 172.21.15.116:0/3164348598 >> 172.21.15.116:0/3164348598 conn(0x7f7a8009c4e0 msgr2=0x7f7a800d2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.669+0000 7f7a87c83640 1 -- 172.21.15.116:0/3164348598 shutdown_connections 2023-11-28T17:16:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.669+0000 7f7a87c83640 1 -- 172.21.15.116:0/3164348598 wait complete. 2023-11-28T17:16:58.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:16:58.776 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.775+0000 7fb9255c9640 1 Processor -- start 2023-11-28T17:16:58.777 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.776+0000 7fb9255c9640 1 -- start start 2023-11-28T17:16:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.777+0000 7fb9255c9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb92015e840 0x7fb92015f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.777+0000 7fb9255c9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb9200d17b0 0x7fb9200d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.777+0000 7fb9255c9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb920161050 con 0x7fb9200b41b0 2023-11-28T17:16:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.777+0000 7fb9255c9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9200d2180 con 0x7fb9200d17b0 2023-11-28T17:16:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.777+0000 7fb9255c9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92015ec20 con 0x7fb92015e840 2023-11-28T17:16:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.777+0000 7fb91f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb9200d17b0 0x7fb9200d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.777+0000 7fb91f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb9200d17b0 0x7fb9200d1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53516/0 (socket says 172.21.15.116:53516) 2023-11-28T17:16:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.777+0000 7fb91f7fe640 1 -- 172.21.15.116:0/460247383 learned_addr learned my addr 172.21.15.116:0/460247383 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.778+0000 7fb91effd640 1 --2- 172.21.15.116:0/460247383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb92015e840 0x7fb92015f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.778+0000 7fb91f7fe640 1 -- 172.21.15.116:0/460247383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb92015e840 msgr2=0x7fb92015f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.778+0000 7fb91f7fe640 1 --2- 172.21.15.116:0/460247383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb92015e840 0x7fb92015f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.778+0000 7fb91f7fe640 1 -- 172.21.15.116:0/460247383 >> v1:172.21.15.156:6789/0 conn(0x7fb9200b41b0 legacy=0x7fb920160940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.778+0000 7fb91f7fe640 1 -- 172.21.15.116:0/460247383 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92015f8a0 con 0x7fb9200d17b0 2023-11-28T17:16:58.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.780+0000 7fb91f7fe640 1 --2- 172.21.15.116:0/460247383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9200d17b0 0x7fb9200d1c40 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb918009e50 tx=0x7fb91802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73d16db9a5ead47 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.781 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.780+0000 7fb91e7fc640 1 -- 172.21.15.116:0/460247383 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91802cb00 con 0x7fb9200d17b0 2023-11-28T17:16:58.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.780+0000 7fb91e7fc640 1 -- 172.21.15.116:0/460247383 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91802cc60 con 0x7fb9200d17b0 2023-11-28T17:16:58.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.780+0000 7fb91e7fc640 1 -- 172.21.15.116:0/460247383 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9180343f0 con 0x7fb9200d17b0 2023-11-28T17:16:58.782 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.781+0000 7fb9255c9640 1 -- 172.21.15.116:0/460247383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9200d17b0 msgr2=0x7fb9200d1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.781+0000 7fb9255c9640 1 --2- 172.21.15.116:0/460247383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9200d17b0 0x7fb9200d1c40 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb918009e50 tx=0x7fb91802bf60 comp rx=0 tx=0).stop 2023-11-28T17:16:58.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.781+0000 7fb9255c9640 1 -- 172.21.15.116:0/460247383 shutdown_connections 2023-11-28T17:16:58.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.781+0000 7fb9255c9640 1 --2- 172.21.15.116:0/460247383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9200d17b0 0x7fb9200d1c40 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.781+0000 7fb9255c9640 1 --2- 172.21.15.116:0/460247383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb92015e840 0x7fb92015f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.781+0000 7fb9255c9640 1 -- 172.21.15.116:0/460247383 >> 172.21.15.116:0/460247383 conn(0x7fb92009c4c0 msgr2=0x7fb9200d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:58.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.781+0000 7fb9255c9640 1 -- 172.21.15.116:0/460247383 shutdown_connections 2023-11-28T17:16:58.783 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.781+0000 7fb9255c9640 1 -- 172.21.15.116:0/460247383 wait complete. 2023-11-28T17:16:58.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.783+0000 7fb9255c9640 1 Processor -- start 2023-11-28T17:16:58.784 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.783+0000 7fb9255c9640 1 -- start start 2023-11-28T17:16:58.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.784+0000 7fb9255c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9200b41b0 0x7fb92014f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.784+0000 7fb9255c9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9200d17b0 0x7fb9200cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.784+0000 7fb9255c9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb92015e840 0x7fb9200cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.784+0000 7fb9255c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9200b8c40 con 0x7fb9200b41b0 2023-11-28T17:16:58.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.784+0000 7fb9255c9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9200b7fc0 con 0x7fb9200d17b0 2023-11-28T17:16:58.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.784+0000 7fb9255c9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9200b82a0 con 0x7fb92015e840 2023-11-28T17:16:58.786 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.784+0000 7fb91effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9200b41b0 0x7fb92014f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.784+0000 7fb91effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9200b41b0 0x7fb92014f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53526/0 (socket says 172.21.15.116:53526) 2023-11-28T17:16:58.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.784+0000 7fb91effd640 1 -- 172.21.15.116:0/1213505736 learned_addr learned my addr 172.21.15.116:0/1213505736 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:58.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.785+0000 7fb91effd640 1 -- 172.21.15.116:0/1213505736 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb92015e840 msgr2=0x7fb9200cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.785+0000 7fb91f7fe640 1 --2- 172.21.15.116:0/1213505736 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb92015e840 0x7fb9200cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.785+0000 7fb91effd640 1 --2- 172.21.15.116:0/1213505736 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb92015e840 0x7fb9200cce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.785+0000 7fb9167fc640 1 --2- 172.21.15.116:0/1213505736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9200d17b0 0x7fb9200cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.785+0000 7fb91effd640 1 -- 172.21.15.116:0/1213505736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9200d17b0 msgr2=0x7fb9200cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.785+0000 7fb91effd640 1 --2- 172.21.15.116:0/1213505736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9200d17b0 0x7fb9200cc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.785+0000 7fb91effd640 1 -- 172.21.15.116:0/1213505736 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb918009b30 con 0x7fb9200b41b0 2023-11-28T17:16:58.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.785+0000 7fb91effd640 1 --2- 172.21.15.116:0/1213505736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9200b41b0 0x7fb92014f750 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fb91000d5f0 tx=0x7fb91000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.785+0000 7fb917fff640 1 -- 172.21.15.116:0/1213505736 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9100125f0 con 0x7fb9200b41b0 2023-11-28T17:16:58.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.785+0000 7fb917fff640 1 -- 172.21.15.116:0/1213505736 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb910012750 con 0x7fb9200b41b0 2023-11-28T17:16:58.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.785+0000 7fb917fff640 1 -- 172.21.15.116:0/1213505736 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb910012b00 con 0x7fb9200b41b0 2023-11-28T17:16:58.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.785+0000 7fb9255c9640 1 -- 172.21.15.116:0/1213505736 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9200b8580 con 0x7fb9200b41b0 2023-11-28T17:16:58.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.785+0000 7fb9255c9640 1 -- 172.21.15.116:0/1213505736 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb920159700 con 0x7fb9200b41b0 2023-11-28T17:16:58.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.787+0000 7fb9255c9640 1 -- 172.21.15.116:0/1213505736 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb920067460 con 0x7fb9200b41b0 2023-11-28T17:16:58.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.787+0000 7fb917fff640 1 -- 172.21.15.116:0/1213505736 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fb910012c60 con 0x7fb9200b41b0 2023-11-28T17:16:58.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.788+0000 7fb917fff640 1 --2- 172.21.15.116:0/1213505736 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb90406b2a0 0x7fb90406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.788+0000 7fb917fff640 1 -- 172.21.15.116:0/1213505736 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fb91008a1f0 con 0x7fb9200b41b0 2023-11-28T17:16:58.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.788+0000 7fb9167fc640 1 --2- 172.21.15.116:0/1213505736 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb90406b2a0 0x7fb90406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.789+0000 7fb9167fc640 1 --2- 172.21.15.116:0/1213505736 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb90406b2a0 0x7fb90406d760 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb9200c8a50 tx=0x7fb908009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.793+0000 7fb917fff640 1 -- 172.21.15.116:0/1213505736 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb910054f10 con 0x7fb9200b41b0 2023-11-28T17:16:58.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.889+0000 7f76db59e640 1 Processor -- start 2023-11-28T17:16:58.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.889+0000 7f76db59e640 1 -- start start 2023-11-28T17:16:58.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.891+0000 7f76db59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f76dc0d0700 0x7f76dc0d1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.891+0000 7f76db59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76dc0b4010 0x7f76dc0c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.891+0000 7f76db59e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76dc0d0da0 con 0x7f76dc160b80 2023-11-28T17:16:58.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.891+0000 7f76db59e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76dc160f60 con 0x7f76dc0d0700 2023-11-28T17:16:58.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.891+0000 7f76db59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc0d1080 con 0x7f76dc0b4010 2023-11-28T17:16:58.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.891+0000 7f76d3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f76dc0d0700 0x7f76dc0d1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.891+0000 7f76d3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f76dc0d0700 0x7f76dc0d1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53534/0 (socket says 172.21.15.116:53534) 2023-11-28T17:16:58.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.891+0000 7f76d3fff640 1 -- 172.21.15.116:0/4011347217 learned_addr learned my addr 172.21.15.116:0/4011347217 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:58.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.891+0000 7f76d3fff640 1 -- 172.21.15.116:0/4011347217 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76dc0b4010 msgr2=0x7f76dc0c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:58.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.891+0000 7f76dad9d640 1 --2- 172.21.15.116:0/4011347217 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76dc0b4010 0x7f76dc0c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.891+0000 7f76d3fff640 1 --2- 172.21.15.116:0/4011347217 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76dc0b4010 0x7f76dc0c6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.891+0000 7f76d3fff640 1 -- 172.21.15.116:0/4011347217 >> v1:172.21.15.156:6789/0 conn(0x7f76dc160b80 legacy=0x7f76dc0d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.892+0000 7f76d3fff640 1 -- 172.21.15.116:0/4011347217 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76dc0ce780 con 0x7f76dc0d0700 2023-11-28T17:16:58.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.893+0000 7f76d3fff640 1 --2- 172.21.15.116:0/4011347217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76dc0d0700 0x7f76dc0d1a70 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009870 tx=0x7f76c402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b11fdd4883a18f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.893+0000 7f76d9d9b640 1 -- 172.21.15.116:0/4011347217 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c4030840 con 0x7f76dc0d0700 2023-11-28T17:16:58.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.893+0000 7f76d9d9b640 1 -- 172.21.15.116:0/4011347217 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c40309a0 con 0x7f76dc0d0700 2023-11-28T17:16:58.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.893+0000 7f76d9d9b640 1 -- 172.21.15.116:0/4011347217 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c4030d30 con 0x7f76dc0d0700 2023-11-28T17:16:58.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.893+0000 7f76db59e640 1 -- 172.21.15.116:0/4011347217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76dc0d0700 msgr2=0x7f76dc0d1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.893+0000 7f76db59e640 1 --2- 172.21.15.116:0/4011347217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76dc0d0700 0x7f76dc0d1a70 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009870 tx=0x7f76c402b980 comp rx=0 tx=0).stop 2023-11-28T17:16:58.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.894+0000 7f76db59e640 1 -- 172.21.15.116:0/4011347217 shutdown_connections 2023-11-28T17:16:58.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.894+0000 7f76db59e640 1 --2- 172.21.15.116:0/4011347217 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76dc0b4010 0x7f76dc0c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.894+0000 7f76db59e640 1 --2- 172.21.15.116:0/4011347217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76dc0d0700 0x7f76dc0d1a70 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.894+0000 7f76db59e640 1 -- 172.21.15.116:0/4011347217 >> 172.21.15.116:0/4011347217 conn(0x7f76dc0d2fc0 msgr2=0x7f76dc0d33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:58.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.894+0000 7f76db59e640 1 -- 172.21.15.116:0/4011347217 shutdown_connections 2023-11-28T17:16:58.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.894+0000 7f76db59e640 1 -- 172.21.15.116:0/4011347217 wait complete. 2023-11-28T17:16:58.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.895+0000 7f76db59e640 1 Processor -- start 2023-11-28T17:16:58.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.896+0000 7f76db59e640 1 -- start start 2023-11-28T17:16:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.896+0000 7f76db59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76dc0b4010 0x7f76dc146a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.896+0000 7f76db59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76dc160b80 0x7f76dc159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.897+0000 7f76db59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76dc0b8900 0x7f76dc1579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.897+0000 7f76db59e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc158980 con 0x7f76dc0b4010 2023-11-28T17:16:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.897+0000 7f76db59e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc146fa0 con 0x7f76dc160b80 2023-11-28T17:16:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.897+0000 7f76db59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc0b7860 con 0x7f76dc0b8900 2023-11-28T17:16:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.897+0000 7f76da59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76dc0b4010 0x7f76dc146a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.897+0000 7f76da59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76dc0b4010 0x7f76dc146a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53546/0 (socket says 172.21.15.116:53546) 2023-11-28T17:16:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.897+0000 7f76da59c640 1 -- 172.21.15.116:0/1522300568 learned_addr learned my addr 172.21.15.116:0/1522300568 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.897+0000 7f76da59c640 1 -- 172.21.15.116:0/1522300568 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76dc0b8900 msgr2=0x7f76dc1579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.897+0000 7f76da59c640 1 --2- 172.21.15.116:0/1522300568 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76dc0b8900 0x7f76dc1579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.897+0000 7f76da59c640 1 -- 172.21.15.116:0/1522300568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76dc160b80 msgr2=0x7f76dc159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.897+0000 7f76da59c640 1 --2- 172.21.15.116:0/1522300568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76dc160b80 0x7f76dc159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.897+0000 7f76da59c640 1 -- 172.21.15.116:0/1522300568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c40094d0 con 0x7f76dc0b4010 2023-11-28T17:16:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.897+0000 7f76da59c640 1 --2- 172.21.15.116:0/1522300568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76dc0b4010 0x7f76dc146a60 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f76cc00c400 tx=0x7f76cc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.898+0000 7f76d2ffd640 1 -- 172.21.15.116:0/1522300568 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0052d0 con 0x7f76dc0b4010 2023-11-28T17:16:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.898+0000 7f76d2ffd640 1 -- 172.21.15.116:0/1522300568 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc005430 con 0x7f76dc0b4010 2023-11-28T17:16:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.898+0000 7f76d2ffd640 1 -- 172.21.15.116:0/1522300568 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc003830 con 0x7f76dc0b4010 2023-11-28T17:16:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.898+0000 7f76db59e640 1 -- 172.21.15.116:0/1522300568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76dc0b7b40 con 0x7f76dc0b4010 2023-11-28T17:16:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.898+0000 7f76db59e640 1 -- 172.21.15.116:0/1522300568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76dc0b8100 con 0x7f76dc0b4010 2023-11-28T17:16:58.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.899+0000 7f76db59e640 1 -- 172.21.15.116:0/1522300568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76dc067980 con 0x7f76dc0b4010 2023-11-28T17:16:58.901 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.900+0000 7f76d2ffd640 1 -- 172.21.15.116:0/1522300568 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f76cc003990 con 0x7f76dc0b4010 2023-11-28T17:16:58.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.901+0000 7f76d2ffd640 1 --2- 172.21.15.116:0/1522300568 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f76bc06b2a0 0x7f76bc06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:58.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.902+0000 7f76d2ffd640 1 -- 172.21.15.116:0/1522300568 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f76cc0892c0 con 0x7f76dc0b4010 2023-11-28T17:16:58.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.902+0000 7f76d3fff640 1 --2- 172.21.15.116:0/1522300568 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f76bc06b2a0 0x7f76bc06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:58.903 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.902+0000 7f76d3fff640 1 --2- 172.21.15.116:0/1522300568 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f76bc06b2a0 0x7f76bc06d760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f76c4036000 tx=0x7f76c403b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:58.906 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.905+0000 7f76d2ffd640 1 -- 172.21.15.116:0/1522300568 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f76cc053ff0 con 0x7f76dc0b4010 2023-11-28T17:16:58.987 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.986+0000 7fb9255c9640 1 -- 172.21.15.116:0/1213505736 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb920066750 con 0x7fb9200b41b0 2023-11-28T17:16:58.988 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.986+0000 7fb917fff640 1 -- 172.21.15.116:0/1213505736 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb920066750 con 0x7fb9200b41b0 2023-11-28T17:16:58.988 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:16:58.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.989+0000 7fb9255c9640 1 -- 172.21.15.116:0/1213505736 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb90406b2a0 msgr2=0x7fb90406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.989+0000 7fb9255c9640 1 --2- 172.21.15.116:0/1213505736 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb90406b2a0 0x7fb90406d760 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb9200c8a50 tx=0x7fb908009040 comp rx=0 tx=0).stop 2023-11-28T17:16:58.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.989+0000 7fb9255c9640 1 -- 172.21.15.116:0/1213505736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9200b41b0 msgr2=0x7fb92014f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:58.991 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.989+0000 7fb9255c9640 1 --2- 172.21.15.116:0/1213505736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9200b41b0 0x7fb92014f750 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fb91000d5f0 tx=0x7fb91000dab0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.989+0000 7fb9255c9640 1 -- 172.21.15.116:0/1213505736 shutdown_connections 2023-11-28T17:16:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.989+0000 7fb9255c9640 1 --2- 172.21.15.116:0/1213505736 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fb90406b2a0 0x7fb90406d760 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.989+0000 7fb9255c9640 1 --2- 172.21.15.116:0/1213505736 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb92015e840 0x7fb9200cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.989+0000 7fb9255c9640 1 --2- 172.21.15.116:0/1213505736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb9200d17b0 0x7fb9200cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.989+0000 7fb9255c9640 1 --2- 172.21.15.116:0/1213505736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9200b41b0 0x7fb92014f750 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.989+0000 7fb9255c9640 1 -- 172.21.15.116:0/1213505736 >> 172.21.15.116:0/1213505736 conn(0x7fb92009c4c0 msgr2=0x7fb92015ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.990+0000 7fb9255c9640 1 -- 172.21.15.116:0/1213505736 shutdown_connections 2023-11-28T17:16:58.993 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:58.990+0000 7fb9255c9640 1 -- 172.21.15.116:0/1213505736 wait complete. 2023-11-28T17:16:59.009 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:16:59.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.094+0000 7f76db59e640 1 -- 172.21.15.116:0/1522300568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f76dc0a0eb0 con 0x7f76dc0b4010 2023-11-28T17:16:59.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.095+0000 7f76d2ffd640 1 -- 172.21.15.116:0/1522300568 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) v1 ==== 104+0+11 (secure 0 0 0) 0x7f76cc057610 con 0x7f76dc0b4010 2023-11-28T17:16:59.096 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:16:59.098 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.097+0000 7f76db59e640 1 -- 172.21.15.116:0/1522300568 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f76bc06b2a0 msgr2=0x7f76bc06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.097+0000 7f76db59e640 1 --2- 172.21.15.116:0/1522300568 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f76bc06b2a0 0x7f76bc06d760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f76c4036000 tx=0x7f76c403b040 comp rx=0 tx=0).stop 2023-11-28T17:16:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.097+0000 7f76db59e640 1 -- 172.21.15.116:0/1522300568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76dc0b4010 msgr2=0x7f76dc146a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.097+0000 7f76db59e640 1 --2- 172.21.15.116:0/1522300568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76dc0b4010 0x7f76dc146a60 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f76cc00c400 tx=0x7f76cc00c8c0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.097+0000 7f76db59e640 1 -- 172.21.15.116:0/1522300568 shutdown_connections 2023-11-28T17:16:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.097+0000 7f76db59e640 1 --2- 172.21.15.116:0/1522300568 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f76bc06b2a0 0x7f76bc06d760 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.097+0000 7f76db59e640 1 --2- 172.21.15.116:0/1522300568 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76dc0b8900 0x7f76dc1579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.097+0000 7f76db59e640 1 --2- 172.21.15.116:0/1522300568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76dc160b80 0x7f76dc159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.097+0000 7f76db59e640 1 --2- 172.21.15.116:0/1522300568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76dc0b4010 0x7f76dc146a60 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.097+0000 7f76db59e640 1 -- 172.21.15.116:0/1522300568 >> 172.21.15.116:0/1522300568 conn(0x7f76dc0d2fc0 msgr2=0x7f76dc0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.097+0000 7f76db59e640 1 -- 172.21.15.116:0/1522300568 shutdown_connections 2023-11-28T17:16:59.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.097+0000 7f76db59e640 1 -- 172.21.15.116:0/1522300568 wait complete. 2023-11-28T17:16:59.114 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:16:59.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.208+0000 7f2b03f97640 1 Processor -- start 2023-11-28T17:16:59.210 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.209+0000 7f2b03f97640 1 -- start start 2023-11-28T17:16:59.211 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.210+0000 7f2b03f97640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2afc146b10 0x7f2afc161a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.210+0000 7f2b03f97640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2afc15dfb0 0x7f2afc0d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.210+0000 7f2b03f97640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2afc06eaf0 con 0x7f2afc0d0f10 2023-11-28T17:16:59.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.210+0000 7f2b03f97640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2afc0d18e0 con 0x7f2afc146b10 2023-11-28T17:16:59.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.210+0000 7f2b03f97640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2afc15e4e0 con 0x7f2afc15dfb0 2023-11-28T17:16:59.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.210+0000 7f2b01d33640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2afc146b10 0x7f2afc161a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.212 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.211+0000 7f2b01d33640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2afc146b10 0x7f2afc161a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53554/0 (socket says 172.21.15.116:53554) 2023-11-28T17:16:59.213 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.211+0000 7f2b01d33640 1 -- 172.21.15.116:0/3020240819 learned_addr learned my addr 172.21.15.116:0/3020240819 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:59.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.211+0000 7f2b01d33640 1 -- 172.21.15.116:0/3020240819 >> v2:172.21.15.156:3300/0 conn(0x7f2afc15dfb0 msgr2=0x7f2afc0d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:59.214 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.211+0000 7f2b01d33640 1 --2- 172.21.15.116:0/3020240819 >> v2:172.21.15.156:3300/0 conn(0x7f2afc15dfb0 0x7f2afc0d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.211+0000 7f2b01d33640 1 -- 172.21.15.116:0/3020240819 >> v1:172.21.15.156:6789/0 conn(0x7f2afc0d0f10 legacy=0x7f2afc15d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:59.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.211+0000 7f2b01d33640 1 -- 172.21.15.116:0/3020240819 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2afc15e7c0 con 0x7f2afc146b10 2023-11-28T17:16:59.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.212+0000 7f2b01d33640 1 --2- 172.21.15.116:0/3020240819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2afc146b10 0x7f2afc161a30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2afc06b990 tx=0x7f2af802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32509a2dca32d235 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:59.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.212+0000 7f2b00d31640 1 -- 172.21.15.116:0/3020240819 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af8031870 con 0x7f2afc146b10 2023-11-28T17:16:59.215 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.212+0000 7f2b00d31640 1 -- 172.21.15.116:0/3020240819 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af80319d0 con 0x7f2afc146b10 2023-11-28T17:16:59.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.212+0000 7f2b00d31640 1 -- 172.21.15.116:0/3020240819 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af8031d60 con 0x7f2afc146b10 2023-11-28T17:16:59.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.213+0000 7f2b03f97640 1 -- 172.21.15.116:0/3020240819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2afc146b10 msgr2=0x7f2afc161a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.213+0000 7f2b03f97640 1 --2- 172.21.15.116:0/3020240819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2afc146b10 0x7f2afc161a30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2afc06b990 tx=0x7f2af802b910 comp rx=0 tx=0).stop 2023-11-28T17:16:59.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.213+0000 7f2b03f97640 1 -- 172.21.15.116:0/3020240819 shutdown_connections 2023-11-28T17:16:59.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.213+0000 7f2b03f97640 1 --2- 172.21.15.116:0/3020240819 >> v2:172.21.15.156:3300/0 conn(0x7f2afc15dfb0 0x7f2afc0d13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.216 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.213+0000 7f2b03f97640 1 --2- 172.21.15.116:0/3020240819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2afc146b10 0x7f2afc161a30 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.213+0000 7f2b03f97640 1 -- 172.21.15.116:0/3020240819 >> 172.21.15.116:0/3020240819 conn(0x7f2afc070af0 msgr2=0x7f2afc0662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:59.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.213+0000 7f2b03f97640 1 -- 172.21.15.116:0/3020240819 shutdown_connections 2023-11-28T17:16:59.217 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.213+0000 7f2b03f97640 1 -- 172.21.15.116:0/3020240819 wait complete. 2023-11-28T17:16:59.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.216+0000 7f2b03f97640 1 Processor -- start 2023-11-28T17:16:59.218 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.216+0000 7f2b03f97640 1 -- start start 2023-11-28T17:16:59.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b03f97640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2afc0d0f10 0x7f2afc0b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b03f97640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2afc146b10 0x7f2afc13c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b03f97640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2afc15dfb0 0x7f2afc15c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b03f97640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc1f3580 con 0x7f2afc146b10 2023-11-28T17:16:59.220 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b03f97640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc1f3860 con 0x7f2afc15dfb0 2023-11-28T17:16:59.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b03f97640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc1f3b40 con 0x7f2afc0d0f10 2023-11-28T17:16:59.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b01532640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2afc146b10 0x7f2afc13c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b01532640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2afc146b10 0x7f2afc13c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53566/0 (socket says 172.21.15.116:53566) 2023-11-28T17:16:59.221 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b01532640 1 -- 172.21.15.116:0/2415199147 learned_addr learned my addr 172.21.15.116:0/2415199147 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:59.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b01d33640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2afc0d0f10 0x7f2afc0b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b02534640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2afc15dfb0 0x7f2afc15c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b01532640 1 -- 172.21.15.116:0/2415199147 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2afc0d0f10 msgr2=0x7f2afc0b2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b01532640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2afc0d0f10 0x7f2afc0b2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.222 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b01532640 1 -- 172.21.15.116:0/2415199147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2afc15dfb0 msgr2=0x7f2afc15c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b01532640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2afc15dfb0 0x7f2afc15c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b01532640 1 -- 172.21.15.116:0/2415199147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af80094d0 con 0x7f2afc146b10 2023-11-28T17:16:59.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.218+0000 7f2b02534640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2afc15dfb0 0x7f2afc15c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:16:59.223 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.219+0000 7f2b01532640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2afc146b10 0x7f2afc13c460 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2aec00c410 tx=0x7f2aec00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:59.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.219+0000 7f2af27fc640 1 -- 172.21.15.116:0/2415199147 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec0035f0 con 0x7f2afc146b10 2023-11-28T17:16:59.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.219+0000 7f2af27fc640 1 -- 172.21.15.116:0/2415199147 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aec003750 con 0x7f2afc146b10 2023-11-28T17:16:59.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.219+0000 7f2af27fc640 1 -- 172.21.15.116:0/2415199147 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec003b40 con 0x7f2afc146b10 2023-11-28T17:16:59.225 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.219+0000 7f2b03f97640 1 -- 172.21.15.116:0/2415199147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2afc1f3e20 con 0x7f2afc146b10 2023-11-28T17:16:59.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.219+0000 7f2b03f97640 1 -- 172.21.15.116:0/2415199147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2afc15c7b0 con 0x7f2afc146b10 2023-11-28T17:16:59.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.220+0000 7f2b03f97640 1 -- 172.21.15.116:0/2415199147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2afc069360 con 0x7f2afc146b10 2023-11-28T17:16:59.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.221+0000 7f2af27fc640 1 -- 172.21.15.116:0/2415199147 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f2aec003ca0 con 0x7f2afc146b10 2023-11-28T17:16:59.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.222+0000 7f2af27fc640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ae006b2a0 0x7f2ae006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.226 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.222+0000 7f2af27fc640 1 -- 172.21.15.116:0/2415199147 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f2aec088f50 con 0x7f2afc146b10 2023-11-28T17:16:59.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.222+0000 7f2b01d33640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ae006b2a0 0x7f2ae006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.223+0000 7f2b01d33640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ae006b2a0 0x7f2ae006d760 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2af8036000 tx=0x7f2af803c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:59.227 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.226+0000 7f2af27fc640 1 -- 172.21.15.116:0/2415199147 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2aec053c80 con 0x7f2afc146b10 2023-11-28T17:16:59.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.303+0000 7fd224a9b640 1 Processor -- start 2023-11-28T17:16:59.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.303+0000 7fd224a9b640 1 -- start start 2023-11-28T17:16:59.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.304+0000 7fd224a9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd220145ee0 0x7fd2200ac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.304+0000 7fd224a9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd220146540 0x7fd220160e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.304+0000 7fd224a9b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2200b5110 con 0x7fd220146a90 2023-11-28T17:16:59.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.304+0000 7fd224a9b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd220161dc0 con 0x7fd220145ee0 2023-11-28T17:16:59.306 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.304+0000 7fd224a9b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2200b2190 con 0x7fd220146540 2023-11-28T17:16:59.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.305+0000 7fd21dd9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd220146540 0x7fd220160e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.305+0000 7fd21dd9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd220146540 0x7fd220160e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.116:35594/0 (socket says 172.21.15.116:35594) 2023-11-28T17:16:59.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.305+0000 7fd21dd9b640 1 -- 172.21.15.116:0/1134968924 learned_addr learned my addr 172.21.15.116:0/1134968924 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:59.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.305+0000 7fd21e59c640 1 --2- 172.21.15.116:0/1134968924 >> v2:172.21.15.156:3300/0 conn(0x7fd220145ee0 0x7fd2200ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.306+0000 7fd21dd9b640 1 -- 172.21.15.116:0/1134968924 >> v2:172.21.15.156:3300/0 conn(0x7fd220145ee0 msgr2=0x7fd2200ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:59.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.306+0000 7fd21dd9b640 1 --2- 172.21.15.116:0/1134968924 >> v2:172.21.15.156:3300/0 conn(0x7fd220145ee0 0x7fd2200ac870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.306+0000 7fd21dd9b640 1 -- 172.21.15.116:0/1134968924 >> v1:172.21.15.156:6789/0 conn(0x7fd220146a90 legacy=0x7fd220071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.306+0000 7fd21dd9b640 1 -- 172.21.15.116:0/1134968924 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd220161ae0 con 0x7fd220146540 2023-11-28T17:16:59.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.306+0000 7fd21e59c640 1 --2- 172.21.15.116:0/1134968924 >> v2:172.21.15.156:3300/0 conn(0x7fd220145ee0 0x7fd2200ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:16:59.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.307+0000 7fd21dd9b640 1 --2- 172.21.15.116:0/1134968924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd220146540 0x7fd220160e10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd214009870 tx=0x7fd21402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fa434392c30e73b server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:59.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.310+0000 7fd21d59a640 1 -- 172.21.15.116:0/1134968924 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd214030840 con 0x7fd220146540 2023-11-28T17:16:59.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.310+0000 7fd21d59a640 1 -- 172.21.15.116:0/1134968924 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2140309a0 con 0x7fd220146540 2023-11-28T17:16:59.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.310+0000 7fd21d59a640 1 -- 172.21.15.116:0/1134968924 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd214030d30 con 0x7fd220146540 2023-11-28T17:16:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.310+0000 7fd224a9b640 1 -- 172.21.15.116:0/1134968924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd220146540 msgr2=0x7fd220160e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.310+0000 7fd224a9b640 1 --2- 172.21.15.116:0/1134968924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd220146540 0x7fd220160e10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd214009870 tx=0x7fd21402b980 comp rx=0 tx=0).stop 2023-11-28T17:16:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.311+0000 7fd224a9b640 1 -- 172.21.15.116:0/1134968924 shutdown_connections 2023-11-28T17:16:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.311+0000 7fd224a9b640 1 --2- 172.21.15.116:0/1134968924 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd220146540 0x7fd220160e10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.311+0000 7fd224a9b640 1 --2- 172.21.15.116:0/1134968924 >> v2:172.21.15.156:3300/0 conn(0x7fd220145ee0 0x7fd2200ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.311+0000 7fd224a9b640 1 -- 172.21.15.116:0/1134968924 >> 172.21.15.116:0/1134968924 conn(0x7fd2200b0cf0 msgr2=0x7fd2200b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.311+0000 7fd224a9b640 1 -- 172.21.15.116:0/1134968924 shutdown_connections 2023-11-28T17:16:59.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.311+0000 7fd224a9b640 1 -- 172.21.15.116:0/1134968924 wait complete. 2023-11-28T17:16:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.312+0000 7fd224a9b640 1 Processor -- start 2023-11-28T17:16:59.314 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.312+0000 7fd224a9b640 1 -- start start 2023-11-28T17:16:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.313+0000 7fd224a9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd220145ee0 0x7fd22015cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.313+0000 7fd224a9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd220146540 0x7fd2201434d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.313+0000 7fd224a9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd220146a90 0x7fd220143a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.313+0000 7fd224a9b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22015e480 con 0x7fd220146a90 2023-11-28T17:16:59.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.313+0000 7fd224a9b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22015d870 con 0x7fd220145ee0 2023-11-28T17:16:59.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.313+0000 7fd224a9b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22015db50 con 0x7fd220146540 2023-11-28T17:16:59.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.314+0000 7fd21ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd220146a90 0x7fd220143a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.314+0000 7fd21ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd220146a90 0x7fd220143a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53568/0 (socket says 172.21.15.116:53568) 2023-11-28T17:16:59.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.314+0000 7fd21ed9d640 1 -- 172.21.15.116:0/3995328631 learned_addr learned my addr 172.21.15.116:0/3995328631 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:59.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.314+0000 7fd21dd9b640 1 --2- 172.21.15.116:0/3995328631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd220146540 0x7fd2201434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.314+0000 7fd21e59c640 1 --2- 172.21.15.116:0/3995328631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd220145ee0 0x7fd22015cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.314+0000 7fd21ed9d640 1 -- 172.21.15.116:0/3995328631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd220146540 msgr2=0x7fd2201434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.314+0000 7fd21ed9d640 1 --2- 172.21.15.116:0/3995328631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd220146540 0x7fd2201434d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.314+0000 7fd21ed9d640 1 -- 172.21.15.116:0/3995328631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd220145ee0 msgr2=0x7fd22015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.314+0000 7fd21ed9d640 1 --2- 172.21.15.116:0/3995328631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd220145ee0 0x7fd22015cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.314+0000 7fd21ed9d640 1 -- 172.21.15.116:0/3995328631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd210009400 con 0x7fd220146a90 2023-11-28T17:16:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.314+0000 7fd21ed9d640 1 --2- 172.21.15.116:0/3995328631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd220146a90 0x7fd220143a10 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fd21800c6b0 tx=0x7fd21800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.315+0000 7fd20effd640 1 -- 172.21.15.116:0/3995328631 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218003b70 con 0x7fd220146a90 2023-11-28T17:16:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.315+0000 7fd20effd640 1 -- 172.21.15.116:0/3995328631 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd218003cf0 con 0x7fd220146a90 2023-11-28T17:16:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.315+0000 7fd20effd640 1 -- 172.21.15.116:0/3995328631 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218003ea0 con 0x7fd220146a90 2023-11-28T17:16:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.315+0000 7fd224a9b640 1 -- 172.21.15.116:0/3995328631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2140094d0 con 0x7fd220146a90 2023-11-28T17:16:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.315+0000 7fd224a9b640 1 -- 172.21.15.116:0/3995328631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22015e110 con 0x7fd220146a90 2023-11-28T17:16:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.316+0000 7fd224a9b640 1 -- 172.21.15.116:0/3995328631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22015e5f0 con 0x7fd220146a90 2023-11-28T17:16:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.317+0000 7fd20effd640 1 -- 172.21.15.116:0/3995328631 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fd218005310 con 0x7fd220146a90 2023-11-28T17:16:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.318+0000 7fd20effd640 1 --2- 172.21.15.116:0/3995328631 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd20006b2d0 0x7fd20006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.318+0000 7fd20effd640 1 -- 172.21.15.116:0/3995328631 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fd218089ab0 con 0x7fd220146a90 2023-11-28T17:16:59.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.318+0000 7fd21e59c640 1 --2- 172.21.15.116:0/3995328631 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd20006b2d0 0x7fd20006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.319+0000 7fd21e59c640 1 --2- 172.21.15.116:0/3995328631 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd20006b2d0 0x7fd20006d790 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd210009c60 tx=0x7fd210008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:59.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.323+0000 7fd20effd640 1 -- 172.21.15.116:0/3995328631 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd2180547e0 con 0x7fd220146a90 2023-11-28T17:16:59.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.426+0000 7f2b03f97640 1 -- 172.21.15.116:0/2415199147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2afc0b3950 con 0x7f2afc146b10 2023-11-28T17:16:59.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.427+0000 7f2af27fc640 1 -- 172.21.15.116:0/2415199147 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2aec0572a0 con 0x7f2afc146b10 2023-11-28T17:16:59.428 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:16:59.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.429+0000 7f2b03f97640 1 -- 172.21.15.116:0/2415199147 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ae006b2a0 msgr2=0x7f2ae006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.429+0000 7f2b03f97640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ae006b2a0 0x7f2ae006d760 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2af8036000 tx=0x7f2af803c040 comp rx=0 tx=0).stop 2023-11-28T17:16:59.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.429+0000 7f2b03f97640 1 -- 172.21.15.116:0/2415199147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2afc146b10 msgr2=0x7f2afc13c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.429+0000 7f2b03f97640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2afc146b10 0x7f2afc13c460 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2aec00c410 tx=0x7f2aec00c8d0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.429+0000 7f2b03f97640 1 -- 172.21.15.116:0/2415199147 shutdown_connections 2023-11-28T17:16:59.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.429+0000 7f2b03f97640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f2ae006b2a0 0x7f2ae006d760 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.429+0000 7f2b03f97640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2afc15dfb0 0x7f2afc15c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.429+0000 7f2b03f97640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2afc146b10 0x7f2afc13c460 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.429+0000 7f2b03f97640 1 --2- 172.21.15.116:0/2415199147 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2afc0d0f10 0x7f2afc0b2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.429+0000 7f2b03f97640 1 -- 172.21.15.116:0/2415199147 >> 172.21.15.116:0/2415199147 conn(0x7f2afc070af0 msgr2=0x7f2afc144980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:59.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.429+0000 7f2b03f97640 1 -- 172.21.15.116:0/2415199147 shutdown_connections 2023-11-28T17:16:59.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.429+0000 7f2b03f97640 1 -- 172.21.15.116:0/2415199147 wait complete. 2023-11-28T17:16:59.450 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:16:59.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.515+0000 7fd224a9b640 1 -- 172.21.15.116:0/3995328631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd220069360 con 0x7fd220146a90 2023-11-28T17:16:59.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.516+0000 7fd20effd640 1 -- 172.21.15.116:0/3995328631 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd218057e00 con 0x7fd220146a90 2023-11-28T17:16:59.517 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:16:59.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.518+0000 7fd224a9b640 1 -- 172.21.15.116:0/3995328631 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd20006b2d0 msgr2=0x7fd20006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.518+0000 7fd224a9b640 1 --2- 172.21.15.116:0/3995328631 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd20006b2d0 0x7fd20006d790 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd210009c60 tx=0x7fd210008040 comp rx=0 tx=0).stop 2023-11-28T17:16:59.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.518+0000 7fd224a9b640 1 -- 172.21.15.116:0/3995328631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd220146a90 msgr2=0x7fd220143a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.518+0000 7fd224a9b640 1 --2- 172.21.15.116:0/3995328631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd220146a90 0x7fd220143a10 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fd21800c6b0 tx=0x7fd21800cb70 comp rx=0 tx=0).stop 2023-11-28T17:16:59.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.518+0000 7fd224a9b640 1 -- 172.21.15.116:0/3995328631 shutdown_connections 2023-11-28T17:16:59.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.518+0000 7fd224a9b640 1 --2- 172.21.15.116:0/3995328631 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd20006b2d0 0x7fd20006d790 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.518+0000 7fd224a9b640 1 --2- 172.21.15.116:0/3995328631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd220146a90 0x7fd220143a10 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.520 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.518+0000 7fd224a9b640 1 --2- 172.21.15.116:0/3995328631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd220146540 0x7fd2201434d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.518+0000 7fd224a9b640 1 --2- 172.21.15.116:0/3995328631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd220145ee0 0x7fd22015cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.518+0000 7fd224a9b640 1 -- 172.21.15.116:0/3995328631 >> 172.21.15.116:0/3995328631 conn(0x7fd2200b0cf0 msgr2=0x7fd220160da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:59.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.518+0000 7fd224a9b640 1 -- 172.21.15.116:0/3995328631 shutdown_connections 2023-11-28T17:16:59.521 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.518+0000 7fd224a9b640 1 -- 172.21.15.116:0/3995328631 wait complete. 2023-11-28T17:16:59.536 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:16:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.644+0000 7f8eb5fd5640 1 Processor -- start 2023-11-28T17:16:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.644+0000 7f8eb5fd5640 1 -- start start 2023-11-28T17:16:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.645+0000 7f8eb5fd5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8eb00cf0f0 0x7f8eb00cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.646+0000 7f8eb5fd5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8eb0160e30 0x7f8eb0143420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.646+0000 7f8eb5fd5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eb00b30e0 0x7f8eb006a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.646+0000 7f8eb5fd5640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb00b4700 con 0x7f8eb00cf0f0 2023-11-28T17:16:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.646+0000 7f8eb5fd5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb00cfa10 con 0x7f8eb0160e30 2023-11-28T17:16:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.646+0000 7f8eb5fd5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb01406f0 con 0x7f8eb00b30e0 2023-11-28T17:16:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.646+0000 7f8eaf7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8eb00cf0f0 0x7f8eb00cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.646+0000 7f8eaf7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8eb00cf0f0 0x7f8eb00cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53576/0 (socket says 172.21.15.116:53576) 2023-11-28T17:16:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.646+0000 7f8eaf7fe640 1 -- 172.21.15.116:0/2791938483 learned_addr learned my addr 172.21.15.116:0/2791938483 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.646+0000 7f8eaf7fe640 1 -- 172.21.15.116:0/2791938483 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eb00b30e0 msgr2=0x7f8eb006a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.646+0000 7f8eaf7fe640 1 --2- 172.21.15.116:0/2791938483 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eb00b30e0 0x7f8eb006a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.646+0000 7f8eaf7fe640 1 -- 172.21.15.116:0/2791938483 >> v2:172.21.15.156:3300/0 conn(0x7f8eb0160e30 msgr2=0x7f8eb0143420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.646+0000 7f8eaf7fe640 1 --2- 172.21.15.116:0/2791938483 >> v2:172.21.15.156:3300/0 conn(0x7f8eb0160e30 0x7f8eb0143420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.646+0000 7f8eaf7fe640 1 -- 172.21.15.116:0/2791938483 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb0143960 con 0x7f8eb00cf0f0 2023-11-28T17:16:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.647+0000 7f8eaf7fe640 1 --2- 172.21.15.116:0/2791938483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eb00cf0f0 0x7f8eb00cf4d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0070010 tx=0x7f8ea002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23bcfa2f88a74830 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:59.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.648+0000 7f8eaeffd640 1 -- 172.21.15.116:0/2791938483 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0031870 con 0x7f8eb00cf0f0 2023-11-28T17:16:59.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.648+0000 7f8eaeffd640 1 -- 172.21.15.116:0/2791938483 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea00319d0 con 0x7f8eb00cf0f0 2023-11-28T17:16:59.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.648+0000 7f8eaeffd640 1 -- 172.21.15.116:0/2791938483 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0031d60 con 0x7f8eb00cf0f0 2023-11-28T17:16:59.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.648+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/2791938483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eb00cf0f0 msgr2=0x7f8eb00cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.648+0000 7f8eb5fd5640 1 --2- 172.21.15.116:0/2791938483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eb00cf0f0 0x7f8eb00cf4d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0070010 tx=0x7f8ea002b910 comp rx=0 tx=0).stop 2023-11-28T17:16:59.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.648+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/2791938483 shutdown_connections 2023-11-28T17:16:59.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.648+0000 7f8eb5fd5640 1 --2- 172.21.15.116:0/2791938483 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eb00b30e0 0x7f8eb006a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.648+0000 7f8eb5fd5640 1 --2- 172.21.15.116:0/2791938483 >> v2:172.21.15.156:3300/0 conn(0x7f8eb0160e30 0x7f8eb0143420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.648+0000 7f8eb5fd5640 1 --2- 172.21.15.116:0/2791938483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eb00cf0f0 0x7f8eb00cf4d0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.649+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/2791938483 >> 172.21.15.116:0/2791938483 conn(0x7f8eb00b1de0 msgr2=0x7f8eb00b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:59.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.649+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/2791938483 shutdown_connections 2023-11-28T17:16:59.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.649+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/2791938483 wait complete. 2023-11-28T17:16:59.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.650+0000 7f8eb5fd5640 1 Processor -- start 2023-11-28T17:16:59.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.650+0000 7f8eb5fd5640 1 -- start start 2023-11-28T17:16:59.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.651+0000 7f8eb5fd5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eb00b30e0 0x7f8eb01409c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.651+0000 7f8eb5fd5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eb00cf0f0 0x7f8eb0146130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.651+0000 7f8eb5fd5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eb0160e30 0x7f8eb013c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.651+0000 7f8eb5fd5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb013d300 con 0x7f8eb0160e30 2023-11-28T17:16:59.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.651+0000 7f8eb5fd5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb01f7210 con 0x7f8eb00b30e0 2023-11-28T17:16:59.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.651+0000 7f8eb5fd5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb013d510 con 0x7f8eb00cf0f0 2023-11-28T17:16:59.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.652+0000 7f8eaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eb0160e30 0x7f8eb013c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.652+0000 7f8eaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eb0160e30 0x7f8eb013c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53592/0 (socket says 172.21.15.116:53592) 2023-11-28T17:16:59.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.652+0000 7f8eaffff640 1 -- 172.21.15.116:0/3568790260 learned_addr learned my addr 172.21.15.116:0/3568790260 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:59.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.652+0000 7f8eaf7fe640 1 --2- 172.21.15.116:0/3568790260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eb00b30e0 0x7f8eb01409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.652+0000 7f8ea7fff640 1 --2- 172.21.15.116:0/3568790260 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eb00cf0f0 0x7f8eb0146130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.652+0000 7f8eaffff640 1 -- 172.21.15.116:0/3568790260 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eb00cf0f0 msgr2=0x7f8eb0146130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.652+0000 7f8eaffff640 1 --2- 172.21.15.116:0/3568790260 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eb00cf0f0 0x7f8eb0146130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.652+0000 7f8eaffff640 1 -- 172.21.15.116:0/3568790260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eb00b30e0 msgr2=0x7f8eb01409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.652+0000 7f8eaffff640 1 --2- 172.21.15.116:0/3568790260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eb00b30e0 0x7f8eb01409c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.652+0000 7f8eaffff640 1 -- 172.21.15.116:0/3568790260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea00094d0 con 0x7f8eb0160e30 2023-11-28T17:16:59.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.653+0000 7f8eaffff640 1 --2- 172.21.15.116:0/3568790260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eb0160e30 0x7f8eb013c770 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800b970 tx=0x7f8ea800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:59.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.653+0000 7f8eacff9640 1 -- 172.21.15.116:0/3568790260 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea800cb40 con 0x7f8eb0160e30 2023-11-28T17:16:59.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.653+0000 7f8eacff9640 1 -- 172.21.15.116:0/3568790260 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea800ccc0 con 0x7f8eb0160e30 2023-11-28T17:16:59.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.653+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/3568790260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb013d7f0 con 0x7f8eb0160e30 2023-11-28T17:16:59.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.653+0000 7f8eacff9640 1 -- 172.21.15.116:0/3568790260 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea8014680 con 0x7f8eb0160e30 2023-11-28T17:16:59.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.653+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/3568790260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb0146670 con 0x7f8eb0160e30 2023-11-28T17:16:59.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.654+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/3568790260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb013dd10 con 0x7f8eb0160e30 2023-11-28T17:16:59.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.655+0000 7f8eacff9640 1 -- 172.21.15.116:0/3568790260 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f8ea801d3b0 con 0x7f8eb0160e30 2023-11-28T17:16:59.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.656+0000 7f8eacff9640 1 --2- 172.21.15.116:0/3568790260 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8e9006b2a0 0x7f8e9006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.656+0000 7f8eacff9640 1 -- 172.21.15.116:0/3568790260 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f8ea8089cb0 con 0x7f8eb0160e30 2023-11-28T17:16:59.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.656+0000 7f8eaf7fe640 1 --2- 172.21.15.116:0/3568790260 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8e9006b2a0 0x7f8e9006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.657+0000 7f8eaf7fe640 1 --2- 172.21.15.116:0/3568790260 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8e9006b2a0 0x7f8e9006d760 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0036000 tx=0x7f8ea003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:59.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.660+0000 7f8eacff9640 1 -- 172.21.15.116:0/3568790260 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8eb013dd10 con 0x7f8eb0160e30 2023-11-28T17:16:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.732+0000 7f198e753640 1 Processor -- start 2023-11-28T17:16:59.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.732+0000 7f198e753640 1 -- start start 2023-11-28T17:16:59.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.734+0000 7f198e753640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1988146a20 0x7f19880b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.734+0000 7f198e753640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1988071500 0x7f198815e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.734+0000 7f198e753640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f198815e560 0x7f198806a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.734+0000 7f198e753640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f198815db90 con 0x7f198815e560 2023-11-28T17:16:59.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.734+0000 7f198e753640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19880b3650 con 0x7f1988146a20 2023-11-28T17:16:59.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.734+0000 7f198e753640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f198815d840 con 0x7f1988071500 2023-11-28T17:16:59.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.734+0000 7f198ccf0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f198815e560 0x7f198806a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.734+0000 7f198ccf0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f198815e560 0x7f198806a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53600/0 (socket says 172.21.15.116:53600) 2023-11-28T17:16:59.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.734+0000 7f198ccf0640 1 -- 172.21.15.116:0/982975822 learned_addr learned my addr 172.21.15.116:0/982975822 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:59.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.734+0000 7f1987fff640 1 --2- 172.21.15.116:0/982975822 >> v2:172.21.15.156:3300/0 conn(0x7f1988146a20 0x7f19880b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.734+0000 7f197ffff640 1 --2- 172.21.15.116:0/982975822 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1988071500 0x7f198815e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.735+0000 7f198ccf0640 1 -- 172.21.15.116:0/982975822 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1988071500 msgr2=0x7f198815e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:59.738 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.735+0000 7f198ccf0640 1 --2- 172.21.15.116:0/982975822 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1988071500 0x7f198815e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.735+0000 7f198ccf0640 1 -- 172.21.15.116:0/982975822 >> v2:172.21.15.156:3300/0 conn(0x7f1988146a20 msgr2=0x7f19880b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:16:59.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.735+0000 7f198ccf0640 1 --2- 172.21.15.116:0/982975822 >> v2:172.21.15.156:3300/0 conn(0x7f1988146a20 0x7f19880b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.735+0000 7f198ccf0640 1 -- 172.21.15.116:0/982975822 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19880b38f0 con 0x7f198815e560 2023-11-28T17:16:59.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.736+0000 7f198ccf0640 1 --2- 172.21.15.116:0/982975822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f198815e560 0x7f198806a4c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f19880cffc0 tx=0x7f198002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784fe371287c96f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:59.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.737+0000 7f19877fe640 1 -- 172.21.15.116:0/982975822 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198002c9e0 con 0x7f198815e560 2023-11-28T17:16:59.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.737+0000 7f19877fe640 1 -- 172.21.15.116:0/982975822 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198002cb40 con 0x7f198815e560 2023-11-28T17:16:59.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.737+0000 7f19877fe640 1 -- 172.21.15.116:0/982975822 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19800343f0 con 0x7f198815e560 2023-11-28T17:16:59.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.737+0000 7f198e753640 1 -- 172.21.15.116:0/982975822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f198815e560 msgr2=0x7f198806a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.737+0000 7f198e753640 1 --2- 172.21.15.116:0/982975822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f198815e560 0x7f198806a4c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f19880cffc0 tx=0x7f198002be40 comp rx=0 tx=0).stop 2023-11-28T17:16:59.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.737+0000 7f198e753640 1 -- 172.21.15.116:0/982975822 shutdown_connections 2023-11-28T17:16:59.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.737+0000 7f198e753640 1 --2- 172.21.15.116:0/982975822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f198815e560 0x7f198806a4c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.737+0000 7f198e753640 1 --2- 172.21.15.116:0/982975822 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1988071500 0x7f198815e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.737+0000 7f198e753640 1 --2- 172.21.15.116:0/982975822 >> v2:172.21.15.156:3300/0 conn(0x7f1988146a20 0x7f19880b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.737+0000 7f198e753640 1 -- 172.21.15.116:0/982975822 >> 172.21.15.116:0/982975822 conn(0x7f198806b5d0 msgr2=0x7f198806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:59.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.737+0000 7f198e753640 1 -- 172.21.15.116:0/982975822 shutdown_connections 2023-11-28T17:16:59.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.738+0000 7f198e753640 1 -- 172.21.15.116:0/982975822 wait complete. 2023-11-28T17:16:59.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.739+0000 7f198e753640 1 Processor -- start 2023-11-28T17:16:59.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.739+0000 7f198e753640 1 -- start start 2023-11-28T17:16:59.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.740+0000 7f198e753640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1988071500 0x7f19881409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.740+0000 7f198e753640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1988146a20 0x7f198813fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.740+0000 7f198e753640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f198813d5d0 0x7f19880ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.740+0000 7f198e753640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198813f390 con 0x7f1988146a20 2023-11-28T17:16:59.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.740+0000 7f198e753640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1988141210 con 0x7f1988071500 2023-11-28T17:16:59.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.740+0000 7f198e753640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19881414f0 con 0x7f198813d5d0 2023-11-28T17:16:59.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.741+0000 7f197ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1988146a20 0x7f198813fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.741+0000 7f197ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1988146a20 0x7f198813fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53614/0 (socket says 172.21.15.116:53614) 2023-11-28T17:16:59.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.741+0000 7f197ffff640 1 -- 172.21.15.116:0/1963253217 learned_addr learned my addr 172.21.15.116:0/1963253217 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:16:59.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.741+0000 7f1987fff640 1 --2- 172.21.15.116:0/1963253217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1988071500 0x7f19881409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.741+0000 7f198ccf0640 1 --2- 172.21.15.116:0/1963253217 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f198813d5d0 0x7f19880ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.741+0000 7f197ffff640 1 -- 172.21.15.116:0/1963253217 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f198813d5d0 msgr2=0x7f19880ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.741+0000 7f197ffff640 1 --2- 172.21.15.116:0/1963253217 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f198813d5d0 0x7f19880ae180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.741+0000 7f197ffff640 1 -- 172.21.15.116:0/1963253217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1988071500 msgr2=0x7f19881409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.741+0000 7f197ffff640 1 --2- 172.21.15.116:0/1963253217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1988071500 0x7f19881409e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.741+0000 7f197ffff640 1 -- 172.21.15.116:0/1963253217 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1980009b30 con 0x7f1988146a20 2023-11-28T17:16:59.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.741+0000 7f197ffff640 1 --2- 172.21.15.116:0/1963253217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1988146a20 0x7f198813fda0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f197000d800 tx=0x7f197000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:59.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.742+0000 7f19857fa640 1 -- 172.21.15.116:0/1963253217 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197000b6c0 con 0x7f1988146a20 2023-11-28T17:16:59.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.742+0000 7f19857fa640 1 -- 172.21.15.116:0/1963253217 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197000b820 con 0x7f1988146a20 2023-11-28T17:16:59.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.742+0000 7f19857fa640 1 -- 172.21.15.116:0/1963253217 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197000bbb0 con 0x7f1988146a20 2023-11-28T17:16:59.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.742+0000 7f198e753640 1 -- 172.21.15.116:0/1963253217 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f198813f5a0 con 0x7f1988146a20 2023-11-28T17:16:59.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.742+0000 7f198e753640 1 -- 172.21.15.116:0/1963253217 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1988140f20 con 0x7f1988146a20 2023-11-28T17:16:59.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.743+0000 7f198e753640 1 -- 172.21.15.116:0/1963253217 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1988067980 con 0x7f1988146a20 2023-11-28T17:16:59.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.745+0000 7f19857fa640 1 -- 172.21.15.116:0/1963253217 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f197000bd10 con 0x7f1988146a20 2023-11-28T17:16:59.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.746+0000 7f19857fa640 1 --2- 172.21.15.116:0/1963253217 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f196806b2a0 0x7f196806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:16:59.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.746+0000 7f19857fa640 1 -- 172.21.15.116:0/1963253217 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f197008a2a0 con 0x7f1988146a20 2023-11-28T17:16:59.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.746+0000 7f1987fff640 1 --2- 172.21.15.116:0/1963253217 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f196806b2a0 0x7f196806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:16:59.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.747+0000 7f1987fff640 1 --2- 172.21.15.116:0/1963253217 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f196806b2a0 0x7f196806d760 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1978006fd0 tx=0x7f1978008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:16:59.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.750+0000 7f19857fa640 1 -- 172.21.15.116:0/1963253217 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1970054fd0 con 0x7f1988146a20 2023-11-28T17:16:59.856 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.855+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/3568790260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8eb013dd10 con 0x7f8eb0160e30 2023-11-28T17:16:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.855+0000 7f8eacff9640 1 -- 172.21.15.116:0/3568790260 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8eb013dd10 con 0x7f8eb0160e30 2023-11-28T17:16:59.857 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:16:59.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.858+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/3568790260 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8e9006b2a0 msgr2=0x7f8e9006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.858+0000 7f8eb5fd5640 1 --2- 172.21.15.116:0/3568790260 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8e9006b2a0 0x7f8e9006d760 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0036000 tx=0x7f8ea003c040 comp rx=0 tx=0).stop 2023-11-28T17:16:59.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.858+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/3568790260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eb0160e30 msgr2=0x7f8eb013c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.858+0000 7f8eb5fd5640 1 --2- 172.21.15.116:0/3568790260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eb0160e30 0x7f8eb013c770 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800b970 tx=0x7f8ea800be30 comp rx=0 tx=0).stop 2023-11-28T17:16:59.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.858+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/3568790260 shutdown_connections 2023-11-28T17:16:59.859 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.858+0000 7f8eb5fd5640 1 --2- 172.21.15.116:0/3568790260 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8e9006b2a0 0x7f8e9006d760 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.858+0000 7f8eb5fd5640 1 --2- 172.21.15.116:0/3568790260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8eb0160e30 0x7f8eb013c770 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.858+0000 7f8eb5fd5640 1 --2- 172.21.15.116:0/3568790260 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8eb00cf0f0 0x7f8eb0146130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.858+0000 7f8eb5fd5640 1 --2- 172.21.15.116:0/3568790260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8eb00b30e0 0x7f8eb01409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.858+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/3568790260 >> 172.21.15.116:0/3568790260 conn(0x7f8eb00b1de0 msgr2=0x7f8eb00b3540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:59.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.858+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/3568790260 shutdown_connections 2023-11-28T17:16:59.860 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.858+0000 7f8eb5fd5640 1 -- 172.21.15.116:0/3568790260 wait complete. 2023-11-28T17:16:59.879 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:16:59.956 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.955+0000 7f198e753640 1 -- 172.21.15.116:0/1963253217 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f19881416b0 con 0x7f1988146a20 2023-11-28T17:16:59.957 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.956+0000 7f19857fa640 1 -- 172.21.15.116:0/1963253217 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v52) v1 ==== 80+0+246 (secure 0 0 0) 0x7f19700585f0 con 0x7f1988146a20 2023-11-28T17:16:59.957 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.156:6833/372407661 2023-11-29T17:12:26.038071+0000 2023-11-28T17:16:59.957 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.156:6832/372407661 2023-11-29T17:12:26.038071+0000 2023-11-28T17:16:59.957 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/3251347229 2023-11-29T17:12:06.030424+0000 2023-11-28T17:16:59.958 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/3251347229 2023-11-29T17:12:06.030424+0000 2023-11-28T17:16:59.958 INFO:teuthology.orchestra.run.smithi116.stderr:listed 4 entries 2023-11-28T17:16:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.958+0000 7f198e753640 1 -- 172.21.15.116:0/1963253217 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f196806b2a0 msgr2=0x7f196806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.958+0000 7f198e753640 1 --2- 172.21.15.116:0/1963253217 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f196806b2a0 0x7f196806d760 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1978006fd0 tx=0x7f1978008040 comp rx=0 tx=0).stop 2023-11-28T17:16:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.958+0000 7f198e753640 1 -- 172.21.15.116:0/1963253217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1988146a20 msgr2=0x7f198813fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:16:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.958+0000 7f198e753640 1 --2- 172.21.15.116:0/1963253217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1988146a20 0x7f198813fda0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f197000d800 tx=0x7f197000dcc0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.958+0000 7f198e753640 1 -- 172.21.15.116:0/1963253217 shutdown_connections 2023-11-28T17:16:59.960 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.958+0000 7f198e753640 1 --2- 172.21.15.116:0/1963253217 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f196806b2a0 0x7f196806d760 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.958+0000 7f198e753640 1 --2- 172.21.15.116:0/1963253217 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f198813d5d0 0x7f19880ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.958+0000 7f198e753640 1 --2- 172.21.15.116:0/1963253217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1988146a20 0x7f198813fda0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.958+0000 7f198e753640 1 --2- 172.21.15.116:0/1963253217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1988071500 0x7f19881409e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:16:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.958+0000 7f198e753640 1 -- 172.21.15.116:0/1963253217 >> 172.21.15.116:0/1963253217 conn(0x7f198806b5d0 msgr2=0x7f19880b4220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:16:59.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.959+0000 7f198e753640 1 -- 172.21.15.116:0/1963253217 shutdown_connections 2023-11-28T17:16:59.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:16:59.959+0000 7f198e753640 1 -- 172.21.15.116:0/1963253217 wait complete. 2023-11-28T17:16:59.977 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:16:59.978 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:17:00.021 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:17:00.086 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.085+0000 7fcdddc7c640 1 Processor -- start 2023-11-28T17:17:00.087 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.085+0000 7fcdddc7c640 1 -- start start 2023-11-28T17:17:00.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.086+0000 7fcdddc7c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcdd80b20a0 0x7fcdd80d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.086+0000 7fcdddc7c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fcdd80b2ad0 0x7fcdd80b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.086+0000 7fcdddc7c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd80cdcb0 con 0x7fcdd80d1780 2023-11-28T17:17:00.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.086+0000 7fcdddc7c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd80d39e0 con 0x7fcdd80b20a0 2023-11-28T17:17:00.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.086+0000 7fcdddc7c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd80d3cc0 con 0x7fcdd80b2ad0 2023-11-28T17:17:00.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.087+0000 7fcdd6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcdd80b20a0 0x7fcdd80d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.087+0000 7fcdd6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcdd80b20a0 0x7fcdd80d01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53620/0 (socket says 172.21.15.116:53620) 2023-11-28T17:17:00.089 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.087+0000 7fcdd6ffd640 1 -- 172.21.15.116:0/754545958 learned_addr learned my addr 172.21.15.116:0/754545958 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:00.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.087+0000 7fcdd6ffd640 1 -- 172.21.15.116:0/754545958 >> v2:172.21.15.156:3300/0 conn(0x7fcdd80b2ad0 msgr2=0x7fcdd80b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:00.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.087+0000 7fcdd6ffd640 1 --2- 172.21.15.116:0/754545958 >> v2:172.21.15.156:3300/0 conn(0x7fcdd80b2ad0 0x7fcdd80b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.087+0000 7fcdd6ffd640 1 -- 172.21.15.116:0/754545958 >> v1:172.21.15.156:6789/0 conn(0x7fcdd80d1780 legacy=0x7fcdd80d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:00.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.087+0000 7fcdd6ffd640 1 -- 172.21.15.116:0/754545958 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd80cdf50 con 0x7fcdd80b20a0 2023-11-28T17:17:00.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.088+0000 7fcdd6ffd640 1 --2- 172.21.15.116:0/754545958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd80b20a0 0x7fcdd80d01e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80d1680 tx=0x7fcdcc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f26435eb5b6275 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:00.090 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.089+0000 7fcdd67fc640 1 -- 172.21.15.116:0/754545958 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0317e0 con 0x7fcdd80b20a0 2023-11-28T17:17:00.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.089+0000 7fcdd67fc640 1 -- 172.21.15.116:0/754545958 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdcc031940 con 0x7fcdd80b20a0 2023-11-28T17:17:00.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.089+0000 7fcdd67fc640 1 -- 172.21.15.116:0/754545958 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc031cd0 con 0x7fcdd80b20a0 2023-11-28T17:17:00.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.089+0000 7fcdddc7c640 1 -- 172.21.15.116:0/754545958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd80b20a0 msgr2=0x7fcdd80d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.089+0000 7fcdddc7c640 1 --2- 172.21.15.116:0/754545958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd80b20a0 0x7fcdd80d01e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80d1680 tx=0x7fcdcc02b880 comp rx=0 tx=0).stop 2023-11-28T17:17:00.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.089+0000 7fcdddc7c640 1 -- 172.21.15.116:0/754545958 shutdown_connections 2023-11-28T17:17:00.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.090+0000 7fcdddc7c640 1 --2- 172.21.15.116:0/754545958 >> v2:172.21.15.156:3300/0 conn(0x7fcdd80b2ad0 0x7fcdd80b2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.090+0000 7fcdddc7c640 1 --2- 172.21.15.116:0/754545958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd80b20a0 0x7fcdd80d01e0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.090+0000 7fcdddc7c640 1 -- 172.21.15.116:0/754545958 >> 172.21.15.116:0/754545958 conn(0x7fcdd809c4e0 msgr2=0x7fcdd80d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:00.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.090+0000 7fcdddc7c640 1 -- 172.21.15.116:0/754545958 shutdown_connections 2023-11-28T17:17:00.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.090+0000 7fcdddc7c640 1 -- 172.21.15.116:0/754545958 wait complete. 2023-11-28T17:17:00.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.091+0000 7fcdddc7c640 1 Processor -- start 2023-11-28T17:17:00.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.092+0000 7fcdddc7c640 1 -- start start 2023-11-28T17:17:00.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.092+0000 7fcdddc7c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd80b20a0 0x7fcdd814f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.092+0000 7fcdddc7c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcdd80b2ad0 0x7fcdd815fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.092+0000 7fcdddc7c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdd80d1780 0x7fcdd80b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.092+0000 7fcdddc7c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd815de90 con 0x7fcdd80b20a0 2023-11-28T17:17:00.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.092+0000 7fcdddc7c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80b7220 con 0x7fcdd80d1780 2023-11-28T17:17:00.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.092+0000 7fcdddc7c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80b6c60 con 0x7fcdd80b2ad0 2023-11-28T17:17:00.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.093+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd80b20a0 0x7fcdd814f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.093+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd80b20a0 0x7fcdd814f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53626/0 (socket says 172.21.15.116:53626) 2023-11-28T17:17:00.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.093+0000 7fcdd77fe640 1 -- 172.21.15.116:0/3609897369 learned_addr learned my addr 172.21.15.116:0/3609897369 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:00.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.093+0000 7fcdd77fe640 1 -- 172.21.15.116:0/3609897369 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcdd80b2ad0 msgr2=0x7fcdd815fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.093+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/3609897369 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcdd80b2ad0 0x7fcdd815fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.093+0000 7fcdd77fe640 1 -- 172.21.15.116:0/3609897369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdd80d1780 msgr2=0x7fcdd80b7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.093+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/3609897369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdd80d1780 0x7fcdd80b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.093+0000 7fcdd77fe640 1 -- 172.21.15.116:0/3609897369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdcc0094d0 con 0x7fcdd80b20a0 2023-11-28T17:17:00.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.093+0000 7fcdd77fe640 1 --2- 172.21.15.116:0/3609897369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd80b20a0 0x7fcdd814f1b0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800b3e0 tx=0x7fcdc800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:00.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.094+0000 7fcdbffff640 1 -- 172.21.15.116:0/3609897369 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8011830 con 0x7fcdd80b20a0 2023-11-28T17:17:00.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.094+0000 7fcdbffff640 1 -- 172.21.15.116:0/3609897369 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc80119b0 con 0x7fcdd80b20a0 2023-11-28T17:17:00.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.094+0000 7fcdbffff640 1 -- 172.21.15.116:0/3609897369 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8011d60 con 0x7fcdd80b20a0 2023-11-28T17:17:00.096 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.094+0000 7fcdddc7c640 1 -- 172.21.15.116:0/3609897369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd80b6f40 con 0x7fcdd80b20a0 2023-11-28T17:17:00.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.094+0000 7fcdddc7c640 1 -- 172.21.15.116:0/3609897369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd80caf60 con 0x7fcdd80b20a0 2023-11-28T17:17:00.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.095+0000 7fcdddc7c640 1 -- 172.21.15.116:0/3609897369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd80693d0 con 0x7fcdd80b20a0 2023-11-28T17:17:00.097 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.096+0000 7fcdbffff640 1 -- 172.21.15.116:0/3609897369 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fcdc801a400 con 0x7fcdd80b20a0 2023-11-28T17:17:00.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.097+0000 7fcdbffff640 1 --2- 172.21.15.116:0/3609897369 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcdc406b320 0x7fcdc406d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.097+0000 7fcdbffff640 1 -- 172.21.15.116:0/3609897369 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fcdc8089ec0 con 0x7fcdd80b20a0 2023-11-28T17:17:00.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.097+0000 7fcdd6ffd640 1 --2- 172.21.15.116:0/3609897369 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcdc406b320 0x7fcdc406d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.099 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.098+0000 7fcdd6ffd640 1 --2- 172.21.15.116:0/3609897369 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcdc406b320 0x7fcdc406d7e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc036000 tx=0x7fcdcc030000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:00.102 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.101+0000 7fcdbffff640 1 -- 172.21.15.116:0/3609897369 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcdc8054bd0 con 0x7fcdd80b20a0 2023-11-28T17:17:00.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.258+0000 7fbf60acf640 1 Processor -- start 2023-11-28T17:17:00.260 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.258+0000 7fbf60acf640 1 -- start start 2023-11-28T17:17:00.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.260+0000 7fbf60acf640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf5c0d11b0 0x7fbf5c0b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.260+0000 7fbf60acf640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbf5c0b5110 0x7fbf5c0b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.260+0000 7fbf60acf640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c0b9150 con 0x7fbf5c160950 2023-11-28T17:17:00.261 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.260+0000 7fbf60acf640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf5c160d30 con 0x7fbf5c0d11b0 2023-11-28T17:17:00.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.260+0000 7fbf60acf640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf5c0b5540 con 0x7fbf5c0b5110 2023-11-28T17:17:00.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.260+0000 7fbf5a59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbf5c160950 0x7fbf5c0b8a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:32914/0 (socket says 172.21.15.116:32914) 2023-11-28T17:17:00.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.260+0000 7fbf5a59c640 1 -- 172.21.15.116:0/3812376166 learned_addr learned my addr 172.21.15.116:0/3812376166 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:17:00.262 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.260+0000 7fbf51d9b640 1 --2- 172.21.15.116:0/3812376166 >> v2:172.21.15.116:3300/0 conn(0x7fbf5c0d11b0 0x7fbf5c0b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.260+0000 7fbf5ad9d640 1 --2- 172.21.15.116:0/3812376166 >> v2:172.21.15.156:3300/0 conn(0x7fbf5c0b5110 0x7fbf5c0b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.261+0000 7fbf59d9b640 1 -- 172.21.15.116:0/3812376166 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380425014 0 0) 0x7fbf44001110 con 0x7fbf5c160950 2023-11-28T17:17:00.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.261+0000 7fbf59d9b640 1 -- 172.21.15.116:0/3812376166 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf48003520 con 0x7fbf5c160950 2023-11-28T17:17:00.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.261+0000 7fbf51d9b640 1 -- 172.21.15.116:0/3812376166 >> v2:172.21.15.156:3300/0 conn(0x7fbf5c0b5110 msgr2=0x7fbf5c0b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:00.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.261+0000 7fbf51d9b640 1 --2- 172.21.15.116:0/3812376166 >> v2:172.21.15.156:3300/0 conn(0x7fbf5c0b5110 0x7fbf5c0b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.261+0000 7fbf51d9b640 1 -- 172.21.15.116:0/3812376166 >> v1:172.21.15.116:6789/0 conn(0x7fbf5c160950 legacy=0x7fbf5c0b8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.261+0000 7fbf51d9b640 1 -- 172.21.15.116:0/3812376166 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c0d1650 con 0x7fbf5c0d11b0 2023-11-28T17:17:00.263 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.262+0000 7fbf51d9b640 1 --2- 172.21.15.116:0/3812376166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf5c0d11b0 0x7fbf5c0b4bd0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c0b8790 tx=0x7fbf4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3522be3629094 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:00.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.262+0000 7fbf59d9b640 1 -- 172.21.15.116:0/3812376166 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c031830 con 0x7fbf5c0d11b0 2023-11-28T17:17:00.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.262+0000 7fbf59d9b640 1 -- 172.21.15.116:0/3812376166 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4c031990 con 0x7fbf5c0d11b0 2023-11-28T17:17:00.264 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.262+0000 7fbf59d9b640 1 -- 172.21.15.116:0/3812376166 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c031d20 con 0x7fbf5c0d11b0 2023-11-28T17:17:00.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.263+0000 7fbf60acf640 1 -- 172.21.15.116:0/3812376166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf5c0d11b0 msgr2=0x7fbf5c0b4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.263+0000 7fbf60acf640 1 --2- 172.21.15.116:0/3812376166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf5c0d11b0 0x7fbf5c0b4bd0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c0b8790 tx=0x7fbf4c02b8d0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.263+0000 7fbf60acf640 1 -- 172.21.15.116:0/3812376166 shutdown_connections 2023-11-28T17:17:00.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.263+0000 7fbf60acf640 1 --2- 172.21.15.116:0/3812376166 >> v2:172.21.15.156:3300/0 conn(0x7fbf5c0b5110 0x7fbf5c0b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.263+0000 7fbf60acf640 1 --2- 172.21.15.116:0/3812376166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf5c0d11b0 0x7fbf5c0b4bd0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.263+0000 7fbf60acf640 1 -- 172.21.15.116:0/3812376166 >> 172.21.15.116:0/3812376166 conn(0x7fbf5c072220 msgr2=0x7fbf5c0cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:00.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.263+0000 7fbf60acf640 1 -- 172.21.15.116:0/3812376166 shutdown_connections 2023-11-28T17:17:00.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.263+0000 7fbf60acf640 1 -- 172.21.15.116:0/3812376166 wait complete. 2023-11-28T17:17:00.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.265+0000 7fbf60acf640 1 Processor -- start 2023-11-28T17:17:00.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.265+0000 7fbf60acf640 1 -- start start 2023-11-28T17:17:00.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.266+0000 7fbf60acf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf5c0b5110 0x7fbf5c0c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.266+0000 7fbf60acf640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf5c0d11b0 0x7fbf5c0b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.266+0000 7fbf60acf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf5c160950 0x7fbf5c0cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.266+0000 7fbf60acf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0b70a0 con 0x7fbf5c160950 2023-11-28T17:17:00.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.266+0000 7fbf60acf640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0ca220 con 0x7fbf5c0b5110 2023-11-28T17:17:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.266+0000 7fbf60acf640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0b7300 con 0x7fbf5c0d11b0 2023-11-28T17:17:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.266+0000 7fbf5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf5c160950 0x7fbf5c0cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.266+0000 7fbf5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf5c160950 0x7fbf5c0cc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53650/0 (socket says 172.21.15.116:53650) 2023-11-28T17:17:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.266+0000 7fbf5ad9d640 1 -- 172.21.15.116:0/1728184261 learned_addr learned my addr 172.21.15.116:0/1728184261 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.267+0000 7fbf51d9b640 1 --2- 172.21.15.116:0/1728184261 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf5c0d11b0 0x7fbf5c0b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.267+0000 7fbf5a59c640 1 --2- 172.21.15.116:0/1728184261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf5c0b5110 0x7fbf5c0c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.267+0000 7fbf5ad9d640 1 -- 172.21.15.116:0/1728184261 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf5c0d11b0 msgr2=0x7fbf5c0b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.267+0000 7fbf5ad9d640 1 --2- 172.21.15.116:0/1728184261 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf5c0d11b0 0x7fbf5c0b6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.267+0000 7fbf5ad9d640 1 -- 172.21.15.116:0/1728184261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf5c0b5110 msgr2=0x7fbf5c0c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.267+0000 7fbf5ad9d640 1 --2- 172.21.15.116:0/1728184261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf5c0b5110 0x7fbf5c0c96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.267+0000 7fbf5ad9d640 1 -- 172.21.15.116:0/1728184261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c009400 con 0x7fbf5c160950 2023-11-28T17:17:00.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.267+0000 7fbf5ad9d640 1 --2- 172.21.15.116:0/1728184261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf5c160950 0x7fbf5c0cc900 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fbf54007e40 tx=0x7fbf5400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:00.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.267+0000 7fbf537fe640 1 -- 172.21.15.116:0/1728184261 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54014620 con 0x7fbf5c160950 2023-11-28T17:17:00.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.267+0000 7fbf537fe640 1 -- 172.21.15.116:0/1728184261 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5400ee90 con 0x7fbf5c160950 2023-11-28T17:17:00.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.267+0000 7fbf60acf640 1 -- 172.21.15.116:0/1728184261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c0b75e0 con 0x7fbf5c160950 2023-11-28T17:17:00.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.267+0000 7fbf537fe640 1 -- 172.21.15.116:0/1728184261 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf540149e0 con 0x7fbf5c160950 2023-11-28T17:17:00.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.268+0000 7fbf60acf640 1 -- 172.21.15.116:0/1728184261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c0b7ba0 con 0x7fbf5c160950 2023-11-28T17:17:00.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.269+0000 7fbf60acf640 1 -- 172.21.15.116:0/1728184261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5c0ca440 con 0x7fbf5c160950 2023-11-28T17:17:00.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.271+0000 7fbf537fe640 1 -- 172.21.15.116:0/1728184261 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fbf54014b40 con 0x7fbf5c160950 2023-11-28T17:17:00.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.271+0000 7fbf537fe640 1 --2- 172.21.15.116:0/1728184261 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf3c06b2a0 0x7fbf3c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.272+0000 7fbf537fe640 1 -- 172.21.15.116:0/1728184261 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fbf5408a6c0 con 0x7fbf5c160950 2023-11-28T17:17:00.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.272+0000 7fbf5a59c640 1 --2- 172.21.15.116:0/1728184261 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf3c06b2a0 0x7fbf3c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.272+0000 7fbf5a59c640 1 --2- 172.21.15.116:0/1728184261 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf3c06b2a0 0x7fbf3c06d760 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbf44001370 tx=0x7fbf44009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:00.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.276+0000 7fbf537fe640 1 -- 172.21.15.116:0/1728184261 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf5c0ca440 con 0x7fbf5c160950 2023-11-28T17:17:00.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.319+0000 7fcdddc7c640 1 -- 172.21.15.116:0/3609897369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcdd815d900 con 0x7fcdd80b20a0 2023-11-28T17:17:00.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.320+0000 7fcdbffff640 1 -- 172.21.15.116:0/3609897369 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v52) v1 ==== 80+0+246 (secure 0 0 0) 0x7fcdc8015020 con 0x7fcdd80b20a0 2023-11-28T17:17:00.322 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.156:6833/372407661 2023-11-29T17:12:26.038071+0000 2023-11-28T17:17:00.322 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.156:6832/372407661 2023-11-29T17:12:26.038071+0000 2023-11-28T17:17:00.322 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/3251347229 2023-11-29T17:12:06.030424+0000 2023-11-28T17:17:00.322 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/3251347229 2023-11-29T17:12:06.030424+0000 2023-11-28T17:17:00.322 INFO:teuthology.orchestra.run.smithi116.stderr:listed 4 entries 2023-11-28T17:17:00.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.322+0000 7fcdddc7c640 1 -- 172.21.15.116:0/3609897369 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcdc406b320 msgr2=0x7fcdc406d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.322+0000 7fcdddc7c640 1 --2- 172.21.15.116:0/3609897369 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcdc406b320 0x7fcdc406d7e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc036000 tx=0x7fcdcc030000 comp rx=0 tx=0).stop 2023-11-28T17:17:00.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.323+0000 7fcdddc7c640 1 -- 172.21.15.116:0/3609897369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd80b20a0 msgr2=0x7fcdd814f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.323+0000 7fcdddc7c640 1 --2- 172.21.15.116:0/3609897369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd80b20a0 0x7fcdd814f1b0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800b3e0 tx=0x7fcdc800b8a0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.323+0000 7fcdddc7c640 1 -- 172.21.15.116:0/3609897369 shutdown_connections 2023-11-28T17:17:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.323+0000 7fcdddc7c640 1 --2- 172.21.15.116:0/3609897369 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcdc406b320 0x7fcdc406d7e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.323+0000 7fcdddc7c640 1 --2- 172.21.15.116:0/3609897369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdd80d1780 0x7fcdd80b7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.323+0000 7fcdddc7c640 1 --2- 172.21.15.116:0/3609897369 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcdd80b2ad0 0x7fcdd815fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.323+0000 7fcdddc7c640 1 --2- 172.21.15.116:0/3609897369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd80b20a0 0x7fcdd814f1b0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.323+0000 7fcdddc7c640 1 -- 172.21.15.116:0/3609897369 >> 172.21.15.116:0/3609897369 conn(0x7fcdd809c4e0 msgr2=0x7fcdd80d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.323+0000 7fcdddc7c640 1 -- 172.21.15.116:0/3609897369 shutdown_connections 2023-11-28T17:17:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.323+0000 7fcdddc7c640 1 -- 172.21.15.116:0/3609897369 wait complete. 2023-11-28T17:17:00.343 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2023-11-28T17:17:00.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:17:00.343 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-11-28T17:17:00.472 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:17:00.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:17:00.472 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-28T17:17:00.478 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:17:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.474+0000 7fbf60acf640 1 -- 172.21.15.116:0/1728184261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf5c0ca440 con 0x7fbf5c160950 2023-11-28T17:17:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.475+0000 7fbf537fe640 1 -- 172.21.15.116:0/1728184261 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+17073 (secure 0 0 0) 0x7fbf5c0ca440 con 0x7fbf5c160950 2023-11-28T17:17:00.479 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":52,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:12:26.038218+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:11:21.518078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T17:11:22.672451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":0}]},{"pgid":"7.8","mappings":[{"from":5,"to":7}]},{"pgid":"7.1b","mappings":[{"from":5,"to":7}]},{"pgid":"7.30","mappings":[{"from":5,"to":7}]},{"pgid":"7.31","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6833/372407661":"2023-11-29T17:12:26.038071+0000","172.21.15.156:6832/372407661":"2023-11-29T17:12:26.038071+0000","172.21.15.116:6835/3251347229":"2023-11-29T17:12:06.030424+0000","172.21.15.116:6834/3251347229":"2023-11-29T17:12:06.030424+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:17:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.477+0000 7fbf60acf640 1 -- 172.21.15.116:0/1728184261 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf3c06b2a0 msgr2=0x7fbf3c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.477+0000 7fbf60acf640 1 --2- 172.21.15.116:0/1728184261 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf3c06b2a0 0x7fbf3c06d760 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbf44001370 tx=0x7fbf44009040 comp rx=0 tx=0).stop 2023-11-28T17:17:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.478+0000 7fbf60acf640 1 -- 172.21.15.116:0/1728184261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf5c160950 msgr2=0x7fbf5c0cc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.478+0000 7fbf60acf640 1 --2- 172.21.15.116:0/1728184261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf5c160950 0x7fbf5c0cc900 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fbf54007e40 tx=0x7fbf5400e4b0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.478+0000 7fbf60acf640 1 -- 172.21.15.116:0/1728184261 shutdown_connections 2023-11-28T17:17:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.478+0000 7fbf60acf640 1 --2- 172.21.15.116:0/1728184261 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fbf3c06b2a0 0x7fbf3c06d760 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.478+0000 7fbf60acf640 1 --2- 172.21.15.116:0/1728184261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf5c160950 0x7fbf5c0cc900 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.478+0000 7fbf60acf640 1 --2- 172.21.15.116:0/1728184261 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf5c0d11b0 0x7fbf5c0b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.478+0000 7fbf60acf640 1 --2- 172.21.15.116:0/1728184261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf5c0b5110 0x7fbf5c0c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.478+0000 7fbf60acf640 1 -- 172.21.15.116:0/1728184261 >> 172.21.15.116:0/1728184261 conn(0x7fbf5c072220 msgr2=0x7fbf5c0b2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.478+0000 7fbf60acf640 1 -- 172.21.15.116:0/1728184261 shutdown_connections 2023-11-28T17:17:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.478+0000 7fbf60acf640 1 -- 172.21.15.116:0/1728184261 wait complete. 2023-11-28T17:17:00.497 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:11:21.518078+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:11:22.672451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:17:00.497 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:17:00.516 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:17:00.517 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:17:00.579 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:17:00.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.753+0000 7f8d97343640 1 Processor -- start 2023-11-28T17:17:00.754 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.753+0000 7f8d97343640 1 -- start start 2023-11-28T17:17:00.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.755+0000 7f8d97343640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d9015d480 0x7f8d900d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.755+0000 7f8d97343640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8d900d01e0 0x7f8d90160270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.755+0000 7f8d97343640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9015ddc0 con 0x7f8d900d1310 2023-11-28T17:17:00.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.755+0000 7f8d97343640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d9015e0a0 con 0x7f8d900d01e0 2023-11-28T17:17:00.756 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.755+0000 7f8d97343640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9015e380 con 0x7f8d9015d480 2023-11-28T17:17:00.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.755+0000 7f8d950df640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d9015d480 0x7f8d900d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.756+0000 7f8d958e0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8d900d01e0 0x7f8d90160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.756+0000 7f8d948de640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f8d900d1310 0x7f8d900cfad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.116:45118/0 (socket says 172.21.15.116:45118) 2023-11-28T17:17:00.757 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.756+0000 7f8d948de640 1 -- 172.21.15.116:0/1060507269 learned_addr learned my addr 172.21.15.116:0/1060507269 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:17:00.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.756+0000 7f8d958e0640 1 -- 172.21.15.116:0/1060507269 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d9015d480 msgr2=0x7f8d900d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:00.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.756+0000 7f8d958e0640 1 --2- 172.21.15.116:0/1060507269 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d9015d480 0x7f8d900d0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.756+0000 7f8d958e0640 1 -- 172.21.15.116:0/1060507269 >> v1:172.21.15.156:6789/0 conn(0x7f8d900d1310 legacy=0x7f8d900cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.756+0000 7f8d958e0640 1 -- 172.21.15.116:0/1060507269 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9015e660 con 0x7f8d900d01e0 2023-11-28T17:17:00.759 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.756+0000 7f8d950df640 1 --2- 172.21.15.116:0/1060507269 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d9015d480 0x7f8d900d0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T17:17:00.760 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.758+0000 7f8d958e0640 1 --2- 172.21.15.116:0/1060507269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d900d01e0 0x7f8d90160270 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009d90 tx=0x7f8d8802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ae536de51567787 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.758+0000 7f8d87fff640 1 -- 172.21.15.116:0/1060507269 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8802cc50 con 0x7f8d900d01e0 2023-11-28T17:17:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.758+0000 7f8d87fff640 1 -- 172.21.15.116:0/1060507269 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8802cdb0 con 0x7f8d900d01e0 2023-11-28T17:17:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.758+0000 7f8d87fff640 1 -- 172.21.15.116:0/1060507269 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88034660 con 0x7f8d900d01e0 2023-11-28T17:17:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.759+0000 7f8d97343640 1 -- 172.21.15.116:0/1060507269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d900d01e0 msgr2=0x7f8d90160270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.759+0000 7f8d97343640 1 --2- 172.21.15.116:0/1060507269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d900d01e0 0x7f8d90160270 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009d90 tx=0x7f8d8802bea0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.759+0000 7f8d97343640 1 -- 172.21.15.116:0/1060507269 shutdown_connections 2023-11-28T17:17:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.759+0000 7f8d97343640 1 --2- 172.21.15.116:0/1060507269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d900d01e0 0x7f8d90160270 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.759+0000 7f8d97343640 1 --2- 172.21.15.116:0/1060507269 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d9015d480 0x7f8d900d0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.759+0000 7f8d97343640 1 -- 172.21.15.116:0/1060507269 >> 172.21.15.116:0/1060507269 conn(0x7f8d9009c540 msgr2=0x7f8d900d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.759+0000 7f8d97343640 1 -- 172.21.15.116:0/1060507269 shutdown_connections 2023-11-28T17:17:00.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.759+0000 7f8d97343640 1 -- 172.21.15.116:0/1060507269 wait complete. 2023-11-28T17:17:00.764 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.761+0000 7f8d97343640 1 Processor -- start 2023-11-28T17:17:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.761+0000 7f8d97343640 1 -- start start 2023-11-28T17:17:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d97343640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d900d01e0 0x7f8d901f2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d97343640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d900d1310 0x7f8d901f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d97343640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d9015d480 0x7f8d901487f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d97343640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900b1610 con 0x7f8d9015d480 2023-11-28T17:17:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d97343640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d90146be0 con 0x7f8d900d01e0 2023-11-28T17:17:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d97343640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d901f3100 con 0x7f8d900d1310 2023-11-28T17:17:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d958e0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d9015d480 0x7f8d901487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.765 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d958e0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d9015d480 0x7f8d901487f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53662/0 (socket says 172.21.15.116:53662) 2023-11-28T17:17:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d958e0640 1 -- 172.21.15.116:0/2896611728 learned_addr learned my addr 172.21.15.116:0/2896611728 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d948de640 1 --2- 172.21.15.116:0/2896611728 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d900d1310 0x7f8d901f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d958e0640 1 -- 172.21.15.116:0/2896611728 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d900d1310 msgr2=0x7f8d901f2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d950df640 1 --2- 172.21.15.116:0/2896611728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d900d01e0 0x7f8d901f2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d958e0640 1 --2- 172.21.15.116:0/2896611728 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d900d1310 0x7f8d901f2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d958e0640 1 -- 172.21.15.116:0/2896611728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d900d01e0 msgr2=0x7f8d901f2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.766 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d958e0640 1 --2- 172.21.15.116:0/2896611728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d900d01e0 0x7f8d901f2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.762+0000 7f8d958e0640 1 -- 172.21.15.116:0/2896611728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c009400 con 0x7f8d9015d480 2023-11-28T17:17:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.763+0000 7f8d958e0640 1 --2- 172.21.15.116:0/2896611728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d9015d480 0x7f8d901487f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009d90 tx=0x7f8d88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:00.769 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.763+0000 7f8d85ffb640 1 -- 172.21.15.116:0/2896611728 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88040650 con 0x7f8d9015d480 2023-11-28T17:17:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.763+0000 7f8d85ffb640 1 -- 172.21.15.116:0/2896611728 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d880407b0 con 0x7f8d9015d480 2023-11-28T17:17:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.763+0000 7f8d85ffb640 1 -- 172.21.15.116:0/2896611728 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88040b60 con 0x7f8d9015d480 2023-11-28T17:17:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.763+0000 7f8d97343640 1 -- 172.21.15.116:0/2896611728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d88009a70 con 0x7f8d9015d480 2023-11-28T17:17:00.770 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.763+0000 7f8d97343640 1 -- 172.21.15.116:0/2896611728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d900ca010 con 0x7f8d9015d480 2023-11-28T17:17:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.764+0000 7f8d97343640 1 -- 172.21.15.116:0/2896611728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d90067980 con 0x7f8d9015d480 2023-11-28T17:17:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.765+0000 7f8d85ffb640 1 -- 172.21.15.116:0/2896611728 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f8d88040cc0 con 0x7f8d9015d480 2023-11-28T17:17:00.771 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.766+0000 7f8d85ffb640 1 --2- 172.21.15.116:0/2896611728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8d7406b2a0 0x7f8d7406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.766+0000 7f8d85ffb640 1 -- 172.21.15.116:0/2896611728 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f8d880acaa0 con 0x7f8d9015d480 2023-11-28T17:17:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.766+0000 7f8d950df640 1 --2- 172.21.15.116:0/2896611728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8d7406b2a0 0x7f8d7406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.772 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.767+0000 7f8d950df640 1 --2- 172.21.15.116:0/2896611728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8d7406b2a0 0x7f8d7406d760 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009ae0 tx=0x7f8d8c008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:00.773 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.770+0000 7f8d85ffb640 1 -- 172.21.15.116:0/2896611728 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d880777c0 con 0x7f8d9015d480 2023-11-28T17:17:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.786+0000 7fcd21a2e640 1 Processor -- start 2023-11-28T17:17:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.786+0000 7fcd21a2e640 1 -- start start 2023-11-28T17:17:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.787+0000 7fcd21a2e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fcd1c0ce840 0x7fcd1c0d1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.787+0000 7fcd21a2e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcd1c0c71a0 0x7fcd1c0c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.787+0000 7fcd21a2e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1c0d09b0 con 0x7fcd1c15d4d0 2023-11-28T17:17:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.787+0000 7fcd21a2e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1c0cf710 con 0x7fcd1c0c71a0 2023-11-28T17:17:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.787+0000 7fcd21a2e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1c0cee70 con 0x7fcd1c0ce840 2023-11-28T17:17:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.788+0000 7fcd20a2c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcd1c15d4d0 0x7fcd1c0d0e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:32920/0 (socket says 172.21.15.116:32920) 2023-11-28T17:17:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.788+0000 7fcd20a2c640 1 -- 172.21.15.116:0/1121400328 learned_addr learned my addr 172.21.15.116:0/1121400328 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:17:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.788+0000 7fcd2122d640 1 --2- 172.21.15.116:0/1121400328 >> v2:172.21.15.116:3300/0 conn(0x7fcd1c0c71a0 0x7fcd1c0c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.788+0000 7fcd1bfff640 1 --2- 172.21.15.116:0/1121400328 >> v2:172.21.15.156:3300/0 conn(0x7fcd1c0ce840 0x7fcd1c0d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.788+0000 7fcd1b7fe640 1 -- 172.21.15.116:0/1121400328 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865910372 0 0) 0x7fcd1c0d09b0 con 0x7fcd1c15d4d0 2023-11-28T17:17:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.788+0000 7fcd1b7fe640 1 -- 172.21.15.116:0/1121400328 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd0c003520 con 0x7fcd1c15d4d0 2023-11-28T17:17:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.788+0000 7fcd2122d640 1 -- 172.21.15.116:0/1121400328 >> v2:172.21.15.156:3300/0 conn(0x7fcd1c0ce840 msgr2=0x7fcd1c0d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:00.791 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.788+0000 7fcd2122d640 1 --2- 172.21.15.116:0/1121400328 >> v2:172.21.15.156:3300/0 conn(0x7fcd1c0ce840 0x7fcd1c0d1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.788+0000 7fcd2122d640 1 -- 172.21.15.116:0/1121400328 >> v1:172.21.15.116:6789/0 conn(0x7fcd1c15d4d0 legacy=0x7fcd1c0d0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.792 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.788+0000 7fcd2122d640 1 -- 172.21.15.116:0/1121400328 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c0cf150 con 0x7fcd1c0c71a0 2023-11-28T17:17:00.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.790+0000 7fcd2122d640 1 --2- 172.21.15.116:0/1121400328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd1c0c71a0 0x7fcd1c0c7630 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c067e10 tx=0x7fcd1402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3576ed314fc19ef3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:00.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.790+0000 7fcd1b7fe640 1 -- 172.21.15.116:0/1121400328 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1402cc80 con 0x7fcd1c0c71a0 2023-11-28T17:17:00.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.790+0000 7fcd1b7fe640 1 -- 172.21.15.116:0/1121400328 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1402cde0 con 0x7fcd1c0c71a0 2023-11-28T17:17:00.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.790+0000 7fcd1b7fe640 1 -- 172.21.15.116:0/1121400328 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14034660 con 0x7fcd1c0c71a0 2023-11-28T17:17:00.794 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.791+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1121400328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd1c0c71a0 msgr2=0x7fcd1c0c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.791+0000 7fcd21a2e640 1 --2- 172.21.15.116:0/1121400328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd1c0c71a0 0x7fcd1c0c7630 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c067e10 tx=0x7fcd1402bed0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.791+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1121400328 shutdown_connections 2023-11-28T17:17:00.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.791+0000 7fcd21a2e640 1 --2- 172.21.15.116:0/1121400328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd1c0c71a0 0x7fcd1c0c7630 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.791+0000 7fcd21a2e640 1 --2- 172.21.15.116:0/1121400328 >> v2:172.21.15.156:3300/0 conn(0x7fcd1c0ce840 0x7fcd1c0d1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.791+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1121400328 >> 172.21.15.116:0/1121400328 conn(0x7fcd1c0d2f90 msgr2=0x7fcd1c0d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:00.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.791+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1121400328 shutdown_connections 2023-11-28T17:17:00.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.791+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1121400328 wait complete. 2023-11-28T17:17:00.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.792+0000 7fcd21a2e640 1 Processor -- start 2023-11-28T17:17:00.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.793+0000 7fcd21a2e640 1 -- start start 2023-11-28T17:17:00.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.793+0000 7fcd21a2e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd1c0c71a0 0x7fcd1c150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.793+0000 7fcd21a2e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd1c0ce840 0x7fcd1c1578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.794+0000 7fcd21a2e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd1c15d4d0 0x7fcd1c0c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.794+0000 7fcd21a2e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c1f2ce0 con 0x7fcd1c0ce840 2023-11-28T17:17:00.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.794+0000 7fcd21a2e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c150990 con 0x7fcd1c15d4d0 2023-11-28T17:17:00.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.794+0000 7fcd21a2e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c158260 con 0x7fcd1c0c71a0 2023-11-28T17:17:00.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.794+0000 7fcd1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd1c0ce840 0x7fcd1c1578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.794+0000 7fcd1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd1c0ce840 0x7fcd1c1578c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53686/0 (socket says 172.21.15.116:53686) 2023-11-28T17:17:00.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.794+0000 7fcd1bfff640 1 -- 172.21.15.116:0/1696838231 learned_addr learned my addr 172.21.15.116:0/1696838231 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.794+0000 7fcd20a2c640 1 --2- 172.21.15.116:0/1696838231 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd1c0c71a0 0x7fcd1c150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.794+0000 7fcd1bfff640 1 -- 172.21.15.116:0/1696838231 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd1c0c71a0 msgr2=0x7fcd1c150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.794+0000 7fcd1bfff640 1 --2- 172.21.15.116:0/1696838231 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd1c0c71a0 0x7fcd1c150450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.794+0000 7fcd1bfff640 1 -- 172.21.15.116:0/1696838231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd1c15d4d0 msgr2=0x7fcd1c0c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.794+0000 7fcd1bfff640 1 --2- 172.21.15.116:0/1696838231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd1c15d4d0 0x7fcd1c0c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.794+0000 7fcd1bfff640 1 -- 172.21.15.116:0/1696838231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd14009a70 con 0x7fcd1c0ce840 2023-11-28T17:17:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.795+0000 7fcd1bfff640 1 --2- 172.21.15.116:0/1696838231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd1c0ce840 0x7fcd1c1578c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fcd0800d800 tx=0x7fcd0800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.795+0000 7fcd197fa640 1 -- 172.21.15.116:0/1696838231 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0800b6c0 con 0x7fcd1c0ce840 2023-11-28T17:17:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.795+0000 7fcd197fa640 1 -- 172.21.15.116:0/1696838231 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd0800b820 con 0x7fcd1c0ce840 2023-11-28T17:17:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.795+0000 7fcd197fa640 1 -- 172.21.15.116:0/1696838231 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0800bbb0 con 0x7fcd1c0ce840 2023-11-28T17:17:00.798 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.795+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1696838231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1c0cb210 con 0x7fcd1c0ce840 2023-11-28T17:17:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.795+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1696838231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1c0cb7d0 con 0x7fcd1c0ce840 2023-11-28T17:17:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.796+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1696838231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1c067c40 con 0x7fcd1c0ce840 2023-11-28T17:17:00.799 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.797+0000 7fcd197fa640 1 -- 172.21.15.116:0/1696838231 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fcd0800bd10 con 0x7fcd1c0ce840 2023-11-28T17:17:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.798+0000 7fcd197fa640 1 --2- 172.21.15.116:0/1696838231 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcd0006b2a0 0x7fcd0006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.798+0000 7fcd197fa640 1 -- 172.21.15.116:0/1696838231 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fcd0808a320 con 0x7fcd1c0ce840 2023-11-28T17:17:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.798+0000 7fcd20a2c640 1 --2- 172.21.15.116:0/1696838231 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcd0006b2a0 0x7fcd0006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:00.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.799+0000 7fcd20a2c640 1 --2- 172.21.15.116:0/1696838231 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcd0006b2a0 0x7fcd0006d760 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fcd100011e0 tx=0x7fcd10009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:00.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.802+0000 7fcd197fa640 1 -- 172.21.15.116:0/1696838231 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd08054fc0 con 0x7fcd1c0ce840 2023-11-28T17:17:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.957+0000 7f8d97343640 1 -- 172.21.15.116:0/2896611728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8d90146d50 con 0x7f8d9015d480 2023-11-28T17:17:00.958 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.957+0000 7f8d85ffb640 1 -- 172.21.15.116:0/2896611728 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8d88009450 con 0x7f8d9015d480 2023-11-28T17:17:00.959 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:17:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.959+0000 7f8d97343640 1 -- 172.21.15.116:0/2896611728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8d7406b2a0 msgr2=0x7f8d7406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.960+0000 7f8d97343640 1 --2- 172.21.15.116:0/2896611728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8d7406b2a0 0x7f8d7406d760 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009ae0 tx=0x7f8d8c008040 comp rx=0 tx=0).stop 2023-11-28T17:17:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.960+0000 7f8d97343640 1 -- 172.21.15.116:0/2896611728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d9015d480 msgr2=0x7f8d901487f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.960+0000 7f8d97343640 1 --2- 172.21.15.116:0/2896611728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d9015d480 0x7f8d901487f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009d90 tx=0x7f8d88004a40 comp rx=0 tx=0).stop 2023-11-28T17:17:00.961 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.960+0000 7f8d97343640 1 -- 172.21.15.116:0/2896611728 shutdown_connections 2023-11-28T17:17:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.960+0000 7f8d97343640 1 --2- 172.21.15.116:0/2896611728 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8d7406b2a0 0x7f8d7406d760 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.960+0000 7f8d97343640 1 --2- 172.21.15.116:0/2896611728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d9015d480 0x7f8d901487f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.960+0000 7f8d97343640 1 --2- 172.21.15.116:0/2896611728 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8d900d1310 0x7f8d901f2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.960+0000 7f8d97343640 1 --2- 172.21.15.116:0/2896611728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d900d01e0 0x7f8d901f2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.960+0000 7f8d97343640 1 -- 172.21.15.116:0/2896611728 >> 172.21.15.116:0/2896611728 conn(0x7f8d9009c540 msgr2=0x7f8d9006b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.960+0000 7f8d97343640 1 -- 172.21.15.116:0/2896611728 shutdown_connections 2023-11-28T17:17:00.962 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.960+0000 7f8d97343640 1 -- 172.21.15.116:0/2896611728 wait complete. 2023-11-28T17:17:00.977 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:17:00.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.996+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1696838231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd1c0a0f60 con 0x7fcd1c0ce840 2023-11-28T17:17:00.999 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:00.998+0000 7fcd197fa640 1 -- 172.21.15.116:0/1696838231 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+17073 (secure 0 0 0) 0x7fcd080585e0 con 0x7fcd1c0ce840 2023-11-28T17:17:00.999 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:17:00.999 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":52,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:12:26.038218+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:11:21.518078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T17:11:22.672451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":0}]},{"pgid":"7.8","mappings":[{"from":5,"to":7}]},{"pgid":"7.1b","mappings":[{"from":5,"to":7}]},{"pgid":"7.30","mappings":[{"from":5,"to":7}]},{"pgid":"7.31","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6833/372407661":"2023-11-29T17:12:26.038071+0000","172.21.15.156:6832/372407661":"2023-11-29T17:12:26.038071+0000","172.21.15.116:6835/3251347229":"2023-11-29T17:12:06.030424+0000","172.21.15.116:6834/3251347229":"2023-11-29T17:12:06.030424+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:17:01.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.000+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1696838231 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcd0006b2a0 msgr2=0x7fcd0006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.000+0000 7fcd21a2e640 1 --2- 172.21.15.116:0/1696838231 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcd0006b2a0 0x7fcd0006d760 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fcd100011e0 tx=0x7fcd10009040 comp rx=0 tx=0).stop 2023-11-28T17:17:01.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.000+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1696838231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd1c0ce840 msgr2=0x7fcd1c1578c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.000+0000 7fcd21a2e640 1 --2- 172.21.15.116:0/1696838231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd1c0ce840 0x7fcd1c1578c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fcd0800d800 tx=0x7fcd0800dcc0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.000+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1696838231 shutdown_connections 2023-11-28T17:17:01.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.000+0000 7fcd21a2e640 1 --2- 172.21.15.116:0/1696838231 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fcd0006b2a0 0x7fcd0006d760 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.000+0000 7fcd21a2e640 1 --2- 172.21.15.116:0/1696838231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd1c15d4d0 0x7fcd1c0c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.002 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.000+0000 7fcd21a2e640 1 --2- 172.21.15.116:0/1696838231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd1c0ce840 0x7fcd1c1578c0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.000+0000 7fcd21a2e640 1 --2- 172.21.15.116:0/1696838231 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd1c0c71a0 0x7fcd1c150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.000+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1696838231 >> 172.21.15.116:0/1696838231 conn(0x7fcd1c0d2f90 msgr2=0x7fcd1c06be80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:01.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.001+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1696838231 shutdown_connections 2023-11-28T17:17:01.003 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.001+0000 7fcd21a2e640 1 -- 172.21.15.116:0/1696838231 wait complete. 2023-11-28T17:17:01.018 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:11:21.518078+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:11:22.672451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:17:01.018 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:17:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.162+0000 7f894e7cf640 1 Processor -- start 2023-11-28T17:17:01.164 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.162+0000 7f894e7cf640 1 -- start start 2023-11-28T17:17:01.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.163+0000 7f894e7cf640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8948160b80 0x7f89480d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.165 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894e7cf640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f89480b4010 0x7f89480d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894e7cf640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f89480cf6f0 0x7f89480cfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894e7cf640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8948160f60 con 0x7f89480cf6f0 2023-11-28T17:17:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894e7cf640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89480d0da0 con 0x7f89480b4010 2023-11-28T17:17:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894e7cf640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89480d1080 con 0x7f8948160b80 2023-11-28T17:17:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894dfce640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f89480cf6f0 0x7f89480cfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894dfce640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f89480cf6f0 0x7f89480cfb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53694/0 (socket says 172.21.15.116:53694) 2023-11-28T17:17:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894dfce640 1 -- 172.21.15.116:0/3596166701 learned_addr learned my addr 172.21.15.116:0/3596166701 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894cfcc640 1 --2- 172.21.15.116:0/3596166701 >> v2:172.21.15.156:3300/0 conn(0x7f89480b4010 0x7f89480d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894d7cd640 1 --2- 172.21.15.116:0/3596166701 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8948160b80 0x7f89480d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894dfce640 1 -- 172.21.15.116:0/3596166701 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8948160b80 msgr2=0x7f89480d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894dfce640 1 --2- 172.21.15.116:0/3596166701 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8948160b80 0x7f89480d1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894dfce640 1 -- 172.21.15.116:0/3596166701 >> v2:172.21.15.156:3300/0 conn(0x7f89480b4010 msgr2=0x7f89480d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:01.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894dfce640 1 --2- 172.21.15.116:0/3596166701 >> v2:172.21.15.156:3300/0 conn(0x7f89480b4010 0x7f89480d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.167 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.164+0000 7f894dfce640 1 -- 172.21.15.116:0/3596166701 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89480ce780 con 0x7f89480cf6f0 2023-11-28T17:17:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.165+0000 7f894dfce640 1 --2- 172.21.15.116:0/3596166701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89480cf6f0 0x7f89480cfb80 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f894808bb80 tx=0x7f894002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0536c5c289328a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.166+0000 7f893ffff640 1 -- 172.21.15.116:0/3596166701 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894002ca70 con 0x7f89480cf6f0 2023-11-28T17:17:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.166+0000 7f893ffff640 1 -- 172.21.15.116:0/3596166701 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f894002cbd0 con 0x7f89480cf6f0 2023-11-28T17:17:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.166+0000 7f893ffff640 1 -- 172.21.15.116:0/3596166701 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89400343f0 con 0x7f89480cf6f0 2023-11-28T17:17:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.166+0000 7f894e7cf640 1 -- 172.21.15.116:0/3596166701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89480cf6f0 msgr2=0x7f89480cfb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.166+0000 7f894e7cf640 1 --2- 172.21.15.116:0/3596166701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89480cf6f0 0x7f89480cfb80 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f894808bb80 tx=0x7f894002bed0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.166+0000 7f894e7cf640 1 -- 172.21.15.116:0/3596166701 shutdown_connections 2023-11-28T17:17:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.166+0000 7f894e7cf640 1 --2- 172.21.15.116:0/3596166701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89480cf6f0 0x7f89480cfb80 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.166+0000 7f894e7cf640 1 --2- 172.21.15.116:0/3596166701 >> v2:172.21.15.156:3300/0 conn(0x7f89480b4010 0x7f89480d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.166+0000 7f894e7cf640 1 --2- 172.21.15.116:0/3596166701 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8948160b80 0x7f89480d1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.166+0000 7f894e7cf640 1 -- 172.21.15.116:0/3596166701 >> 172.21.15.116:0/3596166701 conn(0x7f89480d2fc0 msgr2=0x7f89480d33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.166+0000 7f894e7cf640 1 -- 172.21.15.116:0/3596166701 shutdown_connections 2023-11-28T17:17:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.166+0000 7f894e7cf640 1 -- 172.21.15.116:0/3596166701 wait complete. 2023-11-28T17:17:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.168+0000 7f894e7cf640 1 Processor -- start 2023-11-28T17:17:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.168+0000 7f894e7cf640 1 -- start start 2023-11-28T17:17:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.169+0000 7f894e7cf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89480b4010 0x7f8948148ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.169+0000 7f894e7cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89480cf6f0 0x7f89480c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.169+0000 7f894e7cf640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8948160b80 0x7f89480c7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.169+0000 7f894e7cf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89480b88c0 con 0x7f89480cf6f0 2023-11-28T17:17:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.169+0000 7f894e7cf640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89480cb8f0 con 0x7f89480b4010 2023-11-28T17:17:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.169+0000 7f894e7cf640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89480cbbd0 con 0x7f8948160b80 2023-11-28T17:17:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.169+0000 7f894cfcc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89480cf6f0 0x7f89480c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.169+0000 7f894cfcc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89480cf6f0 0x7f89480c6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53698/0 (socket says 172.21.15.116:53698) 2023-11-28T17:17:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.169+0000 7f894cfcc640 1 -- 172.21.15.116:0/3241280299 learned_addr learned my addr 172.21.15.116:0/3241280299 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:01.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.170+0000 7f894cfcc640 1 -- 172.21.15.116:0/3241280299 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8948160b80 msgr2=0x7f89480c7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.170+0000 7f894dfce640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8948160b80 0x7f89480c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.170+0000 7f894d7cd640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89480b4010 0x7f8948148ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.170+0000 7f894cfcc640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8948160b80 0x7f89480c7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.173 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.170+0000 7f894cfcc640 1 -- 172.21.15.116:0/3241280299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89480b4010 msgr2=0x7f8948148ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.170+0000 7f894cfcc640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89480b4010 0x7f8948148ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.170+0000 7f894cfcc640 1 -- 172.21.15.116:0/3241280299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8940009b30 con 0x7f89480cf6f0 2023-11-28T17:17:01.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.170+0000 7f894d7cd640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89480b4010 0x7f8948148ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:17:01.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.170+0000 7f894cfcc640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89480cf6f0 0x7f89480c6c10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f893800d800 tx=0x7f893800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:01.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.170+0000 7f893dffb640 1 -- 172.21.15.116:0/3241280299 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89380096a0 con 0x7f89480cf6f0 2023-11-28T17:17:01.174 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.170+0000 7f893dffb640 1 -- 172.21.15.116:0/3241280299 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8938009800 con 0x7f89480cf6f0 2023-11-28T17:17:01.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.170+0000 7f894e7cf640 1 -- 172.21.15.116:0/3241280299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89480cbeb0 con 0x7f89480cf6f0 2023-11-28T17:17:01.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.170+0000 7f893dffb640 1 -- 172.21.15.116:0/3241280299 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8938009990 con 0x7f89480cf6f0 2023-11-28T17:17:01.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.170+0000 7f894e7cf640 1 -- 172.21.15.116:0/3241280299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89480b8ba0 con 0x7f89480cf6f0 2023-11-28T17:17:01.175 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.172+0000 7f894e7cf640 1 -- 172.21.15.116:0/3241280299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8948067980 con 0x7f89480cf6f0 2023-11-28T17:17:01.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.172+0000 7f893dffb640 1 -- 172.21.15.116:0/3241280299 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f8938009af0 con 0x7f89480cf6f0 2023-11-28T17:17:01.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.174+0000 7f893dffb640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f892c06b2a0 0x7f892c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.174+0000 7f893dffb640 1 -- 172.21.15.116:0/3241280299 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f893808a1c0 con 0x7f89480cf6f0 2023-11-28T17:17:01.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.174+0000 7f894d7cd640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f892c06b2a0 0x7f892c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.176 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.175+0000 7f894d7cd640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f892c06b2a0 0x7f892c06d760 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8944008890 tx=0x7f8944008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:01.179 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.178+0000 7f893dffb640 1 -- 172.21.15.116:0/3241280299 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8938054ef0 con 0x7f89480cf6f0 2023-11-28T17:17:01.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.243+0000 7fe898672640 1 Processor -- start 2023-11-28T17:17:01.245 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.243+0000 7fe898672640 1 -- start start 2023-11-28T17:17:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.245+0000 7fe898672640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe890160e30 0x7fe8900cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.245+0000 7fe898672640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe8900b30e0 0x7fe89006a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.245+0000 7fe898672640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8900cfa10 con 0x7fe8900cf0f0 2023-11-28T17:17:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.245+0000 7fe898672640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8900b4700 con 0x7fe8900b30e0 2023-11-28T17:17:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.245+0000 7fe898672640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8901406f0 con 0x7fe890160e30 2023-11-28T17:17:01.252 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.245+0000 7fe89640e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe8900cf0f0 0x7fe890143420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:32932/0 (socket says 172.21.15.116:32932) 2023-11-28T17:17:01.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.245+0000 7fe89640e640 1 -- 172.21.15.116:0/3182303219 learned_addr learned my addr 172.21.15.116:0/3182303219 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:17:01.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.245+0000 7fe89540c640 1 -- 172.21.15.116:0/3182303219 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638542019 0 0) 0x7fe8900cfa10 con 0x7fe8900cf0f0 2023-11-28T17:17:01.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.245+0000 7fe895c0d640 1 --2- 172.21.15.116:0/3182303219 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe890160e30 0x7fe8900cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.245+0000 7fe89540c640 1 -- 172.21.15.116:0/3182303219 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe87c003520 con 0x7fe8900cf0f0 2023-11-28T17:17:01.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe896c0f640 1 --2- 172.21.15.116:0/3182303219 >> v2:172.21.15.156:3300/0 conn(0x7fe8900b30e0 0x7fe89006a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe89540c640 1 -- 172.21.15.116:0/3182303219 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1806736777 0 0) 0x7fe87c003520 con 0x7fe8900cf0f0 2023-11-28T17:17:01.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe89540c640 1 -- 172.21.15.116:0/3182303219 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe890160e30 msgr2=0x7fe8900cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:01.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe89540c640 1 --2- 172.21.15.116:0/3182303219 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe890160e30 0x7fe8900cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe89540c640 1 -- 172.21.15.116:0/3182303219 >> v2:172.21.15.156:3300/0 conn(0x7fe8900b30e0 msgr2=0x7fe89006a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:01.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe89540c640 1 --2- 172.21.15.116:0/3182303219 >> v2:172.21.15.156:3300/0 conn(0x7fe8900b30e0 0x7fe89006a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe89540c640 1 -- 172.21.15.116:0/3182303219 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe890161360 con 0x7fe8900cf0f0 2023-11-28T17:17:01.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe89540c640 1 -- 172.21.15.116:0/3182303219 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fe88c001670 con 0x7fe8900cf0f0 2023-11-28T17:17:01.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe89540c640 1 -- 172.21.15.116:0/3182303219 >> v1:172.21.15.116:6789/0 conn(0x7fe8900cf0f0 legacy=0x7fe890143420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe89540c640 1 --2- 172.21.15.116:0/3182303219 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe890160e30 0x7fe87c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe89540c640 1 --2- 172.21.15.116:0/3182303219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe87c003ff0 0x7fe87c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe89540c640 1 --2- 172.21.15.116:0/3182303219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c004d00 0x7fe87c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.255 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe89540c640 1 -- 172.21.15.116:0/3182303219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900b4700 con 0x7fe87c003ff0 2023-11-28T17:17:01.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe89540c640 1 -- 172.21.15.116:0/3182303219 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8901406f0 con 0x7fe87c004d00 2023-11-28T17:17:01.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe89540c640 1 -- 172.21.15.116:0/3182303219 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c00bae0 con 0x7fe890160e30 2023-11-28T17:17:01.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.246+0000 7fe895c0d640 1 --2- 172.21.15.116:0/3182303219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe87c003ff0 0x7fe87c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.247+0000 7fe895c0d640 1 -- 172.21.15.116:0/3182303219 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe890160e30 msgr2=0x7fe87c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:01.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.247+0000 7fe895c0d640 1 --2- 172.21.15.116:0/3182303219 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe890160e30 0x7fe87c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.256 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.247+0000 7fe895c0d640 1 -- 172.21.15.116:0/3182303219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c004d00 msgr2=0x7fe87c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:01.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.247+0000 7fe895c0d640 1 --2- 172.21.15.116:0/3182303219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c004d00 0x7fe87c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.247+0000 7fe895c0d640 1 -- 172.21.15.116:0/3182303219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c003520 con 0x7fe87c003ff0 2023-11-28T17:17:01.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.248+0000 7fe895c0d640 1 --2- 172.21.15.116:0/3182303219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe87c003ff0 0x7fe87c0047c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fe88000ba10 tx=0x7fe87c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fd6d44aa2c4d91 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:01.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.248+0000 7fe89540c640 1 -- 172.21.15.116:0/3182303219 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880032680 con 0x7fe87c003ff0 2023-11-28T17:17:01.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.248+0000 7fe89540c640 1 -- 172.21.15.116:0/3182303219 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8800327e0 con 0x7fe87c003ff0 2023-11-28T17:17:01.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.248+0000 7fe89540c640 1 -- 172.21.15.116:0/3182303219 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880032b70 con 0x7fe87c003ff0 2023-11-28T17:17:01.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.248+0000 7fe898672640 1 -- 172.21.15.116:0/3182303219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe87c003ff0 msgr2=0x7fe87c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.248+0000 7fe898672640 1 --2- 172.21.15.116:0/3182303219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe87c003ff0 0x7fe87c0047c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fe88000ba10 tx=0x7fe87c0057f0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.248+0000 7fe896c0f640 1 -- 172.21.15.116:0/3182303219 reap_dead start 2023-11-28T17:17:01.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.249+0000 7fe898672640 1 -- 172.21.15.116:0/3182303219 shutdown_connections 2023-11-28T17:17:01.258 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.249+0000 7fe898672640 1 -- 172.21.15.116:0/3182303219 >> 172.21.15.116:0/3182303219 conn(0x7fe8900b1de0 msgr2=0x7fe8900b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:01.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.249+0000 7fe898672640 1 -- 172.21.15.116:0/3182303219 shutdown_connections 2023-11-28T17:17:01.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.249+0000 7fe898672640 1 -- 172.21.15.116:0/3182303219 wait complete. 2023-11-28T17:17:01.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.250+0000 7fe898672640 1 Processor -- start 2023-11-28T17:17:01.259 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.250+0000 7fe898672640 1 -- start start 2023-11-28T17:17:01.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.251+0000 7fe898672640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c003ff0 0x7fe89015bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.251+0000 7fe898672640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900af450 0x7fe890145730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.251+0000 7fe898672640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8900ae5e0 0x7fe8900aea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.265 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.251+0000 7fe898672640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900aefa0 con 0x7fe8900af450 2023-11-28T17:17:01.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.251+0000 7fe898672640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89013dcd0 con 0x7fe87c003ff0 2023-11-28T17:17:01.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.251+0000 7fe898672640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe890145c70 con 0x7fe8900ae5e0 2023-11-28T17:17:01.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.252+0000 7fe895c0d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900af450 0x7fe890145730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.252+0000 7fe895c0d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900af450 0x7fe890145730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53710/0 (socket says 172.21.15.116:53710) 2023-11-28T17:17:01.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.252+0000 7fe895c0d640 1 -- 172.21.15.116:0/2345836190 learned_addr learned my addr 172.21.15.116:0/2345836190 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:01.266 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.252+0000 7fe89640e640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c003ff0 0x7fe89015bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.252+0000 7fe896c0f640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8900ae5e0 0x7fe8900aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.252+0000 7fe895c0d640 1 -- 172.21.15.116:0/2345836190 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8900ae5e0 msgr2=0x7fe8900aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.252+0000 7fe895c0d640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8900ae5e0 0x7fe8900aea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.252+0000 7fe895c0d640 1 -- 172.21.15.116:0/2345836190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c003ff0 msgr2=0x7fe89015bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.252+0000 7fe895c0d640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c003ff0 0x7fe89015bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.252+0000 7fe895c0d640 1 -- 172.21.15.116:0/2345836190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89015e9d0 con 0x7fe8900af450 2023-11-28T17:17:01.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.252+0000 7fe89640e640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c003ff0 0x7fe89015bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:17:01.267 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.253+0000 7fe895c0d640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900af450 0x7fe890145730 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fe880000c00 tx=0x7fe880004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:01.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.253+0000 7fe886ffd640 1 -- 172.21.15.116:0/2345836190 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8800309e0 con 0x7fe8900af450 2023-11-28T17:17:01.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.253+0000 7fe886ffd640 1 -- 172.21.15.116:0/2345836190 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe880030b60 con 0x7fe8900af450 2023-11-28T17:17:01.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.253+0000 7fe898672640 1 -- 172.21.15.116:0/2345836190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89015dc30 con 0x7fe8900af450 2023-11-28T17:17:01.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.253+0000 7fe886ffd640 1 -- 172.21.15.116:0/2345836190 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88003b420 con 0x7fe8900af450 2023-11-28T17:17:01.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.253+0000 7fe898672640 1 -- 172.21.15.116:0/2345836190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89015e1d0 con 0x7fe8900af450 2023-11-28T17:17:01.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.254+0000 7fe898672640 1 -- 172.21.15.116:0/2345836190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe890069360 con 0x7fe8900af450 2023-11-28T17:17:01.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.255+0000 7fe886ffd640 1 -- 172.21.15.116:0/2345836190 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fe88003b580 con 0x7fe8900af450 2023-11-28T17:17:01.268 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.256+0000 7fe886ffd640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe87406b2a0 0x7fe87406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.256+0000 7fe886ffd640 1 -- 172.21.15.116:0/2345836190 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fe8800ac670 con 0x7fe8900af450 2023-11-28T17:17:01.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.257+0000 7fe89640e640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe87406b2a0 0x7fe87406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.257+0000 7fe89640e640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe87406b2a0 0x7fe87406d760 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe88c005af0 tx=0x7fe88c001780 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:01.269 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.260+0000 7fe886ffd640 1 -- 172.21.15.116:0/2345836190 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe880077420 con 0x7fe8900af450 2023-11-28T17:17:01.375 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.373+0000 7f894e7cf640 1 -- 172.21.15.116:0/3241280299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f89480cb750 con 0x7f89480cf6f0 2023-11-28T17:17:01.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.374+0000 7f893dffb640 1 -- 172.21.15.116:0/3241280299 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) v1 ==== 104+0+11 (secure 0 0 0) 0x7f89480cb750 con 0x7f89480cf6f0 2023-11-28T17:17:01.376 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:17:01.378 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.377+0000 7f894e7cf640 1 -- 172.21.15.116:0/3241280299 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f892c06b2a0 msgr2=0x7f892c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.377+0000 7f894e7cf640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f892c06b2a0 0x7f892c06d760 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8944008890 tx=0x7f8944008040 comp rx=0 tx=0).stop 2023-11-28T17:17:01.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.377+0000 7f894e7cf640 1 -- 172.21.15.116:0/3241280299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89480cf6f0 msgr2=0x7f89480c6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.377+0000 7f894e7cf640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89480cf6f0 0x7f89480c6c10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f893800d800 tx=0x7f893800dcc0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.381 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.377+0000 7f894e7cf640 1 -- 172.21.15.116:0/3241280299 shutdown_connections 2023-11-28T17:17:01.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.377+0000 7f894e7cf640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f892c06b2a0 0x7f892c06d760 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.377+0000 7f894e7cf640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8948160b80 0x7f89480c7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.377+0000 7f894e7cf640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89480cf6f0 0x7f89480c6c10 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.377+0000 7f894e7cf640 1 --2- 172.21.15.116:0/3241280299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89480b4010 0x7f8948148ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.377+0000 7f894e7cf640 1 -- 172.21.15.116:0/3241280299 >> 172.21.15.116:0/3241280299 conn(0x7f89480d2fc0 msgr2=0x7f894815c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:01.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.377+0000 7f894e7cf640 1 -- 172.21.15.116:0/3241280299 shutdown_connections 2023-11-28T17:17:01.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.377+0000 7f894e7cf640 1 -- 172.21.15.116:0/3241280299 wait complete. 2023-11-28T17:17:01.395 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:17:01.443 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.442+0000 7fe898672640 1 -- 172.21.15.116:0/2345836190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe89013def0 con 0x7fe8900af450 2023-11-28T17:17:01.444 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.442+0000 7fe886ffd640 1 -- 172.21.15.116:0/2345836190 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe88007aa40 con 0x7fe8900af450 2023-11-28T17:17:01.444 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:17:01.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.445+0000 7fe898672640 1 -- 172.21.15.116:0/2345836190 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe87406b2a0 msgr2=0x7fe87406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.446 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.445+0000 7fe898672640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe87406b2a0 0x7fe87406d760 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe88c005af0 tx=0x7fe88c001780 comp rx=0 tx=0).stop 2023-11-28T17:17:01.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.445+0000 7fe898672640 1 -- 172.21.15.116:0/2345836190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900af450 msgr2=0x7fe890145730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.445+0000 7fe898672640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900af450 0x7fe890145730 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fe880000c00 tx=0x7fe880004730 comp rx=0 tx=0).stop 2023-11-28T17:17:01.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.445+0000 7fe898672640 1 -- 172.21.15.116:0/2345836190 shutdown_connections 2023-11-28T17:17:01.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.445+0000 7fe898672640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fe87406b2a0 0x7fe87406d760 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.445+0000 7fe898672640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8900ae5e0 0x7fe8900aea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.445+0000 7fe898672640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900af450 0x7fe890145730 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.445+0000 7fe898672640 1 --2- 172.21.15.116:0/2345836190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c003ff0 0x7fe89015bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.445+0000 7fe898672640 1 -- 172.21.15.116:0/2345836190 >> 172.21.15.116:0/2345836190 conn(0x7fe8900b1de0 msgr2=0x7fe8900b3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:01.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.445+0000 7fe898672640 1 -- 172.21.15.116:0/2345836190 shutdown_connections 2023-11-28T17:17:01.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.445+0000 7fe898672640 1 -- 172.21.15.116:0/2345836190 wait complete. 2023-11-28T17:17:01.462 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:17:01.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.591+0000 7f48853fb640 1 Processor -- start 2023-11-28T17:17:01.593 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.592+0000 7f48853fb640 1 -- start start 2023-11-28T17:17:01.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.593+0000 7f48853fb640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4880160b80 0x7f48800d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.593+0000 7f48853fb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f48800d0700 0x7f48800c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.593+0000 7f48853fb640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48800d0da0 con 0x7f48800b4010 2023-11-28T17:17:01.594 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.593+0000 7f48853fb640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4880160f60 con 0x7f4880160b80 2023-11-28T17:17:01.595 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.593+0000 7f48853fb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48800d1080 con 0x7f48800d0700 2023-11-28T17:17:01.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.593+0000 7f487ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4880160b80 0x7f48800d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.593+0000 7f487ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4880160b80 0x7f48800d1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53720/0 (socket says 172.21.15.116:53720) 2023-11-28T17:17:01.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.593+0000 7f487ffff640 1 -- 172.21.15.116:0/1574582430 learned_addr learned my addr 172.21.15.116:0/1574582430 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:01.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.593+0000 7f487ffff640 1 -- 172.21.15.116:0/1574582430 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f48800d0700 msgr2=0x7f48800c6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:01.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.593+0000 7f487ffff640 1 --2- 172.21.15.116:0/1574582430 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f48800d0700 0x7f48800c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.593+0000 7f487ffff640 1 -- 172.21.15.116:0/1574582430 >> v1:172.21.15.156:6789/0 conn(0x7f48800b4010 legacy=0x7f48800d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:01.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.593+0000 7f487ffff640 1 -- 172.21.15.116:0/1574582430 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48800ce780 con 0x7f4880160b80 2023-11-28T17:17:01.596 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.594+0000 7f487ffff640 1 --2- 172.21.15.116:0/1574582430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4880160b80 0x7f48800d1360 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4870009870 tx=0x7f487002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abd0fe419c9f9a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.595+0000 7f487effd640 1 -- 172.21.15.116:0/1574582430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4870031870 con 0x7f4880160b80 2023-11-28T17:17:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.595+0000 7f487effd640 1 -- 172.21.15.116:0/1574582430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48700319d0 con 0x7f4880160b80 2023-11-28T17:17:01.597 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.595+0000 7f487effd640 1 -- 172.21.15.116:0/1574582430 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4870031d60 con 0x7f4880160b80 2023-11-28T17:17:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.595+0000 7f48853fb640 1 -- 172.21.15.116:0/1574582430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4880160b80 msgr2=0x7f48800d1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.595+0000 7f48853fb640 1 --2- 172.21.15.116:0/1574582430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4880160b80 0x7f48800d1360 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4870009870 tx=0x7f487002b910 comp rx=0 tx=0).stop 2023-11-28T17:17:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.595+0000 7f48853fb640 1 -- 172.21.15.116:0/1574582430 shutdown_connections 2023-11-28T17:17:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.595+0000 7f48853fb640 1 --2- 172.21.15.116:0/1574582430 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f48800d0700 0x7f48800c6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.595+0000 7f48853fb640 1 --2- 172.21.15.116:0/1574582430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4880160b80 0x7f48800d1360 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.595+0000 7f48853fb640 1 -- 172.21.15.116:0/1574582430 >> 172.21.15.116:0/1574582430 conn(0x7f48800d2fc0 msgr2=0x7f48800d33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.596+0000 7f48853fb640 1 -- 172.21.15.116:0/1574582430 shutdown_connections 2023-11-28T17:17:01.598 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.596+0000 7f48853fb640 1 -- 172.21.15.116:0/1574582430 wait complete. 2023-11-28T17:17:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.597+0000 7f48853fb640 1 Processor -- start 2023-11-28T17:17:01.599 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.597+0000 7f48853fb640 1 -- start start 2023-11-28T17:17:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.598+0000 7f48853fb640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48800b4010 0x7f488015bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.601 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.598+0000 7f48853fb640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4880160b80 0x7f4880153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.598+0000 7f48853fb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4880153570 0x7f48801f3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.598+0000 7f48853fb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48801f4ae0 con 0x7f4880153570 2023-11-28T17:17:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.598+0000 7f48853fb640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48800ccf60 con 0x7f48800b4010 2023-11-28T17:17:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.598+0000 7f48853fb640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4880153a20 con 0x7f4880160b80 2023-11-28T17:17:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.599+0000 7f4884bfa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4880153570 0x7f48801f3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.599+0000 7f4884bfa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4880153570 0x7f48801f3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53736/0 (socket says 172.21.15.116:53736) 2023-11-28T17:17:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.599+0000 7f4884bfa640 1 -- 172.21.15.116:0/1825573713 learned_addr learned my addr 172.21.15.116:0/1825573713 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.599+0000 7f487ffff640 1 --2- 172.21.15.116:0/1825573713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48800b4010 0x7f488015bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.599+0000 7f4884bfa640 1 -- 172.21.15.116:0/1825573713 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4880160b80 msgr2=0x7f4880153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.602 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.599+0000 7f487f7fe640 1 --2- 172.21.15.116:0/1825573713 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4880160b80 0x7f4880153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.599+0000 7f4884bfa640 1 --2- 172.21.15.116:0/1825573713 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4880160b80 0x7f4880153030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.599+0000 7f4884bfa640 1 -- 172.21.15.116:0/1825573713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48800b4010 msgr2=0x7f488015bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.599+0000 7f4884bfa640 1 --2- 172.21.15.116:0/1825573713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48800b4010 0x7f488015bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.599+0000 7f4884bfa640 1 -- 172.21.15.116:0/1825573713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48700094d0 con 0x7f4880153570 2023-11-28T17:17:01.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.599+0000 7f4884bfa640 1 --2- 172.21.15.116:0/1825573713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4880153570 0x7f48801f3100 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f487800b9e0 tx=0x7f487800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:01.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.600+0000 7f487cff9640 1 -- 172.21.15.116:0/1825573713 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487800cb20 con 0x7f4880153570 2023-11-28T17:17:01.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.600+0000 7f487cff9640 1 -- 172.21.15.116:0/1825573713 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487800cca0 con 0x7f4880153570 2023-11-28T17:17:01.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.600+0000 7f48853fb640 1 -- 172.21.15.116:0/1825573713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48801f3e40 con 0x7f4880153570 2023-11-28T17:17:01.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.600+0000 7f487cff9640 1 -- 172.21.15.116:0/1825573713 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4878014680 con 0x7f4880153570 2023-11-28T17:17:01.603 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.600+0000 7f48853fb640 1 -- 172.21.15.116:0/1825573713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48801f4700 con 0x7f4880153570 2023-11-28T17:17:01.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.601+0000 7f48853fb640 1 -- 172.21.15.116:0/1825573713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4880067980 con 0x7f4880153570 2023-11-28T17:17:01.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.602+0000 7f487cff9640 1 -- 172.21.15.116:0/1825573713 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f487801d3b0 con 0x7f4880153570 2023-11-28T17:17:01.604 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.603+0000 7f487cff9640 1 --2- 172.21.15.116:0/1825573713 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f486c06b2a0 0x7f486c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.603+0000 7f487cff9640 1 -- 172.21.15.116:0/1825573713 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f4878089c10 con 0x7f4880153570 2023-11-28T17:17:01.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.603+0000 7f487ffff640 1 --2- 172.21.15.116:0/1825573713 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f486c06b2a0 0x7f486c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.604+0000 7f487ffff640 1 --2- 172.21.15.116:0/1825573713 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f486c06b2a0 0x7f486c06d760 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4870036000 tx=0x7f487003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:01.610 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.607+0000 7f487cff9640 1 -- 172.21.15.116:0/1825573713 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4878054930 con 0x7f4880153570 2023-11-28T17:17:01.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.645+0000 7f0fd87ec640 1 Processor -- start 2023-11-28T17:17:01.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.645+0000 7f0fd87ec640 1 -- start start 2023-11-28T17:17:01.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.646+0000 7f0fd87ec640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0fd00cd420 0x7f0fd0140f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.646+0000 7f0fd87ec640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0fd0141450 0x7f0fd00cda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.646+0000 7f0fd87ec640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd00d0d80 con 0x7f0fd00cd040 2023-11-28T17:17:01.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.646+0000 7f0fd87ec640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd015f3e0 con 0x7f0fd00cd420 2023-11-28T17:17:01.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.646+0000 7f0fd87ec640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd015f6c0 con 0x7f0fd0141450 2023-11-28T17:17:01.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.646+0000 7f0fd5d87640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0fd00cd420 0x7f0fd0140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.646+0000 7f0fd5d87640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0fd00cd420 0x7f0fd0140f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53748/0 (socket says 172.21.15.116:53748) 2023-11-28T17:17:01.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.646+0000 7f0fd5d87640 1 -- 172.21.15.116:0/2677247893 learned_addr learned my addr 172.21.15.116:0/2677247893 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:01.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.647+0000 7f0fd5d87640 1 -- 172.21.15.116:0/2677247893 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0fd0141450 msgr2=0x7f0fd00cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:01.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.647+0000 7f0fd5d87640 1 --2- 172.21.15.116:0/2677247893 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0fd0141450 0x7f0fd00cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.647+0000 7f0fd5d87640 1 -- 172.21.15.116:0/2677247893 >> v1:172.21.15.156:6789/0 conn(0x7f0fd00cd040 legacy=0x7f0fd015ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:01.649 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.647+0000 7f0fd5d87640 1 -- 172.21.15.116:0/2677247893 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd015f9a0 con 0x7f0fd00cd420 2023-11-28T17:17:01.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.648+0000 7f0fd5d87640 1 --2- 172.21.15.116:0/2677247893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fd00cd420 0x7f0fd0140f10 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00d1d30 tx=0x7f0fc002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=316b3cf6a93278be server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:01.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.648+0000 7f0fd5586640 1 -- 172.21.15.116:0/2677247893 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0031640 con 0x7f0fd00cd420 2023-11-28T17:17:01.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.648+0000 7f0fd5586640 1 -- 172.21.15.116:0/2677247893 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc00317a0 con 0x7f0fd00cd420 2023-11-28T17:17:01.650 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.648+0000 7f0fd5586640 1 -- 172.21.15.116:0/2677247893 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0031b30 con 0x7f0fd00cd420 2023-11-28T17:17:01.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.648+0000 7f0fd87ec640 1 -- 172.21.15.116:0/2677247893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fd00cd420 msgr2=0x7f0fd0140f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.648+0000 7f0fd87ec640 1 --2- 172.21.15.116:0/2677247893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fd00cd420 0x7f0fd0140f10 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0fd00d1d30 tx=0x7f0fc002b8f0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.649+0000 7f0fd87ec640 1 -- 172.21.15.116:0/2677247893 shutdown_connections 2023-11-28T17:17:01.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.649+0000 7f0fd87ec640 1 --2- 172.21.15.116:0/2677247893 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0fd0141450 0x7f0fd00cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.649+0000 7f0fd87ec640 1 --2- 172.21.15.116:0/2677247893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fd00cd420 0x7f0fd0140f10 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.649+0000 7f0fd87ec640 1 -- 172.21.15.116:0/2677247893 >> 172.21.15.116:0/2677247893 conn(0x7f0fd0070af0 msgr2=0x7f0fd00662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:01.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.649+0000 7f0fd87ec640 1 -- 172.21.15.116:0/2677247893 shutdown_connections 2023-11-28T17:17:01.651 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.649+0000 7f0fd87ec640 1 -- 172.21.15.116:0/2677247893 wait complete. 2023-11-28T17:17:01.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.650+0000 7f0fd87ec640 1 Processor -- start 2023-11-28T17:17:01.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.650+0000 7f0fd87ec640 1 -- start start 2023-11-28T17:17:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.651+0000 7f0fd87ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fd00cd040 0x7f0fd013cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.651+0000 7f0fd87ec640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fd00cd420 0x7f0fd01439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.651+0000 7f0fd87ec640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0fd01f0f20 0x7f0fd00b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.651+0000 7f0fd87ec640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd00c5fd0 con 0x7f0fd00cd040 2023-11-28T17:17:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.651+0000 7f0fd87ec640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd013d470 con 0x7f0fd00cd420 2023-11-28T17:17:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.651+0000 7f0fd87ec640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd01444a0 con 0x7f0fd01f0f20 2023-11-28T17:17:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.652+0000 7f0fd6588640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fd00cd040 0x7f0fd013cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.652+0000 7f0fd6588640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fd00cd040 0x7f0fd013cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53754/0 (socket says 172.21.15.116:53754) 2023-11-28T17:17:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.652+0000 7f0fd6588640 1 -- 172.21.15.116:0/446170519 learned_addr learned my addr 172.21.15.116:0/446170519 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:01.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.652+0000 7f0fd5d87640 1 --2- 172.21.15.116:0/446170519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fd00cd420 0x7f0fd01439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.652+0000 7f0fd6588640 1 -- 172.21.15.116:0/446170519 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0fd01f0f20 msgr2=0x7f0fd00b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.652+0000 7f0fd6588640 1 --2- 172.21.15.116:0/446170519 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0fd01f0f20 0x7f0fd00b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.652+0000 7f0fd6588640 1 -- 172.21.15.116:0/446170519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fd00cd420 msgr2=0x7f0fd01439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.652+0000 7f0fd6588640 1 --2- 172.21.15.116:0/446170519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fd00cd420 0x7f0fd01439b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.652+0000 7f0fd6588640 1 -- 172.21.15.116:0/446170519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc00094d0 con 0x7f0fd00cd040 2023-11-28T17:17:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.652+0000 7f0fd6588640 1 --2- 172.21.15.116:0/446170519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fd00cd040 0x7f0fd013cf30 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00b3f0 tx=0x7f0fcc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.653+0000 7f0fc6ffd640 1 -- 172.21.15.116:0/446170519 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc011830 con 0x7f0fd00cd040 2023-11-28T17:17:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.653+0000 7f0fc6ffd640 1 -- 172.21.15.116:0/446170519 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fcc011990 con 0x7f0fd00cd040 2023-11-28T17:17:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.653+0000 7f0fc6ffd640 1 -- 172.21.15.116:0/446170519 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc011d40 con 0x7f0fd00cd040 2023-11-28T17:17:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.653+0000 7f0fd87ec640 1 -- 172.21.15.116:0/446170519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd00c6220 con 0x7f0fd00cd040 2023-11-28T17:17:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.653+0000 7f0fd87ec640 1 -- 172.21.15.116:0/446170519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd00c67e0 con 0x7f0fd00cd040 2023-11-28T17:17:01.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.654+0000 7f0fd87ec640 1 -- 172.21.15.116:0/446170519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd0069360 con 0x7f0fd00cd040 2023-11-28T17:17:01.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.655+0000 7f0fc6ffd640 1 -- 172.21.15.116:0/446170519 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f0fcc01a400 con 0x7f0fd00cd040 2023-11-28T17:17:01.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.656+0000 7f0fc6ffd640 1 --2- 172.21.15.116:0/446170519 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0fb406b2a0 0x7f0fb406d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:01.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.656+0000 7f0fc6ffd640 1 -- 172.21.15.116:0/446170519 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f0fcc089bd0 con 0x7f0fd00cd040 2023-11-28T17:17:01.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.656+0000 7f0fd5d87640 1 --2- 172.21.15.116:0/446170519 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0fb406b2a0 0x7f0fb406d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:01.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.656+0000 7f0fd5d87640 1 --2- 172.21.15.116:0/446170519 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0fb406b2a0 0x7f0fb406d760 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0036000 tx=0x7f0fc003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:01.666 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.664+0000 7f0fc6ffd640 1 -- 172.21.15.116:0/446170519 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0fcc0548f0 con 0x7f0fd00cd040 2023-11-28T17:17:01.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.804+0000 7f48853fb640 1 -- 172.21.15.116:0/1825573713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f48800c7150 con 0x7f4880153570 2023-11-28T17:17:01.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.805+0000 7f487cff9640 1 -- 172.21.15.116:0/1825573713 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+11 (secure 0 0 0) 0x7f48800c7150 con 0x7f4880153570 2023-11-28T17:17:01.807 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:17:01.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.808+0000 7f48853fb640 1 -- 172.21.15.116:0/1825573713 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f486c06b2a0 msgr2=0x7f486c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.808+0000 7f48853fb640 1 --2- 172.21.15.116:0/1825573713 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f486c06b2a0 0x7f486c06d760 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4870036000 tx=0x7f487003c040 comp rx=0 tx=0).stop 2023-11-28T17:17:01.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.808+0000 7f48853fb640 1 -- 172.21.15.116:0/1825573713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4880153570 msgr2=0x7f48801f3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.808+0000 7f48853fb640 1 --2- 172.21.15.116:0/1825573713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4880153570 0x7f48801f3100 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f487800b9e0 tx=0x7f487800bea0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.808+0000 7f48853fb640 1 -- 172.21.15.116:0/1825573713 shutdown_connections 2023-11-28T17:17:01.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.808+0000 7f48853fb640 1 --2- 172.21.15.116:0/1825573713 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f486c06b2a0 0x7f486c06d760 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.808+0000 7f48853fb640 1 --2- 172.21.15.116:0/1825573713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4880153570 0x7f48801f3100 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.808+0000 7f48853fb640 1 --2- 172.21.15.116:0/1825573713 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4880160b80 0x7f4880153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.808+0000 7f48853fb640 1 --2- 172.21.15.116:0/1825573713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48800b4010 0x7f488015bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.808+0000 7f48853fb640 1 -- 172.21.15.116:0/1825573713 >> 172.21.15.116:0/1825573713 conn(0x7f48800d2fc0 msgr2=0x7f488015fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:01.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.808+0000 7f48853fb640 1 -- 172.21.15.116:0/1825573713 shutdown_connections 2023-11-28T17:17:01.811 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.808+0000 7f48853fb640 1 -- 172.21.15.116:0/1825573713 wait complete. 2023-11-28T17:17:01.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:17:01.848 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.847+0000 7f0fd87ec640 1 -- 172.21.15.116:0/446170519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0fd00b3970 con 0x7f0fd00cd040 2023-11-28T17:17:01.849 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.847+0000 7f0fc6ffd640 1 -- 172.21.15.116:0/446170519 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0fcc057f10 con 0x7f0fd00cd040 2023-11-28T17:17:01.849 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:17:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.850+0000 7f0fd87ec640 1 -- 172.21.15.116:0/446170519 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0fb406b2a0 msgr2=0x7f0fb406d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.850+0000 7f0fd87ec640 1 --2- 172.21.15.116:0/446170519 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0fb406b2a0 0x7f0fb406d760 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0036000 tx=0x7f0fc003c040 comp rx=0 tx=0).stop 2023-11-28T17:17:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.850+0000 7f0fd87ec640 1 -- 172.21.15.116:0/446170519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fd00cd040 msgr2=0x7f0fd013cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.850+0000 7f0fd87ec640 1 --2- 172.21.15.116:0/446170519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fd00cd040 0x7f0fd013cf30 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc00b3f0 tx=0x7f0fcc00b8b0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.850+0000 7f0fd87ec640 1 -- 172.21.15.116:0/446170519 shutdown_connections 2023-11-28T17:17:01.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.850+0000 7f0fd87ec640 1 --2- 172.21.15.116:0/446170519 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f0fb406b2a0 0x7f0fb406d760 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.850+0000 7f0fd87ec640 1 --2- 172.21.15.116:0/446170519 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0fd01f0f20 0x7f0fd00b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.850+0000 7f0fd87ec640 1 --2- 172.21.15.116:0/446170519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fd00cd420 0x7f0fd01439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.850+0000 7f0fd87ec640 1 --2- 172.21.15.116:0/446170519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fd00cd040 0x7f0fd013cf30 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:01.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.850+0000 7f0fd87ec640 1 -- 172.21.15.116:0/446170519 >> 172.21.15.116:0/446170519 conn(0x7f0fd0070af0 msgr2=0x7f0fd00662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:01.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.850+0000 7f0fd87ec640 1 -- 172.21.15.116:0/446170519 shutdown_connections 2023-11-28T17:17:01.852 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:01.850+0000 7f0fd87ec640 1 -- 172.21.15.116:0/446170519 wait complete. 2023-11-28T17:17:01.868 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:17:02.013 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.012+0000 7f007dd88640 1 Processor -- start 2023-11-28T17:17:02.014 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.012+0000 7f007dd88640 1 -- start start 2023-11-28T17:17:02.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.013+0000 7f007dd88640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f00780b1d90 0x7f00780cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.014+0000 7f007dd88640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00780cf040 con 0x7f0078160680 2023-11-28T17:17:02.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.014+0000 7f007dd88640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00780d3a10 con 0x7f00780d14d0 2023-11-28T17:17:02.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.014+0000 7f007dd88640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00780d3cd0 con 0x7f00780b1d90 2023-11-28T17:17:02.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.014+0000 7f007cd86640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0078160680 0x7f00780ce930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:32944/0 (socket says 172.21.15.116:32944) 2023-11-28T17:17:02.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.014+0000 7f007cd86640 1 -- 172.21.15.116:0/2422217248 learned_addr learned my addr 172.21.15.116:0/2422217248 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:17:02.015 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.014+0000 7f0077fff640 1 --2- 172.21.15.116:0/2422217248 >> v2:172.21.15.116:3300/0 conn(0x7f00780b1d90 0x7f00780cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.014+0000 7f00777fe640 1 -- 172.21.15.116:0/2422217248 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263636358 0 0) 0x7f00780cf040 con 0x7f0078160680 2023-11-28T17:17:02.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.014+0000 7f00777fe640 1 -- 172.21.15.116:0/2422217248 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0068003520 con 0x7f0078160680 2023-11-28T17:17:02.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.014+0000 7f0077fff640 1 -- 172.21.15.116:0/2422217248 >> v1:172.21.15.156:6789/0 conn(0x7f00780d14d0 legacy=0x7f00780afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:02.016 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.014+0000 7f0077fff640 1 -- 172.21.15.116:0/2422217248 >> v1:172.21.15.116:6789/0 conn(0x7f0078160680 legacy=0x7f00780ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.017 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.014+0000 7f0077fff640 1 -- 172.21.15.116:0/2422217248 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00780cdb70 con 0x7f00780b1d90 2023-11-28T17:17:02.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.015+0000 7f0077fff640 1 --2- 172.21.15.116:0/2422217248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00780b1d90 0x7f00780cff30 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f00640097a0 tx=0x7f006402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3cab1b9f18cf780 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:02.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.016+0000 7f00777fe640 1 -- 172.21.15.116:0/2422217248 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064031830 con 0x7f00780b1d90 2023-11-28T17:17:02.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.016+0000 7f00777fe640 1 -- 172.21.15.116:0/2422217248 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0064031990 con 0x7f00780b1d90 2023-11-28T17:17:02.018 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.016+0000 7f00777fe640 1 -- 172.21.15.116:0/2422217248 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064031d20 con 0x7f00780b1d90 2023-11-28T17:17:02.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.016+0000 7f007dd88640 1 -- 172.21.15.116:0/2422217248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00780b1d90 msgr2=0x7f00780cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.016+0000 7f007dd88640 1 --2- 172.21.15.116:0/2422217248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00780b1d90 0x7f00780cff30 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f00640097a0 tx=0x7f006402b8d0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.016+0000 7f007dd88640 1 -- 172.21.15.116:0/2422217248 shutdown_connections 2023-11-28T17:17:02.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.016+0000 7f007dd88640 1 --2- 172.21.15.116:0/2422217248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00780b1d90 0x7f00780cff30 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.019 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.016+0000 7f007dd88640 1 -- 172.21.15.116:0/2422217248 >> 172.21.15.116:0/2422217248 conn(0x7f00780d2ef0 msgr2=0x7f00780d3300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:02.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.016+0000 7f007dd88640 1 -- 172.21.15.116:0/2422217248 shutdown_connections 2023-11-28T17:17:02.020 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.016+0000 7f007dd88640 1 -- 172.21.15.116:0/2422217248 wait complete. 2023-11-28T17:17:02.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.018+0000 7f007dd88640 1 Processor -- start 2023-11-28T17:17:02.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.018+0000 7f007dd88640 1 -- start start 2023-11-28T17:17:02.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.019+0000 7f007dd88640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00780b1d90 0x7f0078155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.019+0000 7f007dd88640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00780d14d0 0x7f00780b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.019+0000 7f007dd88640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0078160680 0x7f00780b6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.019+0000 7f007dd88640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00780b88b0 con 0x7f0078160680 2023-11-28T17:17:02.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.019+0000 7f007dd88640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00780c98f0 con 0x7f00780b1d90 2023-11-28T17:17:02.023 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.019+0000 7f007dd88640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00780b7cc0 con 0x7f00780d14d0 2023-11-28T17:17:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.019+0000 7f007d587640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0078160680 0x7f00780b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.019+0000 7f007d587640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0078160680 0x7f00780b6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53764/0 (socket says 172.21.15.116:53764) 2023-11-28T17:17:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.019+0000 7f007d587640 1 -- 172.21.15.116:0/1219287048 learned_addr learned my addr 172.21.15.116:0/1219287048 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.020+0000 7f007cd86640 1 --2- 172.21.15.116:0/1219287048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00780b1d90 0x7f0078155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.020+0000 7f007d587640 1 -- 172.21.15.116:0/1219287048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00780d14d0 msgr2=0x7f00780b5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.020+0000 7f0077fff640 1 --2- 172.21.15.116:0/1219287048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00780d14d0 0x7f00780b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.020+0000 7f007d587640 1 --2- 172.21.15.116:0/1219287048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00780d14d0 0x7f00780b5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.020+0000 7f007d587640 1 -- 172.21.15.116:0/1219287048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00780b1d90 msgr2=0x7f0078155280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.020+0000 7f007d587640 1 --2- 172.21.15.116:0/1219287048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00780b1d90 0x7f0078155280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.020+0000 7f007d587640 1 -- 172.21.15.116:0/1219287048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0064009400 con 0x7f0078160680 2023-11-28T17:17:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.020+0000 7f007d587640 1 --2- 172.21.15.116:0/1219287048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0078160680 0x7f00780b6480 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f007000c7b0 tx=0x7f007000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.020+0000 7f00757fa640 1 -- 172.21.15.116:0/1219287048 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0070004b60 con 0x7f0078160680 2023-11-28T17:17:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.020+0000 7f00757fa640 1 -- 172.21.15.116:0/1219287048 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0070004cc0 con 0x7f0078160680 2023-11-28T17:17:02.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.020+0000 7f007dd88640 1 -- 172.21.15.116:0/1219287048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00781557c0 con 0x7f0078160680 2023-11-28T17:17:02.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.020+0000 7f00757fa640 1 -- 172.21.15.116:0/1219287048 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007000aa90 con 0x7f0078160680 2023-11-28T17:17:02.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.020+0000 7f007dd88640 1 -- 172.21.15.116:0/1219287048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00780b81f0 con 0x7f0078160680 2023-11-28T17:17:02.025 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.022+0000 7f007dd88640 1 -- 172.21.15.116:0/1219287048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007804e880 con 0x7f0078160680 2023-11-28T17:17:02.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.022+0000 7f00757fa640 1 -- 172.21.15.116:0/1219287048 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f007000abf0 con 0x7f0078160680 2023-11-28T17:17:02.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.023+0000 7f00757fa640 1 --2- 172.21.15.116:0/1219287048 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f005c06b2a0 0x7f005c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.023+0000 7f00757fa640 1 -- 172.21.15.116:0/1219287048 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f0070089e00 con 0x7f0078160680 2023-11-28T17:17:02.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.024+0000 7f007cd86640 1 --2- 172.21.15.116:0/1219287048 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f005c06b2a0 0x7f005c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.026 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.024+0000 7f007cd86640 1 --2- 172.21.15.116:0/1219287048 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f005c06b2a0 0x7f005c06d760 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f006c0011e0 tx=0x7f006c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:02.028 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.027+0000 7f00757fa640 1 -- 172.21.15.116:0/1219287048 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0070054bc0 con 0x7f0078160680 2023-11-28T17:17:02.104 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.103+0000 7f34d359e640 1 Processor -- start 2023-11-28T17:17:02.105 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.103+0000 7f34d359e640 1 -- start start 2023-11-28T17:17:02.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.105+0000 7f34d359e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34d40b2fe0 0x7f34d40ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.105+0000 7f34d359e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d40b2b50 con 0x7f34d4152050 2023-11-28T17:17:02.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.105+0000 7f34d359e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d40ac2d0 con 0x7f34d40abd30 2023-11-28T17:17:02.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.105+0000 7f34d359e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d40b2220 con 0x7f34d40b2fe0 2023-11-28T17:17:02.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.105+0000 7f34d259c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f34d4152050 0x7f34d415bfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:32948/0 (socket says 172.21.15.116:32948) 2023-11-28T17:17:02.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.105+0000 7f34d259c640 1 -- 172.21.15.116:0/750390936 learned_addr learned my addr 172.21.15.116:0/750390936 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:17:02.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.105+0000 7f34d1d9b640 1 -- 172.21.15.116:0/750390936 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966528325 0 0) 0x7f34d40b2b50 con 0x7f34d4152050 2023-11-28T17:17:02.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.105+0000 7f34d1d9b640 1 -- 172.21.15.116:0/750390936 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c0003520 con 0x7f34d4152050 2023-11-28T17:17:02.108 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.105+0000 7f34cbfff640 1 --2- 172.21.15.116:0/750390936 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34d40b2fe0 0x7f34d40ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.105+0000 7f34d1d9b640 1 -- 172.21.15.116:0/750390936 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 210594442 0 0) 0x7f34c0003520 con 0x7f34d4152050 2023-11-28T17:17:02.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.105+0000 7f34d1d9b640 1 -- 172.21.15.116:0/750390936 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34d40b2fe0 msgr2=0x7f34d40ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:02.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.105+0000 7f34d1d9b640 1 --2- 172.21.15.116:0/750390936 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34d40b2fe0 0x7f34d40ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.105+0000 7f34d1d9b640 1 -- 172.21.15.116:0/750390936 >> v1:172.21.15.156:6789/0 conn(0x7f34d40abd30 legacy=0x7f34d40cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:02.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d1d9b640 1 -- 172.21.15.116:0/750390936 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d40b2500 con 0x7f34d4152050 2023-11-28T17:17:02.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d1d9b640 1 -- 172.21.15.116:0/750390936 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f34c4001670 con 0x7f34d4152050 2023-11-28T17:17:02.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d1d9b640 1 -- 172.21.15.116:0/750390936 >> v1:172.21.15.116:6789/0 conn(0x7f34d4152050 legacy=0x7f34d415bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d1d9b640 1 --2- 172.21.15.116:0/750390936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34c00039a0 0x7f34c0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d1d9b640 1 --2- 172.21.15.116:0/750390936 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34c00047c0 0x7f34c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d1d9b640 1 --2- 172.21.15.116:0/750390936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34c00051d0 0x7f34c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d1d9b640 1 -- 172.21.15.116:0/750390936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d40b2220 con 0x7f34c00039a0 2023-11-28T17:17:02.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d1d9b640 1 -- 172.21.15.116:0/750390936 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c000c100 con 0x7f34c00051d0 2023-11-28T17:17:02.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d1d9b640 1 -- 172.21.15.116:0/750390936 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c000c2c0 con 0x7f34c00047c0 2023-11-28T17:17:02.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34cbfff640 1 --2- 172.21.15.116:0/750390936 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34d40b2fe0 0x7f34d40ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:17:02.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d259c640 1 --2- 172.21.15.116:0/750390936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34c00039a0 0x7f34c0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d259c640 1 -- 172.21.15.116:0/750390936 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34c00047c0 msgr2=0x7f34c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:02.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d259c640 1 --2- 172.21.15.116:0/750390936 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34c00047c0 0x7f34c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d259c640 1 -- 172.21.15.116:0/750390936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34c00051d0 msgr2=0x7f34c000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:02.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d259c640 1 --2- 172.21.15.116:0/750390936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34c00051d0 0x7f34c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.111 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.106+0000 7f34d259c640 1 -- 172.21.15.116:0/750390936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c0003520 con 0x7f34c00039a0 2023-11-28T17:17:02.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.107+0000 7f34d259c640 1 --2- 172.21.15.116:0/750390936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34c00039a0 0x7f34c0003da0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f34c400c240 tx=0x7f34c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb5045d0b8fe02d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:02.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.108+0000 7f34d1d9b640 1 -- 172.21.15.116:0/750390936 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4005ad0 con 0x7f34c00039a0 2023-11-28T17:17:02.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.108+0000 7f34d1d9b640 1 -- 172.21.15.116:0/750390936 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c4005c30 con 0x7f34c00039a0 2023-11-28T17:17:02.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.108+0000 7f34d1d9b640 1 -- 172.21.15.116:0/750390936 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c402c8d0 con 0x7f34c00039a0 2023-11-28T17:17:02.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.108+0000 7f34d359e640 1 -- 172.21.15.116:0/750390936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34c00039a0 msgr2=0x7f34c0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.108+0000 7f34d359e640 1 --2- 172.21.15.116:0/750390936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34c00039a0 0x7f34c0003da0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f34c400c240 tx=0x7f34c00057f0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.108+0000 7f34d2d9d640 1 -- 172.21.15.116:0/750390936 reap_dead start 2023-11-28T17:17:02.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.108+0000 7f34d359e640 1 -- 172.21.15.116:0/750390936 shutdown_connections 2023-11-28T17:17:02.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.108+0000 7f34d359e640 1 -- 172.21.15.116:0/750390936 >> 172.21.15.116:0/750390936 conn(0x7f34d40b0950 msgr2=0x7f34d40b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:02.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.108+0000 7f34d359e640 1 -- 172.21.15.116:0/750390936 shutdown_connections 2023-11-28T17:17:02.115 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.108+0000 7f34d359e640 1 -- 172.21.15.116:0/750390936 wait complete. 2023-11-28T17:17:02.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.110+0000 7f34d359e640 1 Processor -- start 2023-11-28T17:17:02.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.110+0000 7f34d359e640 1 -- start start 2023-11-28T17:17:02.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.111+0000 7f34d359e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34c00039a0 0x7f34d4142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.111+0000 7f34d359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34d4146270 0x7f34d415d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.111+0000 7f34d359e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34d415dde0 0x7f34d41545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.116 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.111+0000 7f34d359e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d409a1f0 con 0x7f34d4146270 2023-11-28T17:17:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.111+0000 7f34d359e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d415e360 con 0x7f34c00039a0 2023-11-28T17:17:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.111+0000 7f34d359e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d4099ba0 con 0x7f34d415dde0 2023-11-28T17:17:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.111+0000 7f34cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34d4146270 0x7f34d415d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.111+0000 7f34cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34d4146270 0x7f34d415d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53780/0 (socket says 172.21.15.116:53780) 2023-11-28T17:17:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.111+0000 7f34cbfff640 1 -- 172.21.15.116:0/337040566 learned_addr learned my addr 172.21.15.116:0/337040566 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:02.117 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.111+0000 7f34cbfff640 1 -- 172.21.15.116:0/337040566 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34d415dde0 msgr2=0x7f34d41545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.111+0000 7f34cbfff640 1 --2- 172.21.15.116:0/337040566 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34d415dde0 0x7f34d41545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.111+0000 7f34cbfff640 1 -- 172.21.15.116:0/337040566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34c00039a0 msgr2=0x7f34d4142560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.111+0000 7f34cbfff640 1 --2- 172.21.15.116:0/337040566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34c00039a0 0x7f34d4142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.111+0000 7f34cbfff640 1 -- 172.21.15.116:0/337040566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d4099e80 con 0x7f34d4146270 2023-11-28T17:17:02.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.112+0000 7f34cbfff640 1 --2- 172.21.15.116:0/337040566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34d4146270 0x7f34d415d8a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f34bc006a90 tx=0x7f34bc00c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:02.118 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.112+0000 7f34caffd640 1 -- 172.21.15.116:0/337040566 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc004230 con 0x7f34d4146270 2023-11-28T17:17:02.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.112+0000 7f34caffd640 1 -- 172.21.15.116:0/337040566 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34bc004390 con 0x7f34d4146270 2023-11-28T17:17:02.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.112+0000 7f34caffd640 1 -- 172.21.15.116:0/337040566 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc003850 con 0x7f34d4146270 2023-11-28T17:17:02.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.112+0000 7f34d359e640 1 -- 172.21.15.116:0/337040566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d4142aa0 con 0x7f34d4146270 2023-11-28T17:17:02.119 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.112+0000 7f34d359e640 1 -- 172.21.15.116:0/337040566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d4154b00 con 0x7f34d4146270 2023-11-28T17:17:02.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.113+0000 7f34d359e640 1 -- 172.21.15.116:0/337040566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d4099a00 con 0x7f34d4146270 2023-11-28T17:17:02.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.114+0000 7f34caffd640 1 -- 172.21.15.116:0/337040566 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f34bc0039b0 con 0x7f34d4146270 2023-11-28T17:17:02.120 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.115+0000 7f34caffd640 1 --2- 172.21.15.116:0/337040566 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f34b406b2d0 0x7f34b406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.115+0000 7f34caffd640 1 -- 172.21.15.116:0/337040566 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f34bc089c10 con 0x7f34d4146270 2023-11-28T17:17:02.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.115+0000 7f34d259c640 1 --2- 172.21.15.116:0/337040566 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f34b406b2d0 0x7f34b406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.116+0000 7f34d259c640 1 --2- 172.21.15.116:0/337040566 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f34b406b2d0 0x7f34b406d790 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f34c4034000 tx=0x7f34c4007000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:02.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.119+0000 7f34caffd640 1 -- 172.21.15.116:0/337040566 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34d4099a00 con 0x7f34d4146270 2023-11-28T17:17:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.237+0000 7f007dd88640 1 -- 172.21.15.116:0/1219287048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f007806cb60 con 0x7f0078160680 2023-11-28T17:17:02.239 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.238+0000 7f00757fa640 1 -- 172.21.15.116:0/1219287048 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v52) v1 ==== 80+0+246 (secure 0 0 0) 0x7f00700581e0 con 0x7f0078160680 2023-11-28T17:17:02.240 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.156:6833/372407661 2023-11-29T17:12:26.038071+0000 2023-11-28T17:17:02.240 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.156:6832/372407661 2023-11-29T17:12:26.038071+0000 2023-11-28T17:17:02.240 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/3251347229 2023-11-29T17:12:06.030424+0000 2023-11-28T17:17:02.240 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/3251347229 2023-11-29T17:12:06.030424+0000 2023-11-28T17:17:02.240 INFO:teuthology.orchestra.run.smithi116.stderr:listed 4 entries 2023-11-28T17:17:02.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.240+0000 7f007dd88640 1 -- 172.21.15.116:0/1219287048 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f005c06b2a0 msgr2=0x7f005c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.240+0000 7f007dd88640 1 --2- 172.21.15.116:0/1219287048 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f005c06b2a0 0x7f005c06d760 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f006c0011e0 tx=0x7f006c009040 comp rx=0 tx=0).stop 2023-11-28T17:17:02.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.240+0000 7f007dd88640 1 -- 172.21.15.116:0/1219287048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0078160680 msgr2=0x7f00780b6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.240+0000 7f007dd88640 1 --2- 172.21.15.116:0/1219287048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0078160680 0x7f00780b6480 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f007000c7b0 tx=0x7f007000cc70 comp rx=0 tx=0).stop 2023-11-28T17:17:02.242 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.241+0000 7f007dd88640 1 -- 172.21.15.116:0/1219287048 shutdown_connections 2023-11-28T17:17:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.241+0000 7f007dd88640 1 --2- 172.21.15.116:0/1219287048 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f005c06b2a0 0x7f005c06d760 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.241+0000 7f007dd88640 1 --2- 172.21.15.116:0/1219287048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0078160680 0x7f00780b6480 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.241+0000 7f007dd88640 1 --2- 172.21.15.116:0/1219287048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00780d14d0 0x7f00780b5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.241+0000 7f007dd88640 1 --2- 172.21.15.116:0/1219287048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00780b1d90 0x7f0078155280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.241+0000 7f007dd88640 1 -- 172.21.15.116:0/1219287048 >> 172.21.15.116:0/1219287048 conn(0x7f00780d2ef0 msgr2=0x7f00780b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.241+0000 7f007dd88640 1 -- 172.21.15.116:0/1219287048 shutdown_connections 2023-11-28T17:17:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.241+0000 7f007dd88640 1 -- 172.21.15.116:0/1219287048 wait complete. 2023-11-28T17:17:02.261 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/fuse_mount.py", line 399, in umount_wait self.umount(cleanup=False) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/fuse_mount.py", line 320, in umount if self.is_blocked(): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/mount.py", line 197, in is_blocked return self.addr in output TypeError: 'in ' requires string as left operand, not NoneType 2023-11-28T17:17:02.263 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:17:02.263 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:17:02.314 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054299,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T17:17:02.314 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.2 is not mounted at ubuntu@smithi116.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2 2023-11-28T17:17:02.314 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:17:02.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:17:02.315 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-11-28T17:17:02.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.313+0000 7f34d359e640 1 -- 172.21.15.116:0/337040566 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34d4099a00 con 0x7f34d4146270 2023-11-28T17:17:02.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.313+0000 7f34caffd640 1 -- 172.21.15.116:0/337040566 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+11 (secure 0 0 0) 0x7f34d4099a00 con 0x7f34d4146270 2023-11-28T17:17:02.316 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:17:02.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.315+0000 7f34d359e640 1 -- 172.21.15.116:0/337040566 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f34b406b2d0 msgr2=0x7f34b406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.315+0000 7f34d359e640 1 --2- 172.21.15.116:0/337040566 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f34b406b2d0 0x7f34b406d790 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f34c4034000 tx=0x7f34c4007000 comp rx=0 tx=0).stop 2023-11-28T17:17:02.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.315+0000 7f34d359e640 1 -- 172.21.15.116:0/337040566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34d4146270 msgr2=0x7f34d415d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.315+0000 7f34d359e640 1 --2- 172.21.15.116:0/337040566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34d4146270 0x7f34d415d8a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f34bc006a90 tx=0x7f34bc00c9d0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.316+0000 7f34d359e640 1 -- 172.21.15.116:0/337040566 shutdown_connections 2023-11-28T17:17:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.316+0000 7f34d359e640 1 --2- 172.21.15.116:0/337040566 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f34b406b2d0 0x7f34b406d790 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.316+0000 7f34d359e640 1 --2- 172.21.15.116:0/337040566 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34d415dde0 0x7f34d41545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.316+0000 7f34d359e640 1 --2- 172.21.15.116:0/337040566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34d4146270 0x7f34d415d8a0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.316+0000 7f34d359e640 1 --2- 172.21.15.116:0/337040566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34c00039a0 0x7f34d4142560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.316+0000 7f34d359e640 1 -- 172.21.15.116:0/337040566 >> 172.21.15.116:0/337040566 conn(0x7f34d40b0950 msgr2=0x7f34d40b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:02.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.316+0000 7f34d359e640 1 -- 172.21.15.116:0/337040566 shutdown_connections 2023-11-28T17:17:02.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.316+0000 7f34d359e640 1 -- 172.21.15.116:0/337040566 wait complete. 2023-11-28T17:17:02.333 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:17:02.356 INFO:teuthology.orchestra.run.smithi116.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-11-28T17:17:02.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:17:02.357 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:17:02.357 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:17:02.419 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:17:02.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.537+0000 7f659f4b4640 1 Processor -- start 2023-11-28T17:17:02.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.537+0000 7f659f4b4640 1 -- start start 2023-11-28T17:17:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.539+0000 7f659f4b4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6598160950 0x7f65980b8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.539+0000 7f659f4b4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65980d1650 con 0x7f65980d11b0 2023-11-28T17:17:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.539+0000 7f659f4b4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65980d1930 con 0x7f65980b52e0 2023-11-28T17:17:02.540 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.539+0000 7f659f4b4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6598160d30 con 0x7f6598160950 2023-11-28T17:17:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.539+0000 7f659d250640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6598160950 0x7f65980b8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.539+0000 7f659d250640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6598160950 0x7f65980b8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53788/0 (socket says 172.21.15.116:53788) 2023-11-28T17:17:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.539+0000 7f659d250640 1 -- 172.21.15.116:0/2050920308 learned_addr learned my addr 172.21.15.116:0/2050920308 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:02.541 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.539+0000 7f659ca4f640 1 --1- 172.21.15.116:0/2050920308 >> v1:172.21.15.116:6789/0 conn(0x7f65980d11b0 0x7f65980b4bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:32950/0 (socket says 172.21.15.116:32950) 2023-11-28T17:17:02.542 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.539+0000 7f659d250640 1 -- 172.21.15.116:0/2050920308 >> v1:172.21.15.156:6789/0 conn(0x7f65980b52e0 legacy=0x7f65980b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.540+0000 7f659d250640 1 -- 172.21.15.116:0/2050920308 >> v1:172.21.15.116:6789/0 conn(0x7f65980d11b0 legacy=0x7f65980b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.540+0000 7f659d250640 1 -- 172.21.15.116:0/2050920308 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65980d1c10 con 0x7f6598160950 2023-11-28T17:17:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.540+0000 7f658ffff640 1 -- 172.21.15.116:0/2050920308 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376117888 0 0) 0x7f65980d1650 con 0x7f65980d11b0 2023-11-28T17:17:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.541+0000 7f659d250640 1 --2- 172.21.15.116:0/2050920308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6598160950 0x7f65980b8a40 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6594009870 tx=0x7f659402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0587ccca1736a47 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.541+0000 7f658ffff640 1 -- 172.21.15.116:0/2050920308 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594030860 con 0x7f6598160950 2023-11-28T17:17:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.541+0000 7f658ffff640 1 -- 172.21.15.116:0/2050920308 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65940309c0 con 0x7f6598160950 2023-11-28T17:17:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.541+0000 7f658ffff640 1 -- 172.21.15.116:0/2050920308 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594030d50 con 0x7f6598160950 2023-11-28T17:17:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.541+0000 7f659f4b4640 1 -- 172.21.15.116:0/2050920308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6598160950 msgr2=0x7f65980b8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.543 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.541+0000 7f659f4b4640 1 --2- 172.21.15.116:0/2050920308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6598160950 0x7f65980b8a40 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6594009870 tx=0x7f659402b9a0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.542+0000 7f659f4b4640 1 -- 172.21.15.116:0/2050920308 shutdown_connections 2023-11-28T17:17:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.542+0000 7f659f4b4640 1 --2- 172.21.15.116:0/2050920308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6598160950 0x7f65980b8a40 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.544 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.542+0000 7f659f4b4640 1 -- 172.21.15.116:0/2050920308 >> 172.21.15.116:0/2050920308 conn(0x7f6598072220 msgr2=0x7f65980cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.542+0000 7f659f4b4640 1 -- 172.21.15.116:0/2050920308 shutdown_connections 2023-11-28T17:17:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.542+0000 7f659f4b4640 1 -- 172.21.15.116:0/2050920308 wait complete. 2023-11-28T17:17:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.543+0000 7f659f4b4640 1 Processor -- start 2023-11-28T17:17:02.545 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.543+0000 7f659f4b4640 1 -- start start 2023-11-28T17:17:02.546 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.544+0000 7f659f4b4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65980b52e0 0x7f65980b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.544+0000 7f659f4b4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65980d11b0 0x7f65980c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.544+0000 7f659f4b4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6598160950 0x7f65980c87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.544+0000 7f659f4b4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65981f30a0 con 0x7f65980b52e0 2023-11-28T17:17:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.544+0000 7f659f4b4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65980b2430 con 0x7f65980d11b0 2023-11-28T17:17:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.544+0000 7f659f4b4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65980c96d0 con 0x7f6598160950 2023-11-28T17:17:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.545+0000 7f659d250640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65980b52e0 0x7f65980b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.545+0000 7f659d250640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65980b52e0 0x7f65980b47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53792/0 (socket says 172.21.15.116:53792) 2023-11-28T17:17:02.547 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.545+0000 7f659d250640 1 -- 172.21.15.116:0/1837947112 learned_addr learned my addr 172.21.15.116:0/1837947112 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.545+0000 7f659d250640 1 -- 172.21.15.116:0/1837947112 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6598160950 msgr2=0x7f65980c87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.545+0000 7f659d250640 1 --2- 172.21.15.116:0/1837947112 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6598160950 0x7f65980c87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.545+0000 7f659d250640 1 -- 172.21.15.116:0/1837947112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65980d11b0 msgr2=0x7f65980c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.545+0000 7f659ca4f640 1 --2- 172.21.15.116:0/1837947112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65980d11b0 0x7f65980c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.545+0000 7f659d250640 1 --2- 172.21.15.116:0/1837947112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65980d11b0 0x7f65980c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.545+0000 7f659d250640 1 -- 172.21.15.116:0/1837947112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65940094d0 con 0x7f65980b52e0 2023-11-28T17:17:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.545+0000 7f659ca4f640 1 --2- 172.21.15.116:0/1837947112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65980d11b0 0x7f65980c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:17:02.548 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.546+0000 7f659d250640 1 --2- 172.21.15.116:0/1837947112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65980b52e0 0x7f65980b47f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6594009870 tx=0x7f6594004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.546+0000 7f658dffb640 1 -- 172.21.15.116:0/1837947112 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594039c20 con 0x7f65980b52e0 2023-11-28T17:17:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.546+0000 7f658dffb640 1 -- 172.21.15.116:0/1837947112 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6594039d80 con 0x7f65980b52e0 2023-11-28T17:17:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.546+0000 7f658dffb640 1 -- 172.21.15.116:0/1837947112 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594043630 con 0x7f65980b52e0 2023-11-28T17:17:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.546+0000 7f659f4b4640 1 -- 172.21.15.116:0/1837947112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65980d2870 con 0x7f65980b52e0 2023-11-28T17:17:02.549 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.546+0000 7f659f4b4640 1 -- 172.21.15.116:0/1837947112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65980b26d0 con 0x7f65980b52e0 2023-11-28T17:17:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.547+0000 7f659f4b4640 1 -- 172.21.15.116:0/1837947112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65980693d0 con 0x7f65980b52e0 2023-11-28T17:17:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.548+0000 7f658dffb640 1 -- 172.21.15.116:0/1837947112 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f659402f3b0 con 0x7f65980b52e0 2023-11-28T17:17:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.549+0000 7f658dffb640 1 --2- 172.21.15.116:0/1837947112 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f657c06b2d0 0x7f657c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.550 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.549+0000 7f658dffb640 1 -- 172.21.15.116:0/1837947112 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f65940ac0a0 con 0x7f65980b52e0 2023-11-28T17:17:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.549+0000 7f659ca4f640 1 --2- 172.21.15.116:0/1837947112 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f657c06b2d0 0x7f657c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.551 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.550+0000 7f659ca4f640 1 --2- 172.21.15.116:0/1837947112 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f657c06b2d0 0x7f657c06d790 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f65981f66f0 tx=0x7f6588009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:02.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.553+0000 7f658dffb640 1 -- 172.21.15.116:0/1837947112 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6594076d30 con 0x7f65980b52e0 2023-11-28T17:17:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.604+0000 7f84954ec640 1 Processor -- start 2023-11-28T17:17:02.605 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.604+0000 7f84954ec640 1 -- start start 2023-11-28T17:17:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.605+0000 7f84954ec640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f84901460c0 0x7f84900b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.605+0000 7f84954ec640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f84900d17c0 0x7f849006bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.605+0000 7f84954ec640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8490161d10 con 0x7f84901464a0 2023-11-28T17:17:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.605+0000 7f84954ec640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8490146fa0 con 0x7f84901460c0 2023-11-28T17:17:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.605+0000 7f84954ec640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f849015d990 con 0x7f84900d17c0 2023-11-28T17:17:02.615 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.606+0000 7f848effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f84901460c0 0x7f84900b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.606+0000 7f84867fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f84901464a0 0x7f84900d10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:32958/0 (socket says 172.21.15.116:32958) 2023-11-28T17:17:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.606+0000 7f84867fc640 1 -- 172.21.15.116:0/2770321690 learned_addr learned my addr 172.21.15.116:0/2770321690 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:17:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.606+0000 7f848f7fe640 1 --2- 172.21.15.116:0/2770321690 >> v2:172.21.15.156:3300/0 conn(0x7f84900d17c0 0x7f849006bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.606+0000 7f848effd640 1 -- 172.21.15.116:0/2770321690 >> v2:172.21.15.156:3300/0 conn(0x7f84900d17c0 msgr2=0x7f849006bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.606+0000 7f848effd640 1 --2- 172.21.15.116:0/2770321690 >> v2:172.21.15.156:3300/0 conn(0x7f84900d17c0 0x7f849006bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.606+0000 7f848effd640 1 -- 172.21.15.116:0/2770321690 >> v1:172.21.15.116:6789/0 conn(0x7f84901464a0 legacy=0x7f84900d10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.616 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.606+0000 7f848effd640 1 -- 172.21.15.116:0/2770321690 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8490161100 con 0x7f84901460c0 2023-11-28T17:17:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.606+0000 7f848e7fc640 1 -- 172.21.15.116:0/2770321690 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123969972 0 0) 0x7f8490161d10 con 0x7f84901464a0 2023-11-28T17:17:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.607+0000 7f848effd640 1 --2- 172.21.15.116:0/2770321690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84901460c0 0x7f84900b4a00 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f8490161c80 tx=0x7f848002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1991d7fb05d88b7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.608+0000 7f848e7fc640 1 -- 172.21.15.116:0/2770321690 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8480031830 con 0x7f84901460c0 2023-11-28T17:17:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.608+0000 7f848e7fc640 1 -- 172.21.15.116:0/2770321690 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8480031990 con 0x7f84901460c0 2023-11-28T17:17:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.608+0000 7f848e7fc640 1 -- 172.21.15.116:0/2770321690 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8480031d20 con 0x7f84901460c0 2023-11-28T17:17:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.608+0000 7f84954ec640 1 -- 172.21.15.116:0/2770321690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84901460c0 msgr2=0x7f84900b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.608+0000 7f84954ec640 1 --2- 172.21.15.116:0/2770321690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84901460c0 0x7f84900b4a00 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f8490161c80 tx=0x7f848002b8d0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.608+0000 7f84954ec640 1 -- 172.21.15.116:0/2770321690 shutdown_connections 2023-11-28T17:17:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.608+0000 7f84954ec640 1 --2- 172.21.15.116:0/2770321690 >> v2:172.21.15.156:3300/0 conn(0x7f84900d17c0 0x7f849006bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.617 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.608+0000 7f84954ec640 1 --2- 172.21.15.116:0/2770321690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84901460c0 0x7f84900b4a00 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.608+0000 7f84954ec640 1 -- 172.21.15.116:0/2770321690 >> 172.21.15.116:0/2770321690 conn(0x7f849006a4c0 msgr2=0x7f849006a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.608+0000 7f84954ec640 1 -- 172.21.15.116:0/2770321690 shutdown_connections 2023-11-28T17:17:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.608+0000 7f84954ec640 1 -- 172.21.15.116:0/2770321690 wait complete. 2023-11-28T17:17:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.610+0000 7f84954ec640 1 Processor -- start 2023-11-28T17:17:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.610+0000 7f84954ec640 1 -- start start 2023-11-28T17:17:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.611+0000 7f84954ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84900d17c0 0x7f8490141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.611+0000 7f84954ec640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84901460c0 0x7f849015e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.618 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.611+0000 7f84954ec640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84901464a0 0x7f84901526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.611+0000 7f84954ec640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8490144210 con 0x7f84900d17c0 2023-11-28T17:17:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.611+0000 7f84954ec640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8490144420 con 0x7f84901464a0 2023-11-28T17:17:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.611+0000 7f84954ec640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8490144700 con 0x7f84901460c0 2023-11-28T17:17:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.611+0000 7f848effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84900d17c0 0x7f8490141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.611+0000 7f848effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84900d17c0 0x7f8490141c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53816/0 (socket says 172.21.15.116:53816) 2023-11-28T17:17:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.611+0000 7f848effd640 1 -- 172.21.15.116:0/4057719736 learned_addr learned my addr 172.21.15.116:0/4057719736 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.612+0000 7f84867fc640 1 --2- 172.21.15.116:0/4057719736 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84901460c0 0x7f849015e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.612+0000 7f848effd640 1 -- 172.21.15.116:0/4057719736 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84901460c0 msgr2=0x7f849015e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.612+0000 7f848effd640 1 --2- 172.21.15.116:0/4057719736 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84901460c0 0x7f849015e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.619 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.612+0000 7f848effd640 1 -- 172.21.15.116:0/4057719736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84901464a0 msgr2=0x7f84901526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.612+0000 7f848effd640 1 --2- 172.21.15.116:0/4057719736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84901464a0 0x7f84901526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.612+0000 7f848effd640 1 -- 172.21.15.116:0/4057719736 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8480009400 con 0x7f84900d17c0 2023-11-28T17:17:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.612+0000 7f848effd640 1 --2- 172.21.15.116:0/4057719736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84900d17c0 0x7f8490141c30 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f8480009890 tx=0x7f84800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.612+0000 7f8487fff640 1 -- 172.21.15.116:0/4057719736 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848003ba70 con 0x7f84900d17c0 2023-11-28T17:17:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.612+0000 7f8487fff640 1 -- 172.21.15.116:0/4057719736 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848003bbd0 con 0x7f84900d17c0 2023-11-28T17:17:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.612+0000 7f8487fff640 1 -- 172.21.15.116:0/4057719736 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84800453f0 con 0x7f84900d17c0 2023-11-28T17:17:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.612+0000 7f84954ec640 1 -- 172.21.15.116:0/4057719736 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84901449e0 con 0x7f84900d17c0 2023-11-28T17:17:02.620 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.613+0000 7f84954ec640 1 -- 172.21.15.116:0/4057719736 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8490142170 con 0x7f84900d17c0 2023-11-28T17:17:02.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.614+0000 7f84954ec640 1 -- 172.21.15.116:0/4057719736 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84900693d0 con 0x7f84900d17c0 2023-11-28T17:17:02.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.615+0000 7f8487fff640 1 -- 172.21.15.116:0/4057719736 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f8480045550 con 0x7f84900d17c0 2023-11-28T17:17:02.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.616+0000 7f8487fff640 1 --2- 172.21.15.116:0/4057719736 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f847006b2a0 0x7f847006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:02.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.616+0000 7f8487fff640 1 -- 172.21.15.116:0/4057719736 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f84800ac250 con 0x7f84900d17c0 2023-11-28T17:17:02.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.616+0000 7f84867fc640 1 --2- 172.21.15.116:0/4057719736 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f847006b2a0 0x7f847006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:02.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.617+0000 7f84867fc640 1 --2- 172.21.15.116:0/4057719736 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f847006b2a0 0x7f847006d760 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f849013d190 tx=0x7f8478009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:02.622 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.620+0000 7f8487fff640 1 -- 172.21.15.116:0/4057719736 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8480077090 con 0x7f84900d17c0 2023-11-28T17:17:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.757+0000 7f659f4b4640 1 -- 172.21.15.116:0/1837947112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f65981540b0 con 0x7f65980b52e0 2023-11-28T17:17:02.758 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.757+0000 7f658dffb640 1 -- 172.21.15.116:0/1837947112 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v52) v1 ==== 80+0+246 (secure 0 0 0) 0x7f6594034110 con 0x7f65980b52e0 2023-11-28T17:17:02.759 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.156:6833/372407661 2023-11-29T17:12:26.038071+0000 2023-11-28T17:17:02.759 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.156:6832/372407661 2023-11-29T17:12:26.038071+0000 2023-11-28T17:17:02.759 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/3251347229 2023-11-29T17:12:06.030424+0000 2023-11-28T17:17:02.760 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/3251347229 2023-11-29T17:12:06.030424+0000 2023-11-28T17:17:02.760 INFO:teuthology.orchestra.run.smithi116.stderr:listed 4 entries 2023-11-28T17:17:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.760+0000 7f659f4b4640 1 -- 172.21.15.116:0/1837947112 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f657c06b2d0 msgr2=0x7f657c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.761 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.760+0000 7f659f4b4640 1 --2- 172.21.15.116:0/1837947112 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f657c06b2d0 0x7f657c06d790 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f65981f66f0 tx=0x7f6588009040 comp rx=0 tx=0).stop 2023-11-28T17:17:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.760+0000 7f659f4b4640 1 -- 172.21.15.116:0/1837947112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65980b52e0 msgr2=0x7f65980b47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.760+0000 7f659f4b4640 1 --2- 172.21.15.116:0/1837947112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65980b52e0 0x7f65980b47f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f6594009870 tx=0x7f6594004730 comp rx=0 tx=0).stop 2023-11-28T17:17:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.760+0000 7f659f4b4640 1 -- 172.21.15.116:0/1837947112 shutdown_connections 2023-11-28T17:17:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.760+0000 7f659f4b4640 1 --2- 172.21.15.116:0/1837947112 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f657c06b2d0 0x7f657c06d790 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.760+0000 7f659f4b4640 1 --2- 172.21.15.116:0/1837947112 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6598160950 0x7f65980c87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.760+0000 7f659f4b4640 1 --2- 172.21.15.116:0/1837947112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f65980d11b0 0x7f65980c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.760+0000 7f659f4b4640 1 --2- 172.21.15.116:0/1837947112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65980b52e0 0x7f65980b47f0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.760+0000 7f659f4b4640 1 -- 172.21.15.116:0/1837947112 >> 172.21.15.116:0/1837947112 conn(0x7f6598072220 msgr2=0x7f659815d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.760+0000 7f659f4b4640 1 -- 172.21.15.116:0/1837947112 shutdown_connections 2023-11-28T17:17:02.762 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.760+0000 7f659f4b4640 1 -- 172.21.15.116:0/1837947112 wait complete. 2023-11-28T17:17:02.778 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2023-11-28T17:17:02.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:17:02.778 DEBUG:teuthology.orchestra.run.smithi156:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-11-28T17:17:02.802 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.801+0000 7f84954ec640 1 -- 172.21.15.116:0/4057719736 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8490140170 con 0x7f84900d17c0 2023-11-28T17:17:02.803 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.802+0000 7f8487fff640 1 -- 172.21.15.116:0/4057719736 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+17073 (secure 0 0 0) 0x7f848007a6b0 con 0x7f84900d17c0 2023-11-28T17:17:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:17:02.804 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":52,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:12:26.038218+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:11:21.518078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T17:11:22.672451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":0}]},{"pgid":"7.8","mappings":[{"from":5,"to":7}]},{"pgid":"7.1b","mappings":[{"from":5,"to":7}]},{"pgid":"7.30","mappings":[{"from":5,"to":7}]},{"pgid":"7.31","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6833/372407661":"2023-11-29T17:12:26.038071+0000","172.21.15.156:6832/372407661":"2023-11-29T17:12:26.038071+0000","172.21.15.116:6835/3251347229":"2023-11-29T17:12:06.030424+0000","172.21.15.116:6834/3251347229":"2023-11-29T17:12:06.030424+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:17:02.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.804+0000 7f84954ec640 1 -- 172.21.15.116:0/4057719736 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f847006b2a0 msgr2=0x7f847006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.804+0000 7f84954ec640 1 --2- 172.21.15.116:0/4057719736 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f847006b2a0 0x7f847006d760 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f849013d190 tx=0x7f8478009040 comp rx=0 tx=0).stop 2023-11-28T17:17:02.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.804+0000 7f84954ec640 1 -- 172.21.15.116:0/4057719736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84900d17c0 msgr2=0x7f8490141c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:02.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.804+0000 7f84954ec640 1 --2- 172.21.15.116:0/4057719736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84900d17c0 0x7f8490141c30 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f8480009890 tx=0x7f84800046c0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.805+0000 7f84954ec640 1 -- 172.21.15.116:0/4057719736 shutdown_connections 2023-11-28T17:17:02.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.805+0000 7f84954ec640 1 --2- 172.21.15.116:0/4057719736 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f847006b2a0 0x7f847006d760 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.805+0000 7f84954ec640 1 --2- 172.21.15.116:0/4057719736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84901464a0 0x7f84901526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.805+0000 7f84954ec640 1 --2- 172.21.15.116:0/4057719736 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84901460c0 0x7f849015e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.805+0000 7f84954ec640 1 --2- 172.21.15.116:0/4057719736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84900d17c0 0x7f8490141c30 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:02.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.805+0000 7f84954ec640 1 -- 172.21.15.116:0/4057719736 >> 172.21.15.116:0/4057719736 conn(0x7f849006a4c0 msgr2=0x7f84900b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:02.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.805+0000 7f84954ec640 1 -- 172.21.15.116:0/4057719736 shutdown_connections 2023-11-28T17:17:02.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:02.805+0000 7f84954ec640 1 -- 172.21.15.116:0/4057719736 wait complete. 2023-11-28T17:17:02.822 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:11:21.518078+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:11:22.672451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:17:02.823 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:17:02.884 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:17:02.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:17:02.884 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-28T17:17:02.936 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:17:02.936 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:17:02.985 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054299,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T17:17:02.985 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:17:02.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:17:02.985 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-11-28T17:17:03.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.065+0000 7f56741b0640 1 Processor -- start 2023-11-28T17:17:03.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.065+0000 7f56741b0640 1 -- start start 2023-11-28T17:17:03.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.066+0000 7f56741b0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f566c0d01e0 0x7f566c160270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.066+0000 7f56741b0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566c15ddc0 con 0x7f566c15d860 2023-11-28T17:17:03.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.066+0000 7f56741b0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566c15e0a0 con 0x7f566c15d480 2023-11-28T17:17:03.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.066+0000 7f56741b0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f566c15e380 con 0x7f566c0d01e0 2023-11-28T17:17:03.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.067+0000 7f567274d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f566c0d01e0 0x7f566c160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.067+0000 7f567174b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f566c15d860 0x7f566c0cfad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:32970/0 (socket says 172.21.15.116:32970) 2023-11-28T17:17:03.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.067+0000 7f567174b640 1 -- 172.21.15.116:0/2561540835 learned_addr learned my addr 172.21.15.116:0/2561540835 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:17:03.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.067+0000 7f567274d640 1 -- 172.21.15.116:0/2561540835 >> v1:172.21.15.156:6789/0 conn(0x7f566c15d480 legacy=0x7f566c0d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:03.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.067+0000 7f567274d640 1 -- 172.21.15.116:0/2561540835 >> v1:172.21.15.116:6789/0 conn(0x7f566c15d860 legacy=0x7f566c0cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.067+0000 7f567274d640 1 -- 172.21.15.116:0/2561540835 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c15e660 con 0x7f566c0d01e0 2023-11-28T17:17:03.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.068+0000 7f567274d640 1 --2- 172.21.15.116:0/2561540835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566c0d01e0 0x7f566c160270 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f5664009dc0 tx=0x7f566402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5431dbe1da29039 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.068+0000 7f5670f4a640 1 -- 172.21.15.116:0/2561540835 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566402cc40 con 0x7f566c0d01e0 2023-11-28T17:17:03.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.068+0000 7f5670f4a640 1 -- 172.21.15.116:0/2561540835 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566402cda0 con 0x7f566c0d01e0 2023-11-28T17:17:03.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.068+0000 7f5670f4a640 1 -- 172.21.15.116:0/2561540835 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664034660 con 0x7f566c0d01e0 2023-11-28T17:17:03.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.069+0000 7f56741b0640 1 -- 172.21.15.116:0/2561540835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566c0d01e0 msgr2=0x7f566c160270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.069+0000 7f56741b0640 1 --2- 172.21.15.116:0/2561540835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566c0d01e0 0x7f566c160270 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f5664009dc0 tx=0x7f566402be90 comp rx=0 tx=0).stop 2023-11-28T17:17:03.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.069+0000 7f56741b0640 1 -- 172.21.15.116:0/2561540835 shutdown_connections 2023-11-28T17:17:03.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.069+0000 7f56741b0640 1 --2- 172.21.15.116:0/2561540835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566c0d01e0 0x7f566c160270 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.071 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.069+0000 7f56741b0640 1 -- 172.21.15.116:0/2561540835 >> 172.21.15.116:0/2561540835 conn(0x7f566c09c540 msgr2=0x7f566c0d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:03.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.069+0000 7f56741b0640 1 -- 172.21.15.116:0/2561540835 shutdown_connections 2023-11-28T17:17:03.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.069+0000 7f56741b0640 1 -- 172.21.15.116:0/2561540835 wait complete. 2023-11-28T17:17:03.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.071+0000 7f56741b0640 1 Processor -- start 2023-11-28T17:17:03.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.071+0000 7f56741b0640 1 -- start start 2023-11-28T17:17:03.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.072+0000 7f56741b0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566c0d01e0 0x7f566c149410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.072+0000 7f56741b0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f566c15d480 0x7f566c0afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.072+0000 7f56741b0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f566c15d860 0x7f566c0cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.072+0000 7f56741b0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c0ca0f0 con 0x7f566c0d01e0 2023-11-28T17:17:03.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.072+0000 7f56741b0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c0b1770 con 0x7f566c15d860 2023-11-28T17:17:03.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.072+0000 7f56741b0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f566c0c9b80 con 0x7f566c15d480 2023-11-28T17:17:03.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.072+0000 7f5671f4c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566c0d01e0 0x7f566c149410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.072+0000 7f5671f4c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566c0d01e0 0x7f566c149410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53834/0 (socket says 172.21.15.116:53834) 2023-11-28T17:17:03.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.072+0000 7f5671f4c640 1 -- 172.21.15.116:0/3853960255 learned_addr learned my addr 172.21.15.116:0/3853960255 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:03.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.072+0000 7f5671f4c640 1 -- 172.21.15.116:0/3853960255 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f566c15d480 msgr2=0x7f566c0afe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.072+0000 7f5671f4c640 1 --2- 172.21.15.116:0/3853960255 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f566c15d480 0x7f566c0afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.072+0000 7f5671f4c640 1 -- 172.21.15.116:0/3853960255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f566c15d860 msgr2=0x7f566c0cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.072+0000 7f5671f4c640 1 --2- 172.21.15.116:0/3853960255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f566c15d860 0x7f566c0cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.072+0000 7f5671f4c640 1 -- 172.21.15.116:0/3853960255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5664009a70 con 0x7f566c0d01e0 2023-11-28T17:17:03.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.073+0000 7f5671f4c640 1 --2- 172.21.15.116:0/3853960255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566c0d01e0 0x7f566c149410 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f566800c430 tx=0x7f566800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.073+0000 7f56627fc640 1 -- 172.21.15.116:0/3853960255 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668011790 con 0x7f566c0d01e0 2023-11-28T17:17:03.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.073+0000 7f56627fc640 1 -- 172.21.15.116:0/3853960255 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56680118f0 con 0x7f566c0d01e0 2023-11-28T17:17:03.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.073+0000 7f56741b0640 1 -- 172.21.15.116:0/3853960255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566c0c9e10 con 0x7f566c0d01e0 2023-11-28T17:17:03.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.073+0000 7f56627fc640 1 -- 172.21.15.116:0/3853960255 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668011ce0 con 0x7f566c0d01e0 2023-11-28T17:17:03.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.073+0000 7f56741b0640 1 -- 172.21.15.116:0/3853960255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566c146bb0 con 0x7f566c0d01e0 2023-11-28T17:17:03.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.074+0000 7f56741b0640 1 -- 172.21.15.116:0/3853960255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f566c067980 con 0x7f566c0d01e0 2023-11-28T17:17:03.078 INFO:teuthology.orchestra.run.smithi116.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-11-28T17:17:03.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.075+0000 7f56627fc640 1 -- 172.21.15.116:0/3853960255 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f566801c460 con 0x7f566c0d01e0 2023-11-28T17:17:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:17:03.080 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:17:03.080 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:17:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.076+0000 7f56627fc640 1 --2- 172.21.15.116:0/3853960255 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f565006b2d0 0x7f565006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.076+0000 7f56627fc640 1 -- 172.21.15.116:0/3853960255 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f5668089d40 con 0x7f566c0d01e0 2023-11-28T17:17:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.076+0000 7f567174b640 1 --2- 172.21.15.116:0/3853960255 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f565006b2d0 0x7f565006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.081 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.077+0000 7f567174b640 1 --2- 172.21.15.116:0/3853960255 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f565006b2d0 0x7f565006d790 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f566c1f69a0 tx=0x7f565c007a40 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.082 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.080+0000 7f56627fc640 1 -- 172.21.15.116:0/3853960255 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56680549d0 con 0x7f566c0d01e0 2023-11-28T17:17:03.142 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T17:17:03.143 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:17:03.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:17:03.143 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-11-28T17:17:03.198 INFO:teuthology.orchestra.run.smithi156.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-11-28T17:17:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:17:03.200 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:17:03.270 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.269+0000 7f56741b0640 1 -- 172.21.15.116:0/3853960255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f566c069360 con 0x7f566c0d01e0 2023-11-28T17:17:03.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.270+0000 7f56627fc640 1 -- 172.21.15.116:0/3853960255 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5668057ff0 con 0x7f566c0d01e0 2023-11-28T17:17:03.271 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:17:03.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.273+0000 7f56741b0640 1 -- 172.21.15.116:0/3853960255 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f565006b2d0 msgr2=0x7f565006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.273+0000 7f56741b0640 1 --2- 172.21.15.116:0/3853960255 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f565006b2d0 0x7f565006d790 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f566c1f69a0 tx=0x7f565c007a40 comp rx=0 tx=0).stop 2023-11-28T17:17:03.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.273+0000 7f56741b0640 1 -- 172.21.15.116:0/3853960255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566c0d01e0 msgr2=0x7f566c149410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.273+0000 7f56741b0640 1 --2- 172.21.15.116:0/3853960255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566c0d01e0 0x7f566c149410 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f566800c430 tx=0x7f566800c8f0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.273+0000 7f56741b0640 1 -- 172.21.15.116:0/3853960255 shutdown_connections 2023-11-28T17:17:03.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.273+0000 7f56741b0640 1 --2- 172.21.15.116:0/3853960255 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f565006b2d0 0x7f565006d790 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.273+0000 7f56741b0640 1 --2- 172.21.15.116:0/3853960255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f566c15d860 0x7f566c0cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.273+0000 7f56741b0640 1 --2- 172.21.15.116:0/3853960255 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f566c15d480 0x7f566c0afe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.273+0000 7f56741b0640 1 --2- 172.21.15.116:0/3853960255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566c0d01e0 0x7f566c149410 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.273+0000 7f56741b0640 1 -- 172.21.15.116:0/3853960255 >> 172.21.15.116:0/3853960255 conn(0x7f566c09c540 msgr2=0x7f566c0b6130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:03.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.273+0000 7f56741b0640 1 -- 172.21.15.116:0/3853960255 shutdown_connections 2023-11-28T17:17:03.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.273+0000 7f56741b0640 1 -- 172.21.15.116:0/3853960255 wait complete. 2023-11-28T17:17:03.291 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:17:03.405 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.404+0000 7fce43335640 1 Processor -- start 2023-11-28T17:17:03.406 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.404+0000 7fce43335640 1 -- start start 2023-11-28T17:17:03.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.406+0000 7fce43335640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce3c15d4d0 0x7fce3c0ce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.406+0000 7fce43335640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce3c0d09b0 con 0x7fce3c0d1970 2023-11-28T17:17:03.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.406+0000 7fce43335640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce3c0cee70 con 0x7fce3c0d0e00 2023-11-28T17:17:03.407 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.406+0000 7fce43335640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c0cf710 con 0x7fce3c15d4d0 2023-11-28T17:17:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.406+0000 7fce42b34640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fce3c0d1970 0x7fce3c0c71a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:32986/0 (socket says 172.21.15.116:32986) 2023-11-28T17:17:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.406+0000 7fce42b34640 1 -- 172.21.15.116:0/4095377030 learned_addr learned my addr 172.21.15.116:0/4095377030 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:17:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce42333640 1 --2- 172.21.15.116:0/4095377030 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce3c15d4d0 0x7fce3c0ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 -- 172.21.15.116:0/4095377030 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116321613 0 0) 0x7fce3c0d09b0 con 0x7fce3c0d1970 2023-11-28T17:17:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 -- 172.21.15.116:0/4095377030 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce28003520 con 0x7fce3c0d1970 2023-11-28T17:17:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 -- 172.21.15.116:0/4095377030 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3387609696 0 0) 0x7fce28003520 con 0x7fce3c0d1970 2023-11-28T17:17:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 -- 172.21.15.116:0/4095377030 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce3c15d4d0 msgr2=0x7fce3c0ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 --2- 172.21.15.116:0/4095377030 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce3c15d4d0 0x7fce3c0ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 -- 172.21.15.116:0/4095377030 >> v1:172.21.15.156:6789/0 conn(0x7fce3c0d0e00 legacy=0x7fce3c0d1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 -- 172.21.15.116:0/4095377030 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c0cf150 con 0x7fce3c0d1970 2023-11-28T17:17:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 -- 172.21.15.116:0/4095377030 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7fce340019f0 con 0x7fce3c0d1970 2023-11-28T17:17:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 -- 172.21.15.116:0/4095377030 >> v1:172.21.15.116:6789/0 conn(0x7fce3c0d1970 legacy=0x7fce3c0c71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 --2- 172.21.15.116:0/4095377030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce280039a0 0x7fce28003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 --2- 172.21.15.116:0/4095377030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce280047c0 0x7fce28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.411 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 --2- 172.21.15.116:0/4095377030 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce3c15d4d0 0x7fce28005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 -- 172.21.15.116:0/4095377030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c0cf710 con 0x7fce280039a0 2023-11-28T17:17:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 -- 172.21.15.116:0/4095377030 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2800ba30 con 0x7fce280047c0 2023-11-28T17:17:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.407+0000 7fce41331640 1 -- 172.21.15.116:0/4095377030 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2800bbf0 con 0x7fce3c15d4d0 2023-11-28T17:17:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.408+0000 7fce42333640 1 --2- 172.21.15.116:0/4095377030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce280039a0 0x7fce28003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.408+0000 7fce41b32640 1 --2- 172.21.15.116:0/4095377030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce280047c0 0x7fce28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.408+0000 7fce42333640 1 -- 172.21.15.116:0/4095377030 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce3c15d4d0 msgr2=0x7fce28005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.408+0000 7fce42333640 1 --2- 172.21.15.116:0/4095377030 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce3c15d4d0 0x7fce28005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.408+0000 7fce42333640 1 -- 172.21.15.116:0/4095377030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce280047c0 msgr2=0x7fce28004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:03.412 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.408+0000 7fce42333640 1 --2- 172.21.15.116:0/4095377030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce280047c0 0x7fce28004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.408+0000 7fce42333640 1 -- 172.21.15.116:0/4095377030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce28003520 con 0x7fce280039a0 2023-11-28T17:17:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.408+0000 7fce41b32640 1 --2- 172.21.15.116:0/4095377030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce280047c0 0x7fce28004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:17:03.414 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.409+0000 7fce42333640 1 --2- 172.21.15.116:0/4095377030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce280039a0 0x7fce28003dc0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fce38009a90 tx=0x7fce280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3e5627901cc4c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.409+0000 7fce41331640 1 -- 172.21.15.116:0/4095377030 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3802c6b0 con 0x7fce280039a0 2023-11-28T17:17:03.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.409+0000 7fce41331640 1 -- 172.21.15.116:0/4095377030 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3802c810 con 0x7fce280039a0 2023-11-28T17:17:03.415 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.409+0000 7fce41331640 1 -- 172.21.15.116:0/4095377030 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3802cba0 con 0x7fce280039a0 2023-11-28T17:17:03.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.410+0000 7fce43335640 1 -- 172.21.15.116:0/4095377030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce280039a0 msgr2=0x7fce28003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.410+0000 7fce43335640 1 --2- 172.21.15.116:0/4095377030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce280039a0 0x7fce28003dc0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fce38009a90 tx=0x7fce280057f0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.410+0000 7fce42b34640 1 -- 172.21.15.116:0/4095377030 reap_dead start 2023-11-28T17:17:03.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.410+0000 7fce43335640 1 -- 172.21.15.116:0/4095377030 shutdown_connections 2023-11-28T17:17:03.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.410+0000 7fce43335640 1 -- 172.21.15.116:0/4095377030 >> 172.21.15.116:0/4095377030 conn(0x7fce3c0d2f90 msgr2=0x7fce3c0d3370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:03.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.410+0000 7fce43335640 1 -- 172.21.15.116:0/4095377030 shutdown_connections 2023-11-28T17:17:03.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.410+0000 7fce43335640 1 -- 172.21.15.116:0/4095377030 wait complete. 2023-11-28T17:17:03.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.413+0000 7fce43335640 1 Processor -- start 2023-11-28T17:17:03.416 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.413+0000 7fce43335640 1 -- start start 2023-11-28T17:17:03.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.415+0000 7fce43335640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce280039a0 0x7fce3c14d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.415+0000 7fce43335640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce3c15ebe0 0x7fce3c0cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.415+0000 7fce43335640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce3c160240 0x7fce3c06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.415+0000 7fce43335640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c14d9a0 con 0x7fce280039a0 2023-11-28T17:17:03.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.415+0000 7fce43335640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c15f510 con 0x7fce3c15ebe0 2023-11-28T17:17:03.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.415+0000 7fce43335640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c15f180 con 0x7fce3c160240 2023-11-28T17:17:03.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.415+0000 7fce42333640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce280039a0 0x7fce3c14d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.415+0000 7fce42333640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce280039a0 0x7fce3c14d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53852/0 (socket says 172.21.15.116:53852) 2023-11-28T17:17:03.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.415+0000 7fce42333640 1 -- 172.21.15.116:0/420959182 learned_addr learned my addr 172.21.15.116:0/420959182 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:03.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.416+0000 7fce42333640 1 -- 172.21.15.116:0/420959182 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce3c160240 msgr2=0x7fce3c06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.416+0000 7fce42333640 1 --2- 172.21.15.116:0/420959182 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce3c160240 0x7fce3c06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.416+0000 7fce42333640 1 -- 172.21.15.116:0/420959182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce3c15ebe0 msgr2=0x7fce3c0cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.416+0000 7fce42333640 1 --2- 172.21.15.116:0/420959182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce3c15ebe0 0x7fce3c0cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.416+0000 7fce42333640 1 -- 172.21.15.116:0/420959182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c15f760 con 0x7fce280039a0 2023-11-28T17:17:03.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.416+0000 7fce42333640 1 --2- 172.21.15.116:0/420959182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce280039a0 0x7fce3c14d460 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fce3802c250 tx=0x7fce38004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.416+0000 7fce32ffd640 1 -- 172.21.15.116:0/420959182 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38003810 con 0x7fce280039a0 2023-11-28T17:17:03.419 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.416+0000 7fce32ffd640 1 -- 172.21.15.116:0/420959182 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce38003970 con 0x7fce280039a0 2023-11-28T17:17:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.416+0000 7fce32ffd640 1 -- 172.21.15.116:0/420959182 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38003d20 con 0x7fce280039a0 2023-11-28T17:17:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.416+0000 7fce43335640 1 -- 172.21.15.116:0/420959182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3c15fa40 con 0x7fce280039a0 2023-11-28T17:17:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.416+0000 7fce43335640 1 -- 172.21.15.116:0/420959182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3c0ccf60 con 0x7fce280039a0 2023-11-28T17:17:03.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.418+0000 7fce43335640 1 -- 172.21.15.116:0/420959182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3c04e960 con 0x7fce280039a0 2023-11-28T17:17:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.418+0000 7fce32ffd640 1 -- 172.21.15.116:0/420959182 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fce38040450 con 0x7fce280039a0 2023-11-28T17:17:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.419+0000 7fce32ffd640 1 --2- 172.21.15.116:0/420959182 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fce2006b2a0 0x7fce2006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.419+0000 7fce32ffd640 1 -- 172.21.15.116:0/420959182 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fce380ac0d0 con 0x7fce280039a0 2023-11-28T17:17:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.419+0000 7fce41b32640 1 --2- 172.21.15.116:0/420959182 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fce2006b2a0 0x7fce2006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.420+0000 7fce41b32640 1 --2- 172.21.15.116:0/420959182 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fce2006b2a0 0x7fce2006d760 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fce3c1f6c30 tx=0x7fce2c009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.423+0000 7fce32ffd640 1 -- 172.21.15.116:0/420959182 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce38076e80 con 0x7fce280039a0 2023-11-28T17:17:03.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.473+0000 7f892cdc9640 1 Processor -- start 2023-11-28T17:17:03.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.473+0000 7f892cdc9640 1 -- start start 2023-11-28T17:17:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.474+0000 7f892cdc9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8928157a20 0x7f892806a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.474+0000 7f892cdc9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f89280b4a30 0x7f8928160e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.474+0000 7f892cdc9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89280ad640 con 0x7f892806aa00 2023-11-28T17:17:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.474+0000 7f892cdc9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8928157e00 con 0x7f89280b4a30 2023-11-28T17:17:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.474+0000 7f892cdc9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892806af80 con 0x7f8928157a20 2023-11-28T17:17:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.474+0000 7f8926d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f89280b4a30 0x7f8928160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.475+0000 7f8926d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f89280b4a30 0x7f8928160e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53860/0 (socket says 172.21.15.116:53860) 2023-11-28T17:17:03.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.475+0000 7f8926d9d640 1 -- 172.21.15.116:0/2673036089 learned_addr learned my addr 172.21.15.116:0/2673036089 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.475+0000 7f8926d9d640 1 -- 172.21.15.116:0/2673036089 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8928157a20 msgr2=0x7f892806a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.475+0000 7f892659c640 1 --2- 172.21.15.116:0/2673036089 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8928157a20 0x7f892806a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.475+0000 7f8926d9d640 1 --2- 172.21.15.116:0/2673036089 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8928157a20 0x7f892806a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.475+0000 7f8926d9d640 1 -- 172.21.15.116:0/2673036089 >> v1:172.21.15.156:6789/0 conn(0x7f892806aa00 legacy=0x7f89280b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.475+0000 7f8926d9d640 1 -- 172.21.15.116:0/2673036089 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892815d990 con 0x7f89280b4a30 2023-11-28T17:17:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.475+0000 7f892659c640 1 --2- 172.21.15.116:0/2673036089 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8928157a20 0x7f892806a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:17:03.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.476+0000 7f8926d9d640 1 --2- 172.21.15.116:0/2673036089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89280b4a30 0x7f8928160e70 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8920009e50 tx=0x7f892002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=489dd1a03fe4c955 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.476+0000 7f892559a640 1 -- 172.21.15.116:0/2673036089 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892002ca70 con 0x7f89280b4a30 2023-11-28T17:17:03.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.476+0000 7f892559a640 1 -- 172.21.15.116:0/2673036089 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892002cbd0 con 0x7f89280b4a30 2023-11-28T17:17:03.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.476+0000 7f892559a640 1 -- 172.21.15.116:0/2673036089 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89200343f0 con 0x7f89280b4a30 2023-11-28T17:17:03.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.477+0000 7f892cdc9640 1 -- 172.21.15.116:0/2673036089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89280b4a30 msgr2=0x7f8928160e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.477+0000 7f892cdc9640 1 --2- 172.21.15.116:0/2673036089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89280b4a30 0x7f8928160e70 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8920009e50 tx=0x7f892002bed0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.477+0000 7f892cdc9640 1 -- 172.21.15.116:0/2673036089 shutdown_connections 2023-11-28T17:17:03.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.477+0000 7f892cdc9640 1 --2- 172.21.15.116:0/2673036089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89280b4a30 0x7f8928160e70 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.477+0000 7f892cdc9640 1 --2- 172.21.15.116:0/2673036089 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8928157a20 0x7f892806a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.477+0000 7f892cdc9640 1 -- 172.21.15.116:0/2673036089 >> 172.21.15.116:0/2673036089 conn(0x7f892806d4a0 msgr2=0x7f892806d880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:03.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.477+0000 7f892cdc9640 1 -- 172.21.15.116:0/2673036089 shutdown_connections 2023-11-28T17:17:03.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.477+0000 7f892cdc9640 1 -- 172.21.15.116:0/2673036089 wait complete. 2023-11-28T17:17:03.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.479+0000 7f892cdc9640 1 Processor -- start 2023-11-28T17:17:03.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.479+0000 7f892cdc9640 1 -- start start 2023-11-28T17:17:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.480+0000 7f892cdc9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f892806aa00 0x7f8928154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.480+0000 7f892cdc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89280b4a30 0x7f89280ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.480+0000 7f892cdc9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8928157a20 0x7f89280cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.480+0000 7f892cdc9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89280cd330 con 0x7f89280b4a30 2023-11-28T17:17:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.480+0000 7f892cdc9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89280cd610 con 0x7f892806aa00 2023-11-28T17:17:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.480+0000 7f892cdc9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89280cd8f0 con 0x7f8928157a20 2023-11-28T17:17:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.480+0000 7f8925d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89280b4a30 0x7f89280ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.480+0000 7f8925d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89280b4a30 0x7f89280ce3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53872/0 (socket says 172.21.15.116:53872) 2023-11-28T17:17:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.480+0000 7f8925d9b640 1 -- 172.21.15.116:0/1232708926 learned_addr learned my addr 172.21.15.116:0/1232708926 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.481+0000 7f8925d9b640 1 -- 172.21.15.116:0/1232708926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8928157a20 msgr2=0x7f89280cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.481+0000 7f8925d9b640 1 --2- 172.21.15.116:0/1232708926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8928157a20 0x7f89280cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.481+0000 7f8925d9b640 1 -- 172.21.15.116:0/1232708926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f892806aa00 msgr2=0x7f8928154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.481+0000 7f892659c640 1 --2- 172.21.15.116:0/1232708926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f892806aa00 0x7f8928154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.481+0000 7f8925d9b640 1 --2- 172.21.15.116:0/1232708926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f892806aa00 0x7f8928154f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.481+0000 7f8925d9b640 1 -- 172.21.15.116:0/1232708926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8920009b30 con 0x7f89280b4a30 2023-11-28T17:17:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.481+0000 7f892659c640 1 --2- 172.21.15.116:0/1232708926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f892806aa00 0x7f8928154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:17:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.481+0000 7f8925d9b640 1 --2- 172.21.15.116:0/1232708926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89280b4a30 0x7f89280ce3d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f891c00b3f0 tx=0x7f891c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.481+0000 7f8916ffd640 1 -- 172.21.15.116:0/1232708926 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c011830 con 0x7f89280b4a30 2023-11-28T17:17:03.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.481+0000 7f8916ffd640 1 -- 172.21.15.116:0/1232708926 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c011990 con 0x7f89280b4a30 2023-11-28T17:17:03.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.481+0000 7f8916ffd640 1 -- 172.21.15.116:0/1232708926 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c011d40 con 0x7f89280b4a30 2023-11-28T17:17:03.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.481+0000 7f892cdc9640 1 -- 172.21.15.116:0/1232708926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89280cdbd0 con 0x7f89280b4a30 2023-11-28T17:17:03.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.481+0000 7f892cdc9640 1 -- 172.21.15.116:0/1232708926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8928155480 con 0x7f89280b4a30 2023-11-28T17:17:03.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.482+0000 7f892cdc9640 1 -- 172.21.15.116:0/1232708926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8928067460 con 0x7f89280b4a30 2023-11-28T17:17:03.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.483+0000 7f8916ffd640 1 -- 172.21.15.116:0/1232708926 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f891c01a400 con 0x7f89280b4a30 2023-11-28T17:17:03.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.484+0000 7f8916ffd640 1 --2- 172.21.15.116:0/1232708926 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f890806b2a0 0x7f890806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.484+0000 7f8916ffd640 1 -- 172.21.15.116:0/1232708926 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f891c089b60 con 0x7f89280b4a30 2023-11-28T17:17:03.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.485+0000 7f892659c640 1 --2- 172.21.15.116:0/1232708926 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f890806b2a0 0x7f890806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.485+0000 7f892659c640 1 --2- 172.21.15.116:0/1232708926 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f890806b2a0 0x7f890806d760 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8918006fd0 tx=0x7f8918008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.493+0000 7f8916ffd640 1 -- 172.21.15.116:0/1232708926 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f891c054880 con 0x7f89280b4a30 2023-11-28T17:17:03.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.658+0000 7fce43335640 1 -- 172.21.15.116:0/420959182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce3c0c8620 con 0x7fce280039a0 2023-11-28T17:17:03.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.659+0000 7fce32ffd640 1 -- 172.21.15.116:0/420959182 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+3403 (secure 0 0 0) 0x7fce380044e0 con 0x7fce280039a0 2023-11-28T17:17:03.660 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:17:03.660 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":40,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:11:23.668388+0000","modified":"2023-11-28T17:15:32.267954+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5826,"mds_1":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"d","rank":1,"incarnation":33,"state":"up:active","state_seq":22,"addr":"172.21.15.156:6835/2474839888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2474839888},{"type":"v1","addr":"172.21.15.156:6835","nonce":2474839888}]},"join_fscid":-1,"laggy_since":"2023-11-28T17:12:46.046020+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5826":{"gid":5826,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":67,"addr":"172.21.15.116:6837/1482499383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1482499383},{"type":"v1","addr":"172.21.15.116:6837","nonce":1482499383}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T17:17:03.661 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 40 2023-11-28T17:17:03.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.661+0000 7fce43335640 1 -- 172.21.15.116:0/420959182 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fce2006b2a0 msgr2=0x7fce2006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.661+0000 7fce43335640 1 --2- 172.21.15.116:0/420959182 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fce2006b2a0 0x7fce2006d760 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fce3c1f6c30 tx=0x7fce2c009040 comp rx=0 tx=0).stop 2023-11-28T17:17:03.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.661+0000 7fce43335640 1 -- 172.21.15.116:0/420959182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce280039a0 msgr2=0x7fce3c14d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.661+0000 7fce43335640 1 --2- 172.21.15.116:0/420959182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce280039a0 0x7fce3c14d460 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fce3802c250 tx=0x7fce38004730 comp rx=0 tx=0).stop 2023-11-28T17:17:03.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.661+0000 7fce43335640 1 -- 172.21.15.116:0/420959182 shutdown_connections 2023-11-28T17:17:03.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.661+0000 7fce43335640 1 --2- 172.21.15.116:0/420959182 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fce2006b2a0 0x7fce2006d760 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.661+0000 7fce43335640 1 --2- 172.21.15.116:0/420959182 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fce3c160240 0x7fce3c06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.661+0000 7fce43335640 1 --2- 172.21.15.116:0/420959182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fce3c15ebe0 0x7fce3c0cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.661+0000 7fce43335640 1 --2- 172.21.15.116:0/420959182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce280039a0 0x7fce3c14d460 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.661+0000 7fce43335640 1 -- 172.21.15.116:0/420959182 >> 172.21.15.116:0/420959182 conn(0x7fce3c0d2f90 msgr2=0x7fce3c06c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:03.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.661+0000 7fce43335640 1 -- 172.21.15.116:0/420959182 shutdown_connections 2023-11-28T17:17:03.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.661+0000 7fce43335640 1 -- 172.21.15.116:0/420959182 wait complete. 2023-11-28T17:17:03.682 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:17:03.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.681+0000 7f892cdc9640 1 -- 172.21.15.116:0/1232708926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f892806fbc0 con 0x7f89280b4a30 2023-11-28T17:17:03.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.682+0000 7f8916ffd640 1 -- 172.21.15.116:0/1232708926 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) v1 ==== 104+0+11 (secure 0 0 0) 0x7f891c057ea0 con 0x7f89280b4a30 2023-11-28T17:17:03.684 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:17:03.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.684+0000 7f892cdc9640 1 -- 172.21.15.116:0/1232708926 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f890806b2a0 msgr2=0x7f890806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.684+0000 7f892cdc9640 1 --2- 172.21.15.116:0/1232708926 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f890806b2a0 0x7f890806d760 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8918006fd0 tx=0x7f8918008040 comp rx=0 tx=0).stop 2023-11-28T17:17:03.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.684+0000 7f892cdc9640 1 -- 172.21.15.116:0/1232708926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89280b4a30 msgr2=0x7f89280ce3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.684+0000 7f892cdc9640 1 --2- 172.21.15.116:0/1232708926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89280b4a30 0x7f89280ce3d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f891c00b3f0 tx=0x7f891c00b8b0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.684+0000 7f892cdc9640 1 -- 172.21.15.116:0/1232708926 shutdown_connections 2023-11-28T17:17:03.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.684+0000 7f892cdc9640 1 --2- 172.21.15.116:0/1232708926 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f890806b2a0 0x7f890806d760 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.684+0000 7f892cdc9640 1 --2- 172.21.15.116:0/1232708926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8928157a20 0x7f89280cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.685+0000 7f892cdc9640 1 --2- 172.21.15.116:0/1232708926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89280b4a30 0x7f89280ce3d0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.685+0000 7f892cdc9640 1 --2- 172.21.15.116:0/1232708926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f892806aa00 0x7f8928154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.685+0000 7f892cdc9640 1 -- 172.21.15.116:0/1232708926 >> 172.21.15.116:0/1232708926 conn(0x7f892806d4a0 msgr2=0x7f89280ad820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:03.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.685+0000 7f892cdc9640 1 -- 172.21.15.116:0/1232708926 shutdown_connections 2023-11-28T17:17:03.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.685+0000 7f892cdc9640 1 -- 172.21.15.116:0/1232708926 wait complete. 2023-11-28T17:17:03.703 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:17:03.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.872+0000 7ff64f6b4640 1 Processor -- start 2023-11-28T17:17:03.873 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.872+0000 7ff64f6b4640 1 -- start start 2023-11-28T17:17:03.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.874+0000 7ff64f6b4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6480d1780 0x7ff6480d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.874+0000 7ff64f6b4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6480cdcb0 con 0x7ff6480b2ad0 2023-11-28T17:17:03.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.874+0000 7ff64f6b4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6480d39e0 con 0x7ff6480b20a0 2023-11-28T17:17:03.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.874+0000 7ff64f6b4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6480d3cc0 con 0x7ff6480d1780 2023-11-28T17:17:03.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.874+0000 7ff64dc51640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff6480b2ad0 0x7ff6480ce750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:32994/0 (socket says 172.21.15.116:32994) 2023-11-28T17:17:03.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.874+0000 7ff64dc51640 1 -- 172.21.15.116:0/2490756863 learned_addr learned my addr 172.21.15.116:0/2490756863 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:17:03.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff64d450640 1 --2- 172.21.15.116:0/2490756863 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6480d1780 0x7ff6480d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 -- 172.21.15.116:0/2490756863 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408614115 0 0) 0x7ff6480cdcb0 con 0x7ff6480b2ad0 2023-11-28T17:17:03.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 -- 172.21.15.116:0/2490756863 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff634003520 con 0x7ff6480b2ad0 2023-11-28T17:17:03.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 -- 172.21.15.116:0/2490756863 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1659073403 0 0) 0x7ff634003520 con 0x7ff6480b2ad0 2023-11-28T17:17:03.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 -- 172.21.15.116:0/2490756863 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6480d1780 msgr2=0x7ff6480d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:03.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 --2- 172.21.15.116:0/2490756863 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6480d1780 0x7ff6480d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 -- 172.21.15.116:0/2490756863 >> v1:172.21.15.156:6789/0 conn(0x7ff6480b20a0 legacy=0x7ff6480d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 -- 172.21.15.116:0/2490756863 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6480cdf50 con 0x7ff6480b2ad0 2023-11-28T17:17:03.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 -- 172.21.15.116:0/2490756863 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7ff6400019f0 con 0x7ff6480b2ad0 2023-11-28T17:17:03.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 -- 172.21.15.116:0/2490756863 >> v1:172.21.15.116:6789/0 conn(0x7ff6480b2ad0 legacy=0x7ff6480ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 --2- 172.21.15.116:0/2490756863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6340039a0 0x7ff634003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 --2- 172.21.15.116:0/2490756863 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6480d1780 0x7ff6340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 --2- 172.21.15.116:0/2490756863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634004d00 0x7ff634005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 -- 172.21.15.116:0/2490756863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6480d3cc0 con 0x7ff634004d00 2023-11-28T17:17:03.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 -- 172.21.15.116:0/2490756863 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63400bab0 con 0x7ff6340039a0 2023-11-28T17:17:03.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.875+0000 7ff63ffff640 1 -- 172.21.15.116:0/2490756863 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63400bc70 con 0x7ff6480d1780 2023-11-28T17:17:03.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.876+0000 7ff64dc51640 1 --2- 172.21.15.116:0/2490756863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634004d00 0x7ff634005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.876+0000 7ff64d450640 1 --2- 172.21.15.116:0/2490756863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6340039a0 0x7ff634003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.876+0000 7ff64cc4f640 1 --2- 172.21.15.116:0/2490756863 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6480d1780 0x7ff6340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.876+0000 7ff64dc51640 1 -- 172.21.15.116:0/2490756863 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6480d1780 msgr2=0x7ff6340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:03.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.876+0000 7ff64dc51640 1 --2- 172.21.15.116:0/2490756863 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6480d1780 0x7ff6340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.876+0000 7ff64dc51640 1 -- 172.21.15.116:0/2490756863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6340039a0 msgr2=0x7ff634003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:03.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.876+0000 7ff64dc51640 1 --2- 172.21.15.116:0/2490756863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6340039a0 0x7ff634003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.876+0000 7ff64dc51640 1 -- 172.21.15.116:0/2490756863 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff634003520 con 0x7ff634004d00 2023-11-28T17:17:03.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.876+0000 7ff64d450640 1 --2- 172.21.15.116:0/2490756863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6340039a0 0x7ff634003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:17:03.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.876+0000 7ff64cc4f640 1 --2- 172.21.15.116:0/2490756863 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6480d1780 0x7ff6340047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:17:03.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.877+0000 7ff64dc51640 1 --2- 172.21.15.116:0/2490756863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634004d00 0x7ff634005180 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff64000a830 tx=0x7ff6340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea31089ac9c88b1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.877+0000 7ff63ffff640 1 -- 172.21.15.116:0/2490756863 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6400072d0 con 0x7ff634004d00 2023-11-28T17:17:03.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.877+0000 7ff63ffff640 1 -- 172.21.15.116:0/2490756863 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64002f070 con 0x7ff634004d00 2023-11-28T17:17:03.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.877+0000 7ff63ffff640 1 -- 172.21.15.116:0/2490756863 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff640007730 con 0x7ff634004d00 2023-11-28T17:17:03.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.878+0000 7ff64f6b4640 1 -- 172.21.15.116:0/2490756863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634004d00 msgr2=0x7ff634005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.878+0000 7ff64f6b4640 1 --2- 172.21.15.116:0/2490756863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff634004d00 0x7ff634005180 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff64000a830 tx=0x7ff6340057f0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.878+0000 7ff64dc51640 1 -- 172.21.15.116:0/2490756863 reap_dead start 2023-11-28T17:17:03.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.878+0000 7ff64f6b4640 1 -- 172.21.15.116:0/2490756863 shutdown_connections 2023-11-28T17:17:03.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.878+0000 7ff64f6b4640 1 -- 172.21.15.116:0/2490756863 >> 172.21.15.116:0/2490756863 conn(0x7ff64809c4e0 msgr2=0x7ff6480d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:03.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.878+0000 7ff64f6b4640 1 -- 172.21.15.116:0/2490756863 shutdown_connections 2023-11-28T17:17:03.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.878+0000 7ff64f6b4640 1 -- 172.21.15.116:0/2490756863 wait complete. 2023-11-28T17:17:03.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.880+0000 7ff64f6b4640 1 Processor -- start 2023-11-28T17:17:03.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.880+0000 7ff64f6b4640 1 -- start start 2023-11-28T17:17:03.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.881+0000 7ff64f6b4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff634004d00 0x7ff6481f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.881+0000 7ff64f6b4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6481f69a0 0x7ff6480c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.881+0000 7ff64f6b4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6480c7ce0 0x7ff6481f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.881+0000 7ff64f6b4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6481f6e20 con 0x7ff6481f69a0 2023-11-28T17:17:03.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.881+0000 7ff64f6b4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6481f4030 con 0x7ff634004d00 2023-11-28T17:17:03.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.881+0000 7ff64f6b4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6480c8500 con 0x7ff6480c7ce0 2023-11-28T17:17:03.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.881+0000 7ff64cc4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6481f69a0 0x7ff6480c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.881+0000 7ff64cc4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6481f69a0 0x7ff6480c77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53880/0 (socket says 172.21.15.116:53880) 2023-11-28T17:17:03.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.881+0000 7ff64cc4f640 1 -- 172.21.15.116:0/4052130031 learned_addr learned my addr 172.21.15.116:0/4052130031 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:03.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.882+0000 7ff64cc4f640 1 -- 172.21.15.116:0/4052130031 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6480c7ce0 msgr2=0x7ff6481f47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.882+0000 7ff64dc51640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6480c7ce0 0x7ff6481f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.882+0000 7ff64d450640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff634004d00 0x7ff6481f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.882+0000 7ff64cc4f640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6480c7ce0 0x7ff6481f47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.882+0000 7ff64cc4f640 1 -- 172.21.15.116:0/4052130031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff634004d00 msgr2=0x7ff6481f6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.882+0000 7ff64cc4f640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff634004d00 0x7ff6481f6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.882+0000 7ff64cc4f640 1 -- 172.21.15.116:0/4052130031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6480c87e0 con 0x7ff6481f69a0 2023-11-28T17:17:03.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.882+0000 7ff64d450640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff634004d00 0x7ff6481f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:17:03.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.882+0000 7ff64cc4f640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6481f69a0 0x7ff6480c77a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7ff63800d7a0 tx=0x7ff63800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.882+0000 7ff63dffb640 1 -- 172.21.15.116:0/4052130031 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff638017070 con 0x7ff6481f69a0 2023-11-28T17:17:03.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.882+0000 7ff63dffb640 1 -- 172.21.15.116:0/4052130031 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff638005070 con 0x7ff6481f69a0 2023-11-28T17:17:03.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.882+0000 7ff64f6b4640 1 -- 172.21.15.116:0/4052130031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6480c8ac0 con 0x7ff6481f69a0 2023-11-28T17:17:03.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.882+0000 7ff63dffb640 1 -- 172.21.15.116:0/4052130031 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff638005420 con 0x7ff6481f69a0 2023-11-28T17:17:03.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.882+0000 7ff64f6b4640 1 -- 172.21.15.116:0/4052130031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6480c8160 con 0x7ff6481f69a0 2023-11-28T17:17:03.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.884+0000 7ff64f6b4640 1 -- 172.21.15.116:0/4052130031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6480c8d00 con 0x7ff6481f69a0 2023-11-28T17:17:03.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.885+0000 7ff63dffb640 1 -- 172.21.15.116:0/4052130031 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff638007030 con 0x7ff6481f69a0 2023-11-28T17:17:03.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.887+0000 7ff63dffb640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff62c06b2d0 0x7ff62c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.887+0000 7ff63dffb640 1 -- 172.21.15.116:0/4052130031 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7ff638089600 con 0x7ff6481f69a0 2023-11-28T17:17:03.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.887+0000 7ff64d450640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff62c06b2d0 0x7ff62c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.888+0000 7ff64d450640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff62c06b2d0 0x7ff62c06d790 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff644009aa0 tx=0x7ff644008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.891+0000 7ff63dffb640 1 -- 172.21.15.116:0/4052130031 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff638054330 con 0x7ff6481f69a0 2023-11-28T17:17:03.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.926+0000 7f8dbf59e640 1 Processor -- start 2023-11-28T17:17:03.928 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.927+0000 7f8dbf59e640 1 -- start start 2023-11-28T17:17:03.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.928+0000 7f8dbf59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8dc006ede0 0x7f8dc00cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.928+0000 7f8dbf59e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8dc006f240 0x7f8dc00cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.929 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.928+0000 7f8dbf59e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc00cfa90 con 0x7f8dc015d4e0 2023-11-28T17:17:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.928+0000 7f8dbf59e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc015d8c0 con 0x7f8dc006f240 2023-11-28T17:17:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.928+0000 7f8dbf59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc00cb7e0 con 0x7f8dc006ede0 2023-11-28T17:17:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.928+0000 7f8dbe59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8dc015d4e0 0x7f8dc006e6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33004/0 (socket says 172.21.15.116:33004) 2023-11-28T17:17:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.928+0000 7f8dbe59c640 1 -- 172.21.15.116:0/802587964 learned_addr learned my addr 172.21.15.116:0/802587964 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:17:03.931 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.928+0000 7f8db7fff640 1 --2- 172.21.15.116:0/802587964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8dc006ede0 0x7f8dc00cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.928+0000 7f8dbdd9b640 1 -- 172.21.15.116:0/802587964 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964701271 0 0) 0x7f8dc00cfa90 con 0x7f8dc015d4e0 2023-11-28T17:17:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 -- 172.21.15.116:0/802587964 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dac003520 con 0x7f8dc015d4e0 2023-11-28T17:17:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbed9d640 1 --2- 172.21.15.116:0/802587964 >> v2:172.21.15.156:3300/0 conn(0x7f8dc006f240 0x7f8dc00cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 -- 172.21.15.116:0/802587964 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3384017692 0 0) 0x7f8dac003520 con 0x7f8dc015d4e0 2023-11-28T17:17:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 -- 172.21.15.116:0/802587964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8dc006ede0 msgr2=0x7f8dc00cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 --2- 172.21.15.116:0/802587964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8dc006ede0 0x7f8dc00cbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 -- 172.21.15.116:0/802587964 >> v2:172.21.15.156:3300/0 conn(0x7f8dc006f240 msgr2=0x7f8dc00cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 --2- 172.21.15.116:0/802587964 >> v2:172.21.15.156:3300/0 conn(0x7f8dc006f240 0x7f8dc00cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 -- 172.21.15.116:0/802587964 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc00cb4e0 con 0x7f8dc015d4e0 2023-11-28T17:17:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 -- 172.21.15.116:0/802587964 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7f8db0001670 con 0x7f8dc015d4e0 2023-11-28T17:17:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 -- 172.21.15.116:0/802587964 >> v1:172.21.15.116:6789/0 conn(0x7f8dc015d4e0 legacy=0x7f8dc006e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 --2- 172.21.15.116:0/802587964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8dc006ede0 0x7f8dac003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 --2- 172.21.15.116:0/802587964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dac003f40 0x7f8dac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 --2- 172.21.15.116:0/802587964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dac004d00 0x7f8dac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 -- 172.21.15.116:0/802587964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc015d8c0 con 0x7f8dac003f40 2023-11-28T17:17:03.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 -- 172.21.15.116:0/802587964 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc00cb7e0 con 0x7f8dac004d00 2023-11-28T17:17:03.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8dbdd9b640 1 -- 172.21.15.116:0/802587964 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dac00ba30 con 0x7f8dc006ede0 2023-11-28T17:17:03.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.929+0000 7f8db7fff640 1 --2- 172.21.15.116:0/802587964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dac003f40 0x7f8dac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.930+0000 7f8db7fff640 1 -- 172.21.15.116:0/802587964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8dc006ede0 msgr2=0x7f8dac003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:03.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.930+0000 7f8db7fff640 1 --2- 172.21.15.116:0/802587964 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8dc006ede0 0x7f8dac003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.930+0000 7f8db7fff640 1 -- 172.21.15.116:0/802587964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dac004d00 msgr2=0x7f8dac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:03.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.930+0000 7f8dbed9d640 1 --2- 172.21.15.116:0/802587964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dac004d00 0x7f8dac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.933 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.930+0000 7f8db7fff640 1 --2- 172.21.15.116:0/802587964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dac004d00 0x7f8dac005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.934 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.930+0000 7f8db7fff640 1 -- 172.21.15.116:0/802587964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dac003520 con 0x7f8dac003f40 2023-11-28T17:17:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.931+0000 7f8db7fff640 1 --2- 172.21.15.116:0/802587964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dac003f40 0x7f8dac0047c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009ac0 tx=0x7f8dac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf617a60d3dcb76 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.931+0000 7f8dbdd9b640 1 -- 172.21.15.116:0/802587964 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da802c710 con 0x7f8dac003f40 2023-11-28T17:17:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.931+0000 7f8dbdd9b640 1 -- 172.21.15.116:0/802587964 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da802c870 con 0x7f8dac003f40 2023-11-28T17:17:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.931+0000 7f8dbdd9b640 1 -- 172.21.15.116:0/802587964 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da802cc00 con 0x7f8dac003f40 2023-11-28T17:17:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.932+0000 7f8dbf59e640 1 -- 172.21.15.116:0/802587964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dac003f40 msgr2=0x7f8dac0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.932+0000 7f8dbf59e640 1 --2- 172.21.15.116:0/802587964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dac003f40 0x7f8dac0047c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009ac0 tx=0x7f8dac0057f0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.932+0000 7f8dbed9d640 1 -- 172.21.15.116:0/802587964 reap_dead start 2023-11-28T17:17:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.932+0000 7f8dbf59e640 1 -- 172.21.15.116:0/802587964 shutdown_connections 2023-11-28T17:17:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.932+0000 7f8dbf59e640 1 -- 172.21.15.116:0/802587964 >> 172.21.15.116:0/802587964 conn(0x7f8dc00b2900 msgr2=0x7f8dc00b2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.932+0000 7f8dbf59e640 1 -- 172.21.15.116:0/802587964 shutdown_connections 2023-11-28T17:17:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.932+0000 7f8dbf59e640 1 -- 172.21.15.116:0/802587964 wait complete. 2023-11-28T17:17:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.934+0000 7f8dbf59e640 1 Processor -- start 2023-11-28T17:17:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.934+0000 7f8dbf59e640 1 -- start start 2023-11-28T17:17:03.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8dbf59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8dac003f40 0x7f8dc0155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.938 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8dbf59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dc00c5480 0x7f8dc00b5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8dbf59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dc01562f0 0x7f8dc00b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8dbf59e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc0156160 con 0x7f8dc01562f0 2023-11-28T17:17:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8dbf59e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc00b5aa0 con 0x7f8dc00c5480 2023-11-28T17:17:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8dbf59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc00b5d20 con 0x7f8dac003f40 2023-11-28T17:17:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8dbed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dc01562f0 0x7f8dc00b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8dbed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dc01562f0 0x7f8dc00b1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53898/0 (socket says 172.21.15.116:53898) 2023-11-28T17:17:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8dbed9d640 1 -- 172.21.15.116:0/1324707688 learned_addr learned my addr 172.21.15.116:0/1324707688 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8dbed9d640 1 -- 172.21.15.116:0/1324707688 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8dac003f40 msgr2=0x7f8dc0155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8dbed9d640 1 --2- 172.21.15.116:0/1324707688 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8dac003f40 0x7f8dc0155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8dbed9d640 1 -- 172.21.15.116:0/1324707688 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dc00c5480 msgr2=0x7f8dc00b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8db7fff640 1 --2- 172.21.15.116:0/1324707688 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dc00c5480 0x7f8dc00b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8dbed9d640 1 --2- 172.21.15.116:0/1324707688 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dc00c5480 0x7f8dc00b5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.935+0000 7f8dbed9d640 1 -- 172.21.15.116:0/1324707688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc00c49a0 con 0x7f8dc01562f0 2023-11-28T17:17:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.936+0000 7f8dbed9d640 1 --2- 172.21.15.116:0/1324707688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dc01562f0 0x7f8dc00b1bb0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f8dc006e270 tx=0x7f8db800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.936+0000 7f8db6ffd640 1 -- 172.21.15.116:0/1324707688 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8016620 con 0x7f8dc01562f0 2023-11-28T17:17:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.936+0000 7f8db6ffd640 1 -- 172.21.15.116:0/1324707688 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db80167a0 con 0x7f8dc01562f0 2023-11-28T17:17:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.936+0000 7f8db6ffd640 1 -- 172.21.15.116:0/1324707688 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8016b50 con 0x7f8dc01562f0 2023-11-28T17:17:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.936+0000 7f8dbf59e640 1 -- 172.21.15.116:0/1324707688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc00c4c80 con 0x7f8dc01562f0 2023-11-28T17:17:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.936+0000 7f8dbf59e640 1 -- 172.21.15.116:0/1324707688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc00b6000 con 0x7f8dc01562f0 2023-11-28T17:17:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.937+0000 7f8dbf59e640 1 -- 172.21.15.116:0/1324707688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc00c51a0 con 0x7f8dc01562f0 2023-11-28T17:17:03.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.938+0000 7f8db6ffd640 1 -- 172.21.15.116:0/1324707688 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f8db8016cb0 con 0x7f8dc01562f0 2023-11-28T17:17:03.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.939+0000 7f8db6ffd640 1 --2- 172.21.15.116:0/1324707688 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8da006b2a0 0x7f8da006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:03.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.939+0000 7f8db6ffd640 1 -- 172.21.15.116:0/1324707688 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f8db808ace0 con 0x7f8dc01562f0 2023-11-28T17:17:03.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.939+0000 7f8dbe59c640 1 --2- 172.21.15.116:0/1324707688 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8da006b2a0 0x7f8da006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:03.941 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.940+0000 7f8dbe59c640 1 --2- 172.21.15.116:0/1324707688 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8da006b2a0 0x7f8da006d760 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8db0001e30 tx=0x7f8db0001430 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:03.944 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:03.943+0000 7f8db6ffd640 1 -- 172.21.15.116:0/1324707688 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8dc00c51a0 con 0x7f8dc01562f0 2023-11-28T17:17:04.088 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.087+0000 7ff64f6b4640 1 -- 172.21.15.116:0/4052130031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff648159f30 con 0x7ff6481f69a0 2023-11-28T17:17:04.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.090+0000 7ff63dffb640 1 -- 172.21.15.116:0/4052130031 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+17073 (secure 0 0 0) 0x7ff638057950 con 0x7ff6481f69a0 2023-11-28T17:17:04.091 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:17:04.092 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":52,"fsid":"a5f2fbcf-fde4-4c38-94da-08178784d9fc","created":"2023-11-28T17:08:36.196614+0000","modified":"2023-11-28T17:12:26.038218+0000","last_up_change":"2023-11-28T17:08:48.963527+0000","last_in_change":"2023-11-28T17:08:40.044485+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T17:08:49.106357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-11-28T17:11:21.518078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-11-28T17:11:22.672451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6991e794-5137-4365-be74-be350afa42e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6801","nonce":671929056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6803","nonce":671929056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6807","nonce":671929056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":671929056},{"type":"v1","addr":"172.21.15.116:6805","nonce":671929056}]},"public_addr":"172.21.15.116:6801/671929056","cluster_addr":"172.21.15.116:6803/671929056","heartbeat_back_addr":"172.21.15.116:6807/671929056","heartbeat_front_addr":"172.21.15.116:6805/671929056","state":["exists","up"]},{"osd":1,"uuid":"e1e544b0-64e0-472e-a4d3-9ded5287a676","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6825","nonce":335787082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6827","nonce":335787082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6831","nonce":335787082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":335787082},{"type":"v1","addr":"172.21.15.116:6829","nonce":335787082}]},"public_addr":"172.21.15.116:6825/335787082","cluster_addr":"172.21.15.116:6827/335787082","heartbeat_back_addr":"172.21.15.116:6831/335787082","heartbeat_front_addr":"172.21.15.116:6829/335787082","state":["exists","up"]},{"osd":2,"uuid":"1e3bb3de-7bb3-4888-a95b-6a22e4805344","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6809","nonce":1361195238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6812","nonce":1361195238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6818","nonce":1361195238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":1361195238},{"type":"v1","addr":"172.21.15.116:6815","nonce":1361195238}]},"public_addr":"172.21.15.116:6809/1361195238","cluster_addr":"172.21.15.116:6812/1361195238","heartbeat_back_addr":"172.21.15.116:6818/1361195238","heartbeat_front_addr":"172.21.15.116:6815/1361195238","state":["exists","up"]},{"osd":3,"uuid":"9c345281-1fc7-42ad-9732-1585e687e9e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6811","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6814","nonce":3907008760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6819","nonce":3907008760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6823","nonce":3907008760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":3907008760},{"type":"v1","addr":"172.21.15.116:6821","nonce":3907008760}]},"public_addr":"172.21.15.116:6814/3907008760","cluster_addr":"172.21.15.116:6819/3907008760","heartbeat_back_addr":"172.21.15.116:6823/3907008760","heartbeat_front_addr":"172.21.15.116:6821/3907008760","state":["exists","up"]},{"osd":4,"uuid":"c815b7c4-e49f-4a53-a514-6546fffbe041","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6801","nonce":467190845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6803","nonce":467190845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6807","nonce":467190845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":467190845},{"type":"v1","addr":"172.21.15.156:6805","nonce":467190845}]},"public_addr":"172.21.15.156:6801/467190845","cluster_addr":"172.21.15.156:6803/467190845","heartbeat_back_addr":"172.21.15.156:6807/467190845","heartbeat_front_addr":"172.21.15.156:6805/467190845","state":["exists","up"]},{"osd":5,"uuid":"5d573d94-b928-49c4-9c7a-792052e375da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6809","nonce":526051277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6811","nonce":526051277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6816","nonce":526051277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":526051277},{"type":"v1","addr":"172.21.15.156:6813","nonce":526051277}]},"public_addr":"172.21.15.156:6809/526051277","cluster_addr":"172.21.15.156:6811/526051277","heartbeat_back_addr":"172.21.15.156:6816/526051277","heartbeat_front_addr":"172.21.15.156:6813/526051277","state":["exists","up"]},{"osd":6,"uuid":"85dd8756-df1c-41fa-b542-d339b6e59925","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6825","nonce":4082913363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6827","nonce":4082913363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6831","nonce":4082913363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":4082913363},{"type":"v1","addr":"172.21.15.156:6829","nonce":4082913363}]},"public_addr":"172.21.15.156:6825/4082913363","cluster_addr":"172.21.15.156:6827/4082913363","heartbeat_back_addr":"172.21.15.156:6831/4082913363","heartbeat_front_addr":"172.21.15.156:6829/4082913363","state":["exists","up"]},{"osd":7,"uuid":"cda8fbf8-de75-4da7-a71b-c6e90e3f568b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6815","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6817","nonce":1619141236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6819","nonce":1619141236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6823","nonce":1619141236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":1619141236},{"type":"v1","addr":"172.21.15.156:6821","nonce":1619141236}]},"public_addr":"172.21.15.156:6817/1619141236","cluster_addr":"172.21.15.156:6819/1619141236","heartbeat_back_addr":"172.21.15.156:6823/1619141236","heartbeat_front_addr":"172.21.15.156:6821/1619141236","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.354315+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.352651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.409211+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.408053+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.346450+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.596449+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.359962+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T17:08:45.416394+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.3","mappings":[{"from":5,"to":0}]},{"pgid":"7.8","mappings":[{"from":5,"to":7}]},{"pgid":"7.1b","mappings":[{"from":5,"to":7}]},{"pgid":"7.30","mappings":[{"from":5,"to":7}]},{"pgid":"7.31","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6833/372407661":"2023-11-29T17:12:26.038071+0000","172.21.15.156:6832/372407661":"2023-11-29T17:12:26.038071+0000","172.21.15.116:6835/3251347229":"2023-11-29T17:12:06.030424+0000","172.21.15.116:6834/3251347229":"2023-11-29T17:12:06.030424+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T17:17:04.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.092+0000 7ff64f6b4640 1 -- 172.21.15.116:0/4052130031 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff62c06b2d0 msgr2=0x7ff62c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.092+0000 7ff64f6b4640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff62c06b2d0 0x7ff62c06d790 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff644009aa0 tx=0x7ff644008040 comp rx=0 tx=0).stop 2023-11-28T17:17:04.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.092+0000 7ff64f6b4640 1 -- 172.21.15.116:0/4052130031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6481f69a0 msgr2=0x7ff6480c77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.092+0000 7ff64f6b4640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6481f69a0 0x7ff6480c77a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7ff63800d7a0 tx=0x7ff63800dc60 comp rx=0 tx=0).stop 2023-11-28T17:17:04.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.093+0000 7ff64f6b4640 1 -- 172.21.15.116:0/4052130031 shutdown_connections 2023-11-28T17:17:04.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.093+0000 7ff64f6b4640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff62c06b2d0 0x7ff62c06d790 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.094 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.093+0000 7ff64f6b4640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6480c7ce0 0x7ff6481f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.093+0000 7ff64f6b4640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6481f69a0 0x7ff6480c77a0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.093+0000 7ff64f6b4640 1 --2- 172.21.15.116:0/4052130031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff634004d00 0x7ff6481f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.093+0000 7ff64f6b4640 1 -- 172.21.15.116:0/4052130031 >> 172.21.15.116:0/4052130031 conn(0x7ff64809c4e0 msgr2=0x7ff64806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:04.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.093+0000 7ff64f6b4640 1 -- 172.21.15.116:0/4052130031 shutdown_connections 2023-11-28T17:17:04.095 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.093+0000 7ff64f6b4640 1 -- 172.21.15.116:0/4052130031 wait complete. 2023-11-28T17:17:04.112 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T17:08:49.106357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-28T17:11:21.518078+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-11-28T17:11:22.672451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T17:17:04.113 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-28T17:17:04.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.129+0000 7f8dbf59e640 1 -- 172.21.15.116:0/1324707688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8dc00c51a0 con 0x7f8dc01562f0 2023-11-28T17:17:04.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.129+0000 7f8db6ffd640 1 -- 172.21.15.116:0/1324707688 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8dc00c51a0 con 0x7f8dc01562f0 2023-11-28T17:17:04.130 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:17:04.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.131+0000 7f8dbf59e640 1 -- 172.21.15.116:0/1324707688 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8da006b2a0 msgr2=0x7f8da006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.131+0000 7f8dbf59e640 1 --2- 172.21.15.116:0/1324707688 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8da006b2a0 0x7f8da006d760 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8db0001e30 tx=0x7f8db0001430 comp rx=0 tx=0).stop 2023-11-28T17:17:04.133 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.132+0000 7f8dbf59e640 1 -- 172.21.15.116:0/1324707688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dc01562f0 msgr2=0x7f8dc00b1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.132+0000 7f8dbf59e640 1 --2- 172.21.15.116:0/1324707688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dc01562f0 0x7f8dc00b1bb0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f8dc006e270 tx=0x7f8db800f520 comp rx=0 tx=0).stop 2023-11-28T17:17:04.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.132+0000 7f8dbf59e640 1 -- 172.21.15.116:0/1324707688 shutdown_connections 2023-11-28T17:17:04.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.132+0000 7f8dbf59e640 1 --2- 172.21.15.116:0/1324707688 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f8da006b2a0 0x7f8da006d760 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.132+0000 7f8dbf59e640 1 --2- 172.21.15.116:0/1324707688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8dc01562f0 0x7f8dc00b1bb0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.132+0000 7f8dbf59e640 1 --2- 172.21.15.116:0/1324707688 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dc00c5480 0x7f8dc00b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.134 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.132+0000 7f8dbf59e640 1 --2- 172.21.15.116:0/1324707688 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8dac003f40 0x7f8dc0155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.132+0000 7f8dbf59e640 1 -- 172.21.15.116:0/1324707688 >> 172.21.15.116:0/1324707688 conn(0x7f8dc00b2900 msgr2=0x7f8dc00cc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:04.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.132+0000 7f8dbf59e640 1 -- 172.21.15.116:0/1324707688 shutdown_connections 2023-11-28T17:17:04.135 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.132+0000 7f8dbf59e640 1 -- 172.21.15.116:0/1324707688 wait complete. 2023-11-28T17:17:04.150 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-28T17:17:04.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.308+0000 7fd6d2f89640 1 Processor -- start 2023-11-28T17:17:04.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.308+0000 7fd6d2f89640 1 -- start start 2023-11-28T17:17:04.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.310+0000 7fd6d2f89640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd6cc13cee0 0x7fd6cc13d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.310+0000 7fd6d2f89640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6cc13f640 0x7fd6cc13fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.311+0000 7fd6d2f89640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc13d2c0 con 0x7fd6cc071d20 2023-11-28T17:17:04.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.311+0000 7fd6d2f89640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6cc1400e0 con 0x7fd6cc13f640 2023-11-28T17:17:04.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.311+0000 7fd6d2f89640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6cc1403c0 con 0x7fd6cc13cee0 2023-11-28T17:17:04.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.311+0000 7fd6cbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6cc13f640 0x7fd6cc13fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.311+0000 7fd6cbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6cc13f640 0x7fd6cc13fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53914/0 (socket says 172.21.15.116:53914) 2023-11-28T17:17:04.312 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.311+0000 7fd6cbfff640 1 -- 172.21.15.116:0/1720007397 learned_addr learned my addr 172.21.15.116:0/1720007397 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:04.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.311+0000 7fd6cbfff640 1 -- 172.21.15.116:0/1720007397 >> v2:172.21.15.156:3300/0 conn(0x7fd6cc13cee0 msgr2=0x7fd6cc13d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:04.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.311+0000 7fd6cbfff640 1 --2- 172.21.15.116:0/1720007397 >> v2:172.21.15.156:3300/0 conn(0x7fd6cc13cee0 0x7fd6cc13d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.311+0000 7fd6cbfff640 1 -- 172.21.15.116:0/1720007397 >> v1:172.21.15.156:6789/0 conn(0x7fd6cc071d20 legacy=0x7fd6cc072130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:04.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.311+0000 7fd6cbfff640 1 -- 172.21.15.116:0/1720007397 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc0cdb50 con 0x7fd6cc13f640 2023-11-28T17:17:04.313 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.312+0000 7fd6cbfff640 1 --2- 172.21.15.116:0/1720007397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6cc13f640 0x7fd6cc13fab0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc06b470 tx=0x7fd6b802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f84a080fa7ad775 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.313+0000 7fd6cb7fe640 1 -- 172.21.15.116:0/1720007397 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8031750 con 0x7fd6cc13f640 2023-11-28T17:17:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.313+0000 7fd6cb7fe640 1 -- 172.21.15.116:0/1720007397 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b80318b0 con 0x7fd6cc13f640 2023-11-28T17:17:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.313+0000 7fd6cb7fe640 1 -- 172.21.15.116:0/1720007397 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8031c40 con 0x7fd6cc13f640 2023-11-28T17:17:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.313+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1720007397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6cc13f640 msgr2=0x7fd6cc13fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.313+0000 7fd6d2f89640 1 --2- 172.21.15.116:0/1720007397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6cc13f640 0x7fd6cc13fab0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc06b470 tx=0x7fd6b802b7f0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.313+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1720007397 shutdown_connections 2023-11-28T17:17:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.313+0000 7fd6d2f89640 1 --2- 172.21.15.116:0/1720007397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6cc13f640 0x7fd6cc13fab0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.313+0000 7fd6d2f89640 1 --2- 172.21.15.116:0/1720007397 >> v2:172.21.15.156:3300/0 conn(0x7fd6cc13cee0 0x7fd6cc13d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.313+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1720007397 >> 172.21.15.116:0/1720007397 conn(0x7fd6cc0b0cf0 msgr2=0x7fd6cc0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.313+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1720007397 shutdown_connections 2023-11-28T17:17:04.315 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.313+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1720007397 wait complete. 2023-11-28T17:17:04.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.315+0000 7fd6d2f89640 1 Processor -- start 2023-11-28T17:17:04.316 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.315+0000 7fd6d2f89640 1 -- start start 2023-11-28T17:17:04.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.316+0000 7fd6d2f89640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6cc071d20 0x7fd6cc15aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.316+0000 7fd6d2f89640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd6cc13cee0 0x7fd6cc136120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.316+0000 7fd6d2f89640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6cc13f640 0x7fd6cc15c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.317 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.316+0000 7fd6d2f89640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc136d00 con 0x7fd6cc13f640 2023-11-28T17:17:04.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.316+0000 7fd6d2f89640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc15b410 con 0x7fd6cc071d20 2023-11-28T17:17:04.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.316+0000 7fd6d2f89640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc1539c0 con 0x7fd6cc13cee0 2023-11-28T17:17:04.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.316+0000 7fd6d1526640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6cc13f640 0x7fd6cc15c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.317+0000 7fd6d1526640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6cc13f640 0x7fd6cc15c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53922/0 (socket says 172.21.15.116:53922) 2023-11-28T17:17:04.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.317+0000 7fd6d1526640 1 -- 172.21.15.116:0/1352666633 learned_addr learned my addr 172.21.15.116:0/1352666633 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:04.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.317+0000 7fd6cbfff640 1 --2- 172.21.15.116:0/1352666633 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd6cc13cee0 0x7fd6cc136120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.317+0000 7fd6d0d25640 1 --2- 172.21.15.116:0/1352666633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6cc071d20 0x7fd6cc15aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.317+0000 7fd6d1526640 1 -- 172.21.15.116:0/1352666633 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd6cc13cee0 msgr2=0x7fd6cc136120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.317+0000 7fd6d1526640 1 --2- 172.21.15.116:0/1352666633 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd6cc13cee0 0x7fd6cc136120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.317+0000 7fd6d1526640 1 -- 172.21.15.116:0/1352666633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6cc071d20 msgr2=0x7fd6cc15aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.317+0000 7fd6d1526640 1 --2- 172.21.15.116:0/1352666633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6cc071d20 0x7fd6cc15aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.317+0000 7fd6d1526640 1 -- 172.21.15.116:0/1352666633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b80094d0 con 0x7fd6cc13f640 2023-11-28T17:17:04.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.317+0000 7fd6d1526640 1 --2- 172.21.15.116:0/1352666633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6cc13f640 0x7fd6cc15c6b0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd6c400c7b0 tx=0x7fd6c400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:04.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.318+0000 7fd6c97fa640 1 -- 172.21.15.116:0/1352666633 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4003d80 con 0x7fd6cc13f640 2023-11-28T17:17:04.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.318+0000 7fd6c97fa640 1 -- 172.21.15.116:0/1352666633 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c400ce60 con 0x7fd6cc13f640 2023-11-28T17:17:04.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.318+0000 7fd6c97fa640 1 -- 172.21.15.116:0/1352666633 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40052b0 con 0x7fd6cc13f640 2023-11-28T17:17:04.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.318+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1352666633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6cc1676b0 con 0x7fd6cc13f640 2023-11-28T17:17:04.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.318+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1352666633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6cc153ef0 con 0x7fd6cc13f640 2023-11-28T17:17:04.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.319+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1352666633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6cc1391a0 con 0x7fd6cc13f640 2023-11-28T17:17:04.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.320+0000 7fd6c97fa640 1 -- 172.21.15.116:0/1352666633 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7fd6c4006b10 con 0x7fd6cc13f640 2023-11-28T17:17:04.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.321+0000 7fd6c97fa640 1 --2- 172.21.15.116:0/1352666633 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd6b006b2a0 0x7fd6b006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.321+0000 7fd6c97fa640 1 -- 172.21.15.116:0/1352666633 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7fd6c4088f80 con 0x7fd6cc13f640 2023-11-28T17:17:04.322 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.321+0000 7fd6d0d25640 1 --2- 172.21.15.116:0/1352666633 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd6b006b2a0 0x7fd6b006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.323 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.322+0000 7fd6d0d25640 1 --2- 172.21.15.116:0/1352666633 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd6b006b2a0 0x7fd6b006d760 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc167ab0 tx=0x7fd6c0008040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:04.326 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.325+0000 7fd6c97fa640 1 -- 172.21.15.116:0/1352666633 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6c40bbe90 con 0x7fd6cc13f640 2023-11-28T17:17:04.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.371+0000 7f3f945c5640 1 Processor -- start 2023-11-28T17:17:04.373 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.372+0000 7f3f945c5640 1 -- start start 2023-11-28T17:17:04.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.374+0000 7f3f945c5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3f8c06c960 0x7f3f8c15a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.374+0000 7f3f945c5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f8c140200 0x7f3f8c15a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.374+0000 7f3f945c5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f8c0aea20 con 0x7f3f8c13cee0 2023-11-28T17:17:04.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.374+0000 7f3f945c5640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f8c13d2c0 con 0x7f3f8c06c960 2023-11-28T17:17:04.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.374+0000 7f3f945c5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c15b3a0 con 0x7f3f8c140200 2023-11-28T17:17:04.376 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.374+0000 7f3f91b60640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3f8c06c960 0x7f3f8c15a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.374+0000 7f3f92361640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3f8c13cee0 0x7f3f8c06c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33008/0 (socket says 172.21.15.116:33008) 2023-11-28T17:17:04.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.374+0000 7f3f92361640 1 -- 172.21.15.116:0/270285887 learned_addr learned my addr 172.21.15.116:0/270285887 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:17:04.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.375+0000 7f3f92b62640 1 --2- 172.21.15.116:0/270285887 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f8c140200 0x7f3f8c15a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.375+0000 7f3f91b60640 1 -- 172.21.15.116:0/270285887 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f8c140200 msgr2=0x7f3f8c15a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:04.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.375+0000 7f3f91b60640 1 --2- 172.21.15.116:0/270285887 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f8c140200 0x7f3f8c15a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.375+0000 7f3f91b60640 1 -- 172.21.15.116:0/270285887 >> v1:172.21.15.116:6789/0 conn(0x7f3f8c13cee0 legacy=0x7f3f8c06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.375+0000 7f3f9135f640 1 -- 172.21.15.116:0/270285887 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688951094 0 0) 0x7f3f8c0aea20 con 0x7f3f8c13cee0 2023-11-28T17:17:04.377 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.375+0000 7f3f91b60640 1 -- 172.21.15.116:0/270285887 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8c15b0c0 con 0x7f3f8c06c960 2023-11-28T17:17:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.376+0000 7f3f91b60640 1 --2- 172.21.15.116:0/270285887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f8c06c960 0x7f3f8c15a490 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c0097a0 tx=0x7f3f7c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa0c440d6e750f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.376+0000 7f3f9135f640 1 -- 172.21.15.116:0/270285887 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c02e070 con 0x7f3f8c06c960 2023-11-28T17:17:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.376+0000 7f3f9135f640 1 -- 172.21.15.116:0/270285887 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7c0315a0 con 0x7f3f8c06c960 2023-11-28T17:17:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.376+0000 7f3f9135f640 1 -- 172.21.15.116:0/270285887 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c031930 con 0x7f3f8c06c960 2023-11-28T17:17:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.377+0000 7f3f945c5640 1 -- 172.21.15.116:0/270285887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f8c06c960 msgr2=0x7f3f8c15a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.377+0000 7f3f945c5640 1 --2- 172.21.15.116:0/270285887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f8c06c960 0x7f3f8c15a490 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c0097a0 tx=0x7f3f7c02b840 comp rx=0 tx=0).stop 2023-11-28T17:17:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.377+0000 7f3f945c5640 1 -- 172.21.15.116:0/270285887 shutdown_connections 2023-11-28T17:17:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.377+0000 7f3f945c5640 1 --2- 172.21.15.116:0/270285887 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f8c140200 0x7f3f8c15a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.377+0000 7f3f945c5640 1 --2- 172.21.15.116:0/270285887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f8c06c960 0x7f3f8c15a490 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.377+0000 7f3f945c5640 1 -- 172.21.15.116:0/270285887 >> 172.21.15.116:0/270285887 conn(0x7f3f8c06a4c0 msgr2=0x7f3f8c06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:04.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.378+0000 7f3f945c5640 1 -- 172.21.15.116:0/270285887 shutdown_connections 2023-11-28T17:17:04.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.378+0000 7f3f945c5640 1 -- 172.21.15.116:0/270285887 wait complete. 2023-11-28T17:17:04.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.379+0000 7f3f945c5640 1 Processor -- start 2023-11-28T17:17:04.380 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.379+0000 7f3f945c5640 1 -- start start 2023-11-28T17:17:04.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.380+0000 7f3f945c5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f8c06c960 0x7f3f8c14d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.380+0000 7f3f945c5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f8c13cee0 0x7f3f8c13ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.380+0000 7f3f945c5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f8c140200 0x7f3f8c159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.380+0000 7f3f945c5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0711c0 con 0x7f3f8c13cee0 2023-11-28T17:17:04.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.380+0000 7f3f945c5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0714a0 con 0x7f3f8c140200 2023-11-28T17:17:04.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.380+0000 7f3f945c5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c071780 con 0x7f3f8c06c960 2023-11-28T17:17:04.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.380+0000 7f3f91b60640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f8c13cee0 0x7f3f8c13ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.381+0000 7f3f91b60640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f8c13cee0 0x7f3f8c13ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53936/0 (socket says 172.21.15.116:53936) 2023-11-28T17:17:04.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.381+0000 7f3f91b60640 1 -- 172.21.15.116:0/472864516 learned_addr learned my addr 172.21.15.116:0/472864516 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:04.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.381+0000 7f3f92361640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f8c06c960 0x7f3f8c14d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.381+0000 7f3f92b62640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f8c140200 0x7f3f8c159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.381+0000 7f3f91b60640 1 -- 172.21.15.116:0/472864516 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f8c06c960 msgr2=0x7f3f8c14d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.381+0000 7f3f91b60640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f8c06c960 0x7f3f8c14d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.381+0000 7f3f91b60640 1 -- 172.21.15.116:0/472864516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f8c140200 msgr2=0x7f3f8c159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.381+0000 7f3f91b60640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f8c140200 0x7f3f8c159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.381+0000 7f3f91b60640 1 -- 172.21.15.116:0/472864516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c009400 con 0x7f3f8c13cee0 2023-11-28T17:17:04.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.381+0000 7f3f92361640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f8c06c960 0x7f3f8c14d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T17:17:04.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.381+0000 7f3f91b60640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f8c13cee0 0x7f3f8c13ad60 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c034040 tx=0x7f3f7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:04.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.382+0000 7f3f82ffd640 1 -- 172.21.15.116:0/472864516 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c032900 con 0x7f3f8c13cee0 2023-11-28T17:17:04.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.382+0000 7f3f82ffd640 1 -- 172.21.15.116:0/472864516 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7c032a60 con 0x7f3f8c13cee0 2023-11-28T17:17:04.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.382+0000 7f3f82ffd640 1 -- 172.21.15.116:0/472864516 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c032e10 con 0x7f3f8c13cee0 2023-11-28T17:17:04.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.382+0000 7f3f945c5640 1 -- 172.21.15.116:0/472864516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8c071a60 con 0x7f3f8c13cee0 2023-11-28T17:17:04.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.382+0000 7f3f945c5640 1 -- 172.21.15.116:0/472864516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8c072260 con 0x7f3f8c13cee0 2023-11-28T17:17:04.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.383+0000 7f3f945c5640 1 -- 172.21.15.116:0/472864516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f8c071f80 con 0x7f3f8c13cee0 2023-11-28T17:17:04.385 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.384+0000 7f3f82ffd640 1 -- 172.21.15.116:0/472864516 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3f7c04e430 con 0x7f3f8c13cee0 2023-11-28T17:17:04.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.385+0000 7f3f82ffd640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3f7006f6b0 0x7f3f70071b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.385+0000 7f3f82ffd640 1 -- 172.21.15.116:0/472864516 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f3f7c0abf90 con 0x7f3f8c13cee0 2023-11-28T17:17:04.386 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.385+0000 7f3f92361640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3f7006f6b0 0x7f3f70071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.387 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.386+0000 7f3f92361640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3f7006f6b0 0x7f3f70071b70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3f88000f60 tx=0x7f3f88007c10 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:04.390 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.389+0000 7f3f82ffd640 1 -- 172.21.15.116:0/472864516 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f8c071f80 con 0x7f3f8c13cee0 2023-11-28T17:17:04.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.523+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1352666633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd6cc069360 con 0x7fd6cc13f640 2023-11-28T17:17:04.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.524+0000 7fd6c97fa640 1 -- 172.21.15.116:0/1352666633 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd6c4017070 con 0x7fd6cc13f640 2023-11-28T17:17:04.526 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2023-11-28T17:17:04.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.527+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1352666633 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd6b006b2a0 msgr2=0x7fd6b006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.527+0000 7fd6d2f89640 1 --2- 172.21.15.116:0/1352666633 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd6b006b2a0 0x7fd6b006d760 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc167ab0 tx=0x7fd6c0008040 comp rx=0 tx=0).stop 2023-11-28T17:17:04.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.527+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1352666633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6cc13f640 msgr2=0x7fd6cc15c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.527+0000 7fd6d2f89640 1 --2- 172.21.15.116:0/1352666633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6cc13f640 0x7fd6cc15c6b0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd6c400c7b0 tx=0x7fd6c400cc70 comp rx=0 tx=0).stop 2023-11-28T17:17:04.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.527+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1352666633 shutdown_connections 2023-11-28T17:17:04.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.527+0000 7fd6d2f89640 1 --2- 172.21.15.116:0/1352666633 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7fd6b006b2a0 0x7fd6b006d760 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.527+0000 7fd6d2f89640 1 --2- 172.21.15.116:0/1352666633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6cc13f640 0x7fd6cc15c6b0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.527+0000 7fd6d2f89640 1 --2- 172.21.15.116:0/1352666633 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd6cc13cee0 0x7fd6cc136120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.527+0000 7fd6d2f89640 1 --2- 172.21.15.116:0/1352666633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6cc071d20 0x7fd6cc15aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.527+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1352666633 >> 172.21.15.116:0/1352666633 conn(0x7fd6cc0b0cf0 msgr2=0x7fd6cc0afb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:04.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.527+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1352666633 shutdown_connections 2023-11-28T17:17:04.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.527+0000 7fd6d2f89640 1 -- 172.21.15.116:0/1352666633 wait complete. 2023-11-28T17:17:04.544 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-28T17:17:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.585+0000 7f3f945c5640 1 -- 172.21.15.116:0/472864516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3f8c071f80 con 0x7f3f8c13cee0 2023-11-28T17:17:04.586 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.585+0000 7f3f82ffd640 1 -- 172.21.15.116:0/472864516 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v52) v1 ==== 80+0+246 (secure 0 0 0) 0x7f3f8c071f80 con 0x7f3f8c13cee0 2023-11-28T17:17:04.587 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.156:6833/372407661 2023-11-29T17:12:26.038071+0000 2023-11-28T17:17:04.587 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.156:6832/372407661 2023-11-29T17:12:26.038071+0000 2023-11-28T17:17:04.587 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/3251347229 2023-11-29T17:12:06.030424+0000 2023-11-28T17:17:04.587 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/3251347229 2023-11-29T17:12:06.030424+0000 2023-11-28T17:17:04.587 INFO:teuthology.orchestra.run.smithi116.stderr:listed 4 entries 2023-11-28T17:17:04.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.587+0000 7f3f945c5640 1 -- 172.21.15.116:0/472864516 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3f7006f6b0 msgr2=0x7f3f70071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.587+0000 7f3f945c5640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3f7006f6b0 0x7f3f70071b70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3f88000f60 tx=0x7f3f88007c10 comp rx=0 tx=0).stop 2023-11-28T17:17:04.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.587+0000 7f3f945c5640 1 -- 172.21.15.116:0/472864516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f8c13cee0 msgr2=0x7f3f8c13ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.587+0000 7f3f945c5640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f8c13cee0 0x7f3f8c13ad60 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c034040 tx=0x7f3f7c0046c0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.587+0000 7f3f945c5640 1 -- 172.21.15.116:0/472864516 shutdown_connections 2023-11-28T17:17:04.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.587+0000 7f3f945c5640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3f7006f6b0 0x7f3f70071b70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.587+0000 7f3f945c5640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f8c140200 0x7f3f8c159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.587+0000 7f3f945c5640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f8c13cee0 0x7f3f8c13ad60 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.587+0000 7f3f945c5640 1 --2- 172.21.15.116:0/472864516 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f8c06c960 0x7f3f8c14d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.589 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.587+0000 7f3f945c5640 1 -- 172.21.15.116:0/472864516 >> 172.21.15.116:0/472864516 conn(0x7f3f8c06a4c0 msgr2=0x7f3f8c139a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:04.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.587+0000 7f3f945c5640 1 -- 172.21.15.116:0/472864516 shutdown_connections 2023-11-28T17:17:04.590 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.588+0000 7f3f945c5640 1 -- 172.21.15.116:0/472864516 wait complete. 2023-11-28T17:17:04.605 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/fuse_mount.py", line 392, in umount_wait if self.is_blocked(): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/mount.py", line 197, in is_blocked return self.addr in output TypeError: 'in ' requires string as left operand, not NoneType 2023-11-28T17:17:04.605 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:17:04.605 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:17:04.665 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T17:17:04.665 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.3 is not mounted at ubuntu@smithi156.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.3 2023-11-28T17:17:04.665 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:17:04.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:17:04.666 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-11-28T17:17:04.721 INFO:teuthology.orchestra.run.smithi156.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-11-28T17:17:04.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:17:04.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.722+0000 7f3fe9e6e640 1 Processor -- start 2023-11-28T17:17:04.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.723+0000 7f3fe9e6e640 1 -- start start 2023-11-28T17:17:04.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.724+0000 7f3fe9e6e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3fe4152050 0x7f3fe415bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.724+0000 7f3fe9e6e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3fe40b2fe0 0x7f3fe40ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.724+0000 7f3fe9e6e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe40b2b50 con 0x7f3fe40abd30 2023-11-28T17:17:04.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.724+0000 7f3fe9e6e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe415c520 con 0x7f3fe40b2fe0 2023-11-28T17:17:04.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.724+0000 7f3fe9e6e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe40ac2d0 con 0x7f3fe4152050 2023-11-28T17:17:04.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.724+0000 7f3fdbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3fe40b2fe0 0x7f3fe40ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.726 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.724+0000 7f3fdbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3fe40b2fe0 0x7f3fe40ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53944/0 (socket says 172.21.15.116:53944) 2023-11-28T17:17:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.724+0000 7f3fdbfff640 1 -- 172.21.15.116:0/3645643663 learned_addr learned my addr 172.21.15.116:0/3645643663 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.724+0000 7f3fe966d640 1 --1- 172.21.15.116:0/3645643663 >> v1:172.21.15.116:6789/0 conn(0x7f3fe40abd30 0x7f3fe40cdc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33014/0 (socket says 172.21.15.116:33014) 2023-11-28T17:17:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.725+0000 7f3fe8e6c640 1 --2- 172.21.15.116:0/3645643663 >> v2:172.21.15.156:3300/0 conn(0x7f3fe4152050 0x7f3fe415bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.725+0000 7f3fdbfff640 1 -- 172.21.15.116:0/3645643663 >> v2:172.21.15.156:3300/0 conn(0x7f3fe4152050 msgr2=0x7f3fe415bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.725+0000 7f3fdbfff640 1 --2- 172.21.15.116:0/3645643663 >> v2:172.21.15.156:3300/0 conn(0x7f3fe4152050 0x7f3fe415bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.725+0000 7f3fdbfff640 1 -- 172.21.15.116:0/3645643663 >> v1:172.21.15.116:6789/0 conn(0x7f3fe40abd30 legacy=0x7f3fe40cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.725+0000 7f3fdb7fe640 1 -- 172.21.15.116:0/3645643663 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79027450 0 0) 0x7f3fe40b2b50 con 0x7f3fe40abd30 2023-11-28T17:17:04.727 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.725+0000 7f3fdbfff640 1 -- 172.21.15.116:0/3645643663 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe40b2220 con 0x7f3fe40b2fe0 2023-11-28T17:17:04.728 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.726+0000 7f3fdbfff640 1 --2- 172.21.15.116:0/3645643663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fe40b2fe0 0x7f3fe40ad930 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f3fe406fe80 tx=0x7f3fd402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19174043269b7967 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:04.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.726+0000 7f3fdb7fe640 1 -- 172.21.15.116:0/3645643663 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4030860 con 0x7f3fe40b2fe0 2023-11-28T17:17:04.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.726+0000 7f3fdb7fe640 1 -- 172.21.15.116:0/3645643663 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd40309c0 con 0x7f3fe40b2fe0 2023-11-28T17:17:04.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.726+0000 7f3fdb7fe640 1 -- 172.21.15.116:0/3645643663 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4030d50 con 0x7f3fe40b2fe0 2023-11-28T17:17:04.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.727+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3645643663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fe40b2fe0 msgr2=0x7f3fe40ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.727+0000 7f3fe9e6e640 1 --2- 172.21.15.116:0/3645643663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fe40b2fe0 0x7f3fe40ad930 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f3fe406fe80 tx=0x7f3fd402b9a0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.727+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3645643663 shutdown_connections 2023-11-28T17:17:04.729 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.727+0000 7f3fe9e6e640 1 --2- 172.21.15.116:0/3645643663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fe40b2fe0 0x7f3fe40ad930 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.727+0000 7f3fe9e6e640 1 --2- 172.21.15.116:0/3645643663 >> v2:172.21.15.156:3300/0 conn(0x7f3fe4152050 0x7f3fe415bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.727+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3645643663 >> 172.21.15.116:0/3645643663 conn(0x7f3fe40b0950 msgr2=0x7f3fe40b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.727+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3645643663 shutdown_connections 2023-11-28T17:17:04.730 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.727+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3645643663 wait complete. 2023-11-28T17:17:04.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.728+0000 7f3fe9e6e640 1 Processor -- start 2023-11-28T17:17:04.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.729+0000 7f3fe9e6e640 1 -- start start 2023-11-28T17:17:04.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.729+0000 7f3fe9e6e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fe40abd30 0x7f3fe41525b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.730+0000 7f3fe9e6e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fe40b2fe0 0x7f3fe4139ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.730+0000 7f3fe9e6e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fe4152050 0x7f3fe413c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.730+0000 7f3fe9e6e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe414c9f0 con 0x7f3fe40abd30 2023-11-28T17:17:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.730+0000 7f3fe9e6e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe414bde0 con 0x7f3fe40b2fe0 2023-11-28T17:17:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.730+0000 7f3fe9e6e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe414ce10 con 0x7f3fe4152050 2023-11-28T17:17:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.730+0000 7f3fe8e6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fe40abd30 0x7f3fe41525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.730+0000 7f3fe8e6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fe40abd30 0x7f3fe41525b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53950/0 (socket says 172.21.15.116:53950) 2023-11-28T17:17:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.730+0000 7f3fe8e6c640 1 -- 172.21.15.116:0/3713187812 learned_addr learned my addr 172.21.15.116:0/3713187812 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.730+0000 7f3fe8e6c640 1 -- 172.21.15.116:0/3713187812 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fe4152050 msgr2=0x7f3fe413c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.734 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.730+0000 7f3fe8e6c640 1 --2- 172.21.15.116:0/3713187812 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fe4152050 0x7f3fe413c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.730+0000 7f3fe8e6c640 1 -- 172.21.15.116:0/3713187812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fe40b2fe0 msgr2=0x7f3fe4139ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.730+0000 7f3fe8e6c640 1 --2- 172.21.15.116:0/3713187812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fe40b2fe0 0x7f3fe4139ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.730+0000 7f3fe8e6c640 1 -- 172.21.15.116:0/3713187812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd40094d0 con 0x7f3fe40abd30 2023-11-28T17:17:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.730+0000 7f3fe8e6c640 1 --2- 172.21.15.116:0/3713187812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fe40abd30 0x7f3fe41525b0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000d620 tx=0x7f3fe000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.731+0000 7f3fd97fa640 1 -- 172.21.15.116:0/3713187812 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe000b620 con 0x7f3fe40abd30 2023-11-28T17:17:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.731+0000 7f3fd97fa640 1 -- 172.21.15.116:0/3713187812 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe000b780 con 0x7f3fe40abd30 2023-11-28T17:17:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.731+0000 7f3fd97fa640 1 -- 172.21.15.116:0/3713187812 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe000bb10 con 0x7f3fe40abd30 2023-11-28T17:17:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.731+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3713187812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe414c050 con 0x7f3fe40abd30 2023-11-28T17:17:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.731+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3713187812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe414c610 con 0x7f3fe40abd30 2023-11-28T17:17:04.735 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.732+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3713187812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe409d0b0 con 0x7f3fe40abd30 2023-11-28T17:17:04.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.733+0000 7f3fd97fa640 1 -- 172.21.15.116:0/3713187812 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f3fe000bc70 con 0x7f3fe40abd30 2023-11-28T17:17:04.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.734+0000 7f3fd97fa640 1 --2- 172.21.15.116:0/3713187812 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3fd006b2a0 0x7f3fd006d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:04.736 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.734+0000 7f3fd97fa640 1 -- 172.21.15.116:0/3713187812 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f3fe008a210 con 0x7f3fe40abd30 2023-11-28T17:17:04.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.734+0000 7f3fdbfff640 1 --2- 172.21.15.116:0/3713187812 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3fd006b2a0 0x7f3fd006d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:04.737 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.735+0000 7f3fdbfff640 1 --2- 172.21.15.116:0/3713187812 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3fd006b2a0 0x7f3fd006d760 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4036000 tx=0x7f3fd403b040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:04.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.738+0000 7f3fd97fa640 1 -- 172.21.15.116:0/3713187812 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3fe0054f30 con 0x7f3fe40abd30 2023-11-28T17:17:04.912 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.911+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3713187812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3fe404e960 con 0x7f3fe40abd30 2023-11-28T17:17:04.913 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.912+0000 7f3fd97fa640 1 -- 172.21.15.116:0/3713187812 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3fe0058550 con 0x7f3fe40abd30 2023-11-28T17:17:04.913 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2023-11-28T17:17:04.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.914+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3713187812 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3fd006b2a0 msgr2=0x7f3fd006d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.914+0000 7f3fe9e6e640 1 --2- 172.21.15.116:0/3713187812 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3fd006b2a0 0x7f3fd006d760 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4036000 tx=0x7f3fd403b040 comp rx=0 tx=0).stop 2023-11-28T17:17:04.916 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.914+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3713187812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fe40abd30 msgr2=0x7f3fe41525b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:04.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.914+0000 7f3fe9e6e640 1 --2- 172.21.15.116:0/3713187812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fe40abd30 0x7f3fe41525b0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000d620 tx=0x7f3fe000dae0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.917 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.914+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3713187812 shutdown_connections 2023-11-28T17:17:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.914+0000 7f3fe9e6e640 1 --2- 172.21.15.116:0/3713187812 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f3fd006b2a0 0x7f3fd006d760 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.914+0000 7f3fe9e6e640 1 --2- 172.21.15.116:0/3713187812 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fe4152050 0x7f3fe413c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.914+0000 7f3fe9e6e640 1 --2- 172.21.15.116:0/3713187812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fe40b2fe0 0x7f3fe4139ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.914+0000 7f3fe9e6e640 1 --2- 172.21.15.116:0/3713187812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fe40abd30 0x7f3fe41525b0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.914+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3713187812 >> 172.21.15.116:0/3713187812 conn(0x7f3fe40b0950 msgr2=0x7f3fe406d040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.914+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3713187812 shutdown_connections 2023-11-28T17:17:04.918 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:04.914+0000 7f3fe9e6e640 1 -- 172.21.15.116:0/3713187812 wait complete. 2023-11-28T17:17:04.934 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-28T17:17:05.137 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.136+0000 7f78eeca6640 1 Processor -- start 2023-11-28T17:17:05.138 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.136+0000 7f78eeca6640 1 -- start start 2023-11-28T17:17:05.139 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.137+0000 7f78eeca6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f78e815d480 0x7f78e80d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:05.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.137+0000 7f78eeca6640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f78e80d01e0 0x7f78e8160270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:05.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.137+0000 7f78eeca6640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78e815ddc0 con 0x7f78e80d1310 2023-11-28T17:17:05.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.137+0000 7f78eeca6640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78e815e0a0 con 0x7f78e815d480 2023-11-28T17:17:05.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.137+0000 7f78eeca6640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78e815e380 con 0x7f78e80d01e0 2023-11-28T17:17:05.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.137+0000 7f78eca42640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f78e815d480 0x7f78e80d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:05.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.137+0000 7f78eca42640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f78e815d480 0x7f78e80d0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53952/0 (socket says 172.21.15.116:53952) 2023-11-28T17:17:05.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.137+0000 7f78eca42640 1 -- 172.21.15.116:0/450273455 learned_addr learned my addr 172.21.15.116:0/450273455 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:05.140 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.138+0000 7f78eca42640 1 -- 172.21.15.116:0/450273455 >> v2:172.21.15.156:3300/0 conn(0x7f78e80d01e0 msgr2=0x7f78e8160270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:05.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.138+0000 7f78ed243640 1 --2- 172.21.15.116:0/450273455 >> v2:172.21.15.156:3300/0 conn(0x7f78e80d01e0 0x7f78e8160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:05.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.138+0000 7f78eca42640 1 --2- 172.21.15.116:0/450273455 >> v2:172.21.15.156:3300/0 conn(0x7f78e80d01e0 0x7f78e8160270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.138+0000 7f78eca42640 1 -- 172.21.15.116:0/450273455 >> v1:172.21.15.156:6789/0 conn(0x7f78e80d1310 legacy=0x7f78e80cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:05.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.138+0000 7f78eca42640 1 -- 172.21.15.116:0/450273455 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e815e660 con 0x7f78e815d480 2023-11-28T17:17:05.141 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.138+0000 7f78ed243640 1 --2- 172.21.15.116:0/450273455 >> v2:172.21.15.156:3300/0 conn(0x7f78e80d01e0 0x7f78e8160270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:17:05.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.139+0000 7f78eca42640 1 --2- 172.21.15.116:0/450273455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e815d480 0x7f78e80d0dd0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f78e808d3f0 tx=0x7f78d802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba217f80f23e1d server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:05.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.139+0000 7f78e77fe640 1 -- 172.21.15.116:0/450273455 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d802ca80 con 0x7f78e815d480 2023-11-28T17:17:05.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.139+0000 7f78e77fe640 1 -- 172.21.15.116:0/450273455 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d802cbe0 con 0x7f78e815d480 2023-11-28T17:17:05.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.139+0000 7f78e77fe640 1 -- 172.21.15.116:0/450273455 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d80343f0 con 0x7f78e815d480 2023-11-28T17:17:05.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.140+0000 7f78eeca6640 1 -- 172.21.15.116:0/450273455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e815d480 msgr2=0x7f78e80d0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:05.143 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.140+0000 7f78eeca6640 1 --2- 172.21.15.116:0/450273455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e815d480 0x7f78e80d0dd0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f78e808d3f0 tx=0x7f78d802bee0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.140+0000 7f78eeca6640 1 -- 172.21.15.116:0/450273455 shutdown_connections 2023-11-28T17:17:05.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.140+0000 7f78eeca6640 1 --2- 172.21.15.116:0/450273455 >> v2:172.21.15.156:3300/0 conn(0x7f78e80d01e0 0x7f78e8160270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.140+0000 7f78eeca6640 1 --2- 172.21.15.116:0/450273455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e815d480 0x7f78e80d0dd0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.140+0000 7f78eeca6640 1 -- 172.21.15.116:0/450273455 >> 172.21.15.116:0/450273455 conn(0x7f78e809c540 msgr2=0x7f78e80d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:05.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.140+0000 7f78eeca6640 1 -- 172.21.15.116:0/450273455 shutdown_connections 2023-11-28T17:17:05.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.140+0000 7f78eeca6640 1 -- 172.21.15.116:0/450273455 wait complete. 2023-11-28T17:17:05.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.142+0000 7f78eeca6640 1 Processor -- start 2023-11-28T17:17:05.144 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.142+0000 7f78eeca6640 1 -- start start 2023-11-28T17:17:05.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.143+0000 7f78eeca6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78e80d01e0 0x7f78e80cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:05.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.143+0000 7f78eeca6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e80d1310 0x7f78e80c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:05.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.143+0000 7f78eeca6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e815d480 0x7f78e80b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:05.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.143+0000 7f78eeca6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e80cc510 con 0x7f78e815d480 2023-11-28T17:17:05.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.143+0000 7f78eeca6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e80b9060 con 0x7f78e80d01e0 2023-11-28T17:17:05.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.143+0000 7f78eeca6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e80cb900 con 0x7f78e80d1310 2023-11-28T17:17:05.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.143+0000 7f78ed243640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e815d480 0x7f78e80b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:05.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.143+0000 7f78ed243640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e815d480 0x7f78e80b8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53962/0 (socket says 172.21.15.116:53962) 2023-11-28T17:17:05.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.143+0000 7f78ed243640 1 -- 172.21.15.116:0/1614711300 learned_addr learned my addr 172.21.15.116:0/1614711300 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:05.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.143+0000 7f78ed243640 1 -- 172.21.15.116:0/1614711300 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e80d1310 msgr2=0x7f78e80c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:05.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.143+0000 7f78ed243640 1 --2- 172.21.15.116:0/1614711300 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e80d1310 0x7f78e80c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.148 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.143+0000 7f78ed243640 1 -- 172.21.15.116:0/1614711300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78e80d01e0 msgr2=0x7f78e80cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:05.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.143+0000 7f78ed243640 1 --2- 172.21.15.116:0/1614711300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78e80d01e0 0x7f78e80cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.143+0000 7f78ed243640 1 -- 172.21.15.116:0/1614711300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d8009b30 con 0x7f78e815d480 2023-11-28T17:17:05.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.144+0000 7f78ed243640 1 --2- 172.21.15.116:0/1614711300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e815d480 0x7f78e80b8b20 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f78e000c7c0 tx=0x7f78e000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:05.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.144+0000 7f78e57fa640 1 -- 172.21.15.116:0/1614711300 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e0004230 con 0x7f78e815d480 2023-11-28T17:17:05.149 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.144+0000 7f78e57fa640 1 -- 172.21.15.116:0/1614711300 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e0004390 con 0x7f78e815d480 2023-11-28T17:17:05.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.144+0000 7f78e57fa640 1 -- 172.21.15.116:0/1614711300 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e0007860 con 0x7f78e815d480 2023-11-28T17:17:05.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.144+0000 7f78eeca6640 1 -- 172.21.15.116:0/1614711300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e80cbbe0 con 0x7f78e815d480 2023-11-28T17:17:05.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.144+0000 7f78eeca6640 1 -- 172.21.15.116:0/1614711300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e80cc1a0 con 0x7f78e815d480 2023-11-28T17:17:05.150 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.145+0000 7f78eeca6640 1 -- 172.21.15.116:0/1614711300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e8067980 con 0x7f78e815d480 2023-11-28T17:17:05.151 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.146+0000 7f78e57fa640 1 -- 172.21.15.116:0/1614711300 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f78e00079c0 con 0x7f78e815d480 2023-11-28T17:17:05.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.147+0000 7f78e57fa640 1 --2- 172.21.15.116:0/1614711300 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f78c806b2a0 0x7f78c806d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:05.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.147+0000 7f78e57fa640 1 -- 172.21.15.116:0/1614711300 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f78e0089ed0 con 0x7f78e815d480 2023-11-28T17:17:05.152 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.147+0000 7f78eca42640 1 --2- 172.21.15.116:0/1614711300 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f78c806b2a0 0x7f78c806d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:05.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.148+0000 7f78eca42640 1 --2- 172.21.15.116:0/1614711300 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f78c806b2a0 0x7f78c806d760 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f78d8035000 tx=0x7f78d802e000 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:05.153 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.151+0000 7f78e57fa640 1 -- 172.21.15.116:0/1614711300 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78e0054e00 con 0x7f78e815d480 2023-11-28T17:17:05.332 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.331+0000 7f78eeca6640 1 -- 172.21.15.116:0/1614711300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f78e8069360 con 0x7f78e815d480 2023-11-28T17:17:05.333 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.332+0000 7f78e57fa640 1 -- 172.21.15.116:0/1614711300 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+11 (secure 0 0 0) 0x7f78e00150a0 con 0x7f78e815d480 2023-11-28T17:17:05.333 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2023-11-28T17:17:05.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.334+0000 7f78eeca6640 1 -- 172.21.15.116:0/1614711300 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f78c806b2a0 msgr2=0x7f78c806d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:05.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.334+0000 7f78eeca6640 1 --2- 172.21.15.116:0/1614711300 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f78c806b2a0 0x7f78c806d760 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f78d8035000 tx=0x7f78d802e000 comp rx=0 tx=0).stop 2023-11-28T17:17:05.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.334+0000 7f78eeca6640 1 -- 172.21.15.116:0/1614711300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e815d480 msgr2=0x7f78e80b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:05.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.334+0000 7f78eeca6640 1 --2- 172.21.15.116:0/1614711300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e815d480 0x7f78e80b8b20 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f78e000c7c0 tx=0x7f78e000cc80 comp rx=0 tx=0).stop 2023-11-28T17:17:05.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.334+0000 7f78eeca6640 1 -- 172.21.15.116:0/1614711300 shutdown_connections 2023-11-28T17:17:05.336 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.334+0000 7f78eeca6640 1 --2- 172.21.15.116:0/1614711300 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f78c806b2a0 0x7f78c806d760 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.334+0000 7f78eeca6640 1 --2- 172.21.15.116:0/1614711300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e815d480 0x7f78e80b8b20 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.334+0000 7f78eeca6640 1 --2- 172.21.15.116:0/1614711300 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e80d1310 0x7f78e80c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.334+0000 7f78eeca6640 1 --2- 172.21.15.116:0/1614711300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78e80d01e0 0x7f78e80cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.334+0000 7f78eeca6640 1 -- 172.21.15.116:0/1614711300 >> 172.21.15.116:0/1614711300 conn(0x7f78e809c540 msgr2=0x7f78e80cf910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:05.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.334+0000 7f78eeca6640 1 -- 172.21.15.116:0/1614711300 shutdown_connections 2023-11-28T17:17:05.337 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.334+0000 7f78eeca6640 1 -- 172.21.15.116:0/1614711300 wait complete. 2023-11-28T17:17:05.354 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-28T17:17:05.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.561+0000 7ff4c0a1c640 1 Processor -- start 2023-11-28T17:17:05.563 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.561+0000 7ff4c0a1c640 1 -- start start 2023-11-28T17:17:05.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.563+0000 7ff4c0a1c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff4bc15f7b0 0x7ff4bc071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:05.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.563+0000 7ff4c0a1c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4bc15fc10 con 0x7ff4bc144430 2023-11-28T17:17:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.563+0000 7ff4c0a1c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4bc070bd0 con 0x7ff4bc0721b0 2023-11-28T17:17:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.563+0000 7ff4c0a1c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4bc144810 con 0x7ff4bc15f7b0 2023-11-28T17:17:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.563+0000 7ff4ba59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff4bc144430 0x7ff4bc0703f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33026/0 (socket says 172.21.15.116:33026) 2023-11-28T17:17:05.565 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.563+0000 7ff4ba59c640 1 -- 172.21.15.116:0/3613859916 learned_addr learned my addr 172.21.15.116:0/3613859916 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-11-28T17:17:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 -- 172.21.15.116:0/3613859916 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907114133 0 0) 0x7ff4bc15fc10 con 0x7ff4bc144430 2023-11-28T17:17:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 -- 172.21.15.116:0/3613859916 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a8003520 con 0x7ff4bc144430 2023-11-28T17:17:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 -- 172.21.15.116:0/3613859916 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4080767208 0 0) 0x7ff4a8003520 con 0x7ff4bc144430 2023-11-28T17:17:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 -- 172.21.15.116:0/3613859916 >> v2:172.21.15.156:3300/0 conn(0x7ff4bc15f7b0 msgr2=0x7ff4bc071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:05.567 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 --2- 172.21.15.116:0/3613859916 >> v2:172.21.15.156:3300/0 conn(0x7ff4bc15f7b0 0x7ff4bc071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 -- 172.21.15.116:0/3613859916 >> v1:172.21.15.156:6789/0 conn(0x7ff4bc0721b0 legacy=0x7ff4bc06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 -- 172.21.15.116:0/3613859916 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc070eb0 con 0x7ff4bc144430 2023-11-28T17:17:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 -- 172.21.15.116:0/3613859916 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3729234566 0 0) 0x7ff4ac001670 con 0x7ff4bc144430 2023-11-28T17:17:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 -- 172.21.15.116:0/3613859916 >> v1:172.21.15.116:6789/0 conn(0x7ff4bc144430 legacy=0x7ff4bc0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 --2- 172.21.15.116:0/3613859916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4a80039c0 0x7ff4a8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:05.568 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 --2- 172.21.15.116:0/3613859916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4a80047c0 0x7ff4a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 --2- 172.21.15.116:0/3613859916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a80052a0 0x7ff4a800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 -- 172.21.15.116:0/3613859916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc144810 con 0x7ff4a80047c0 2023-11-28T17:17:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 -- 172.21.15.116:0/3613859916 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a800c1f0 con 0x7ff4a80052a0 2023-11-28T17:17:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.564+0000 7ff4b9d9b640 1 -- 172.21.15.116:0/3613859916 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a800c3b0 con 0x7ff4a80039c0 2023-11-28T17:17:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.565+0000 7ff4b3fff640 1 --2- 172.21.15.116:0/3613859916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4a80047c0 0x7ff4a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:05.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.565+0000 7ff4bad9d640 1 --2- 172.21.15.116:0/3613859916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a80052a0 0x7ff4a800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.565+0000 7ff4ba59c640 1 --2- 172.21.15.116:0/3613859916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4a80039c0 0x7ff4a8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.565+0000 7ff4b3fff640 1 -- 172.21.15.116:0/3613859916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4a80039c0 msgr2=0x7ff4a8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.565+0000 7ff4b3fff640 1 --2- 172.21.15.116:0/3613859916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4a80039c0 0x7ff4a8003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.565+0000 7ff4b3fff640 1 -- 172.21.15.116:0/3613859916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a80052a0 msgr2=0x7ff4a800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.565+0000 7ff4b3fff640 1 --2- 172.21.15.116:0/3613859916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a80052a0 0x7ff4a800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.565+0000 7ff4b3fff640 1 -- 172.21.15.116:0/3613859916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a8003520 con 0x7ff4a80047c0 2023-11-28T17:17:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.565+0000 7ff4bad9d640 1 -- 172.21.15.116:0/3613859916 reap_dead start 2023-11-28T17:17:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.566+0000 7ff4b3fff640 1 --2- 172.21.15.116:0/3613859916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4a80047c0 0x7ff4a8004c40 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4006fd0 tx=0x7ff4a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b66ed879762e7f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.566+0000 7ff4b9d9b640 1 -- 172.21.15.116:0/3613859916 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4032600 con 0x7ff4a80047c0 2023-11-28T17:17:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.566+0000 7ff4b9d9b640 1 -- 172.21.15.116:0/3613859916 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a402e050 con 0x7ff4a80047c0 2023-11-28T17:17:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.566+0000 7ff4b9d9b640 1 -- 172.21.15.116:0/3613859916 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a40329c0 con 0x7ff4a80047c0 2023-11-28T17:17:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.567+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/3613859916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4a80047c0 msgr2=0x7ff4a8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.567+0000 7ff4c0a1c640 1 --2- 172.21.15.116:0/3613859916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4a80047c0 0x7ff4a8004c40 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4006fd0 tx=0x7ff4a80057f0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.567+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/3613859916 shutdown_connections 2023-11-28T17:17:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.567+0000 7ff4c0a1c640 1 --2- 172.21.15.116:0/3613859916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4a80047c0 0x7ff4a8004c40 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.567+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/3613859916 >> 172.21.15.116:0/3613859916 conn(0x7ff4bc06a4c0 msgr2=0x7ff4bc06a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.567+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/3613859916 shutdown_connections 2023-11-28T17:17:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.567+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/3613859916 wait complete. 2023-11-28T17:17:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.569+0000 7ff4c0a1c640 1 Processor -- start 2023-11-28T17:17:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.569+0000 7ff4c0a1c640 1 -- start start 2023-11-28T17:17:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.570+0000 7ff4c0a1c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a80047c0 0x7ff4bc15b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.570+0000 7ff4c0a1c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4bc13c330 0x7ff4bc15e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:05.575 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.570+0000 7ff4c0a1c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4bc15e5e0 0x7ff4bc140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.570+0000 7ff4c0a1c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc13ff70 con 0x7ff4bc15e5e0 2023-11-28T17:17:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.570+0000 7ff4c0a1c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc1401c0 con 0x7ff4a80047c0 2023-11-28T17:17:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.570+0000 7ff4c0a1c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc1404a0 con 0x7ff4bc13c330 2023-11-28T17:17:05.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.570+0000 7ff4bad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4bc15e5e0 0x7ff4bc140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.570+0000 7ff4bad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4bc15e5e0 0x7ff4bc140f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53990/0 (socket says 172.21.15.116:53990) 2023-11-28T17:17:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.570+0000 7ff4bad9d640 1 -- 172.21.15.116:0/1104282359 learned_addr learned my addr 172.21.15.116:0/1104282359 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.570+0000 7ff4bad9d640 1 -- 172.21.15.116:0/1104282359 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4bc13c330 msgr2=0x7ff4bc15e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.570+0000 7ff4bad9d640 1 --2- 172.21.15.116:0/1104282359 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4bc13c330 0x7ff4bc15e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.570+0000 7ff4bad9d640 1 -- 172.21.15.116:0/1104282359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a80047c0 msgr2=0x7ff4bc15b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:05.577 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.570+0000 7ff4bad9d640 1 --2- 172.21.15.116:0/1104282359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a80047c0 0x7ff4bc15b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.570+0000 7ff4bad9d640 1 -- 172.21.15.116:0/1104282359 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc140780 con 0x7ff4bc15e5e0 2023-11-28T17:17:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.571+0000 7ff4bad9d640 1 --2- 172.21.15.116:0/1104282359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4bc15e5e0 0x7ff4bc140f80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc067fd0 tx=0x7ff4b400e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.571+0000 7ff4b2ffd640 1 -- 172.21.15.116:0/1104282359 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4006d60 con 0x7ff4bc15e5e0 2023-11-28T17:17:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.571+0000 7ff4b2ffd640 1 -- 172.21.15.116:0/1104282359 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b4007030 con 0x7ff4bc15e5e0 2023-11-28T17:17:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.571+0000 7ff4b2ffd640 1 -- 172.21.15.116:0/1104282359 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b40073e0 con 0x7ff4bc15e5e0 2023-11-28T17:17:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.571+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/1104282359 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4bc140a60 con 0x7ff4bc15e5e0 2023-11-28T17:17:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.571+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/1104282359 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4bc15bb90 con 0x7ff4bc15e5e0 2023-11-28T17:17:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.572+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/1104282359 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4bc140c50 con 0x7ff4bc15e5e0 2023-11-28T17:17:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.573+0000 7ff4b2ffd640 1 -- 172.21.15.116:0/1104282359 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7ff4b4007540 con 0x7ff4bc15e5e0 2023-11-28T17:17:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.574+0000 7ff4b2ffd640 1 --2- 172.21.15.116:0/1104282359 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff49c06b2a0 0x7ff49c06d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.574+0000 7ff4b2ffd640 1 -- 172.21.15.116:0/1104282359 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7ff4b4089da0 con 0x7ff4bc15e5e0 2023-11-28T17:17:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.574+0000 7ff4ba59c640 1 --2- 172.21.15.116:0/1104282359 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff49c06b2a0 0x7ff49c06d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.575+0000 7ff4ba59c640 1 --2- 172.21.15.116:0/1104282359 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff49c06b2a0 0x7ff49c06d760 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc050060 tx=0x7ff4ac009040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:05.580 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.579+0000 7ff4b2ffd640 1 -- 172.21.15.116:0/1104282359 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4b4054ac0 con 0x7ff4bc15e5e0 2023-11-28T17:17:05.809 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.808+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/1104282359 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4bc0693d0 con 0x7ff4bc15e5e0 2023-11-28T17:17:05.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.809+0000 7ff4b2ffd640 1 -- 172.21.15.116:0/1104282359 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+3403 (secure 0 0 0) 0x7ff4b401c300 con 0x7ff4bc15e5e0 2023-11-28T17:17:05.811 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-11-28T17:17:05.811 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":40,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-28T17:11:23.668388+0000","modified":"2023-11-28T17:15:32.267954+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5826,"mds_1":4790},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4790":{"gid":4790,"name":"d","rank":1,"incarnation":33,"state":"up:active","state_seq":22,"addr":"172.21.15.156:6835/2474839888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":2474839888},{"type":"v1","addr":"172.21.15.156:6835","nonce":2474839888}]},"join_fscid":-1,"laggy_since":"2023-11-28T17:12:46.046020+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5826":{"gid":5826,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":67,"addr":"172.21.15.116:6837/1482499383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1482499383},{"type":"v1","addr":"172.21.15.116:6837","nonce":1482499383}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-11-28T17:17:05.811 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 40 2023-11-28T17:17:05.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.812+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/1104282359 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff49c06b2a0 msgr2=0x7ff49c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:05.813 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.812+0000 7ff4c0a1c640 1 --2- 172.21.15.116:0/1104282359 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff49c06b2a0 0x7ff49c06d760 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc050060 tx=0x7ff4ac009040 comp rx=0 tx=0).stop 2023-11-28T17:17:05.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.812+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/1104282359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4bc15e5e0 msgr2=0x7ff4bc140f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:05.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.812+0000 7ff4c0a1c640 1 --2- 172.21.15.116:0/1104282359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4bc15e5e0 0x7ff4bc140f80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc067fd0 tx=0x7ff4b400e9e0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.812+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/1104282359 shutdown_connections 2023-11-28T17:17:05.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.812+0000 7ff4c0a1c640 1 --2- 172.21.15.116:0/1104282359 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7ff49c06b2a0 0x7ff49c06d760 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.812+0000 7ff4c0a1c640 1 --2- 172.21.15.116:0/1104282359 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4bc15e5e0 0x7ff4bc140f80 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.814 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.812+0000 7ff4c0a1c640 1 --2- 172.21.15.116:0/1104282359 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4bc13c330 0x7ff4bc15e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.812+0000 7ff4c0a1c640 1 --2- 172.21.15.116:0/1104282359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a80047c0 0x7ff4bc15b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.812+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/1104282359 >> 172.21.15.116:0/1104282359 conn(0x7ff4bc06a4c0 msgr2=0x7ff4bc0cb0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.812+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/1104282359 shutdown_connections 2023-11-28T17:17:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:05.812+0000 7ff4c0a1c640 1 -- 172.21.15.116:0/1104282359 wait complete. 2023-11-28T17:17:05.834 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-28T17:17:06.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.035+0000 7f69cd667640 1 Processor -- start 2023-11-28T17:17:06.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.035+0000 7f69cd667640 1 -- start start 2023-11-28T17:17:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69cd667640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f69c815ae20 0x7f69c806ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69cd667640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f69c80cdf90 0x7f69c80ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69cd667640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69c80cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69cd667640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c80cf560 con 0x7f69c80cdf90 2023-11-28T17:17:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69cd667640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c8070560 con 0x7f69c815ae20 2023-11-28T17:17:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69cd667640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c8070840 con 0x7f69c80ceb90 2023-11-28T17:17:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69bffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f69c80cdf90 0x7f69c80ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69bffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f69c80cdf90 0x7f69c80ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53992/0 (socket says 172.21.15.116:53992) 2023-11-28T17:17:06.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69bffff640 1 -- 172.21.15.116:0/1741252291 learned_addr learned my addr 172.21.15.116:0/1741252291 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:06.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69bffff640 1 -- 172.21.15.116:0/1741252291 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69c80cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:06.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69bffff640 1 --2- 172.21.15.116:0/1741252291 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69c80cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:06.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69bffff640 1 -- 172.21.15.116:0/1741252291 >> v2:172.21.15.156:3300/0 conn(0x7f69c815ae20 msgr2=0x7f69c806ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-28T17:17:06.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69c7fff640 1 --2- 172.21.15.116:0/1741252291 >> v2:172.21.15.156:3300/0 conn(0x7f69c815ae20 0x7f69c806ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:06.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69bffff640 1 --2- 172.21.15.116:0/1741252291 >> v2:172.21.15.156:3300/0 conn(0x7f69c815ae20 0x7f69c806ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:06.040 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.037+0000 7f69bffff640 1 -- 172.21.15.116:0/1741252291 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c8070b20 con 0x7f69c80cdf90 2023-11-28T17:17:06.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.038+0000 7f69bffff640 1 --2- 172.21.15.116:0/1741252291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cdf90 0x7f69c80ce420 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009870 tx=0x7f69b002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c352a1fbbec99c7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:06.041 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.039+0000 7f69c77fe640 1 -- 172.21.15.116:0/1741252291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b0031640 con 0x7f69c80cdf90 2023-11-28T17:17:06.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.039+0000 7f69c77fe640 1 -- 172.21.15.116:0/1741252291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b00317a0 con 0x7f69c80cdf90 2023-11-28T17:17:06.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.039+0000 7f69c77fe640 1 -- 172.21.15.116:0/1741252291 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b0031b30 con 0x7f69c80cdf90 2023-11-28T17:17:06.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.039+0000 7f69cd667640 1 -- 172.21.15.116:0/1741252291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cdf90 msgr2=0x7f69c80ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:06.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.039+0000 7f69cd667640 1 --2- 172.21.15.116:0/1741252291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cdf90 0x7f69c80ce420 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009870 tx=0x7f69b002b8f0 comp rx=0 tx=0).stop 2023-11-28T17:17:06.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.039+0000 7f69cd667640 1 -- 172.21.15.116:0/1741252291 shutdown_connections 2023-11-28T17:17:06.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.039+0000 7f69cd667640 1 --2- 172.21.15.116:0/1741252291 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69c80cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:06.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.039+0000 7f69cd667640 1 --2- 172.21.15.116:0/1741252291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cdf90 0x7f69c80ce420 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:06.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.039+0000 7f69cd667640 1 --2- 172.21.15.116:0/1741252291 >> v2:172.21.15.156:3300/0 conn(0x7f69c815ae20 0x7f69c806ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:06.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.039+0000 7f69cd667640 1 -- 172.21.15.116:0/1741252291 >> 172.21.15.116:0/1741252291 conn(0x7f69c80b0950 msgr2=0x7f69c80b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T17:17:06.042 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.039+0000 7f69cd667640 1 -- 172.21.15.116:0/1741252291 shutdown_connections 2023-11-28T17:17:06.043 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.039+0000 7f69cd667640 1 -- 172.21.15.116:0/1741252291 wait complete. 2023-11-28T17:17:06.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.041+0000 7f69cd667640 1 Processor -- start 2023-11-28T17:17:06.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.041+0000 7f69cd667640 1 -- start start 2023-11-28T17:17:06.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.042+0000 7f69cd667640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cdf90 0x7f69c8159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:06.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.042+0000 7f69cd667640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69c8151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:06.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.042+0000 7f69cd667640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69c814e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:06.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.042+0000 7f69cd667640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c815a5f0 con 0x7f69c80cdf90 2023-11-28T17:17:06.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.042+0000 7f69cd667640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c815c960 con 0x7f69c815ae20 2023-11-28T17:17:06.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.042+0000 7f69cd667640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c815cc00 con 0x7f69c80ceb90 2023-11-28T17:17:06.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.042+0000 7f69c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cdf90 0x7f69c8159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:06.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.042+0000 7f69c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cdf90 0x7f69c8159da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53998/0 (socket says 172.21.15.116:53998) 2023-11-28T17:17:06.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.042+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 learned_addr learned my addr 172.21.15.116:0/64027060 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-11-28T17:17:06.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.043+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69c8151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:06.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.043+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69c814e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:06.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.043+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69c8151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:06.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.043+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69c8151840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:06.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.043+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69c814e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T17:17:06.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.043+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69c814e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:06.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.043+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b00094d0 con 0x7f69c80cdf90 2023-11-28T17:17:06.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.043+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69c814e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-28T17:17:06.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.043+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cdf90 0x7f69c8159da0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f69b800c7f0 tx=0x7f69b800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:06.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.043+0000 7f69c57fa640 1 -- 172.21.15.116:0/64027060 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b8004220 con 0x7f69c80cdf90 2023-11-28T17:17:06.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.043+0000 7f69c57fa640 1 -- 172.21.15.116:0/64027060 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b8004380 con 0x7f69c80cdf90 2023-11-28T17:17:06.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.043+0000 7f69cd667640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c815b380 con 0x7f69c80cdf90 2023-11-28T17:17:06.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.043+0000 7f69c57fa640 1 -- 172.21.15.116:0/64027060 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b8007840 con 0x7f69c80cdf90 2023-11-28T17:17:06.047 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.043+0000 7f69cd667640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c815d1c0 con 0x7f69c80cdf90 2023-11-28T17:17:06.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.045+0000 7f69cd667640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c809d0b0 con 0x7f69c80cdf90 2023-11-28T17:17:06.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.045+0000 7f69c57fa640 1 -- 172.21.15.116:0/64027060 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) v1 ==== 89549+0+0 (secure 0 0 0) 0x7f69b8007bb0 con 0x7f69c80cdf90 2023-11-28T17:17:06.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.047+0000 7f69c57fa640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69a806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:06.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.047+0000 7f69c57fa640 1 -- 172.21.15.116:0/64027060 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6026+0+0 (secure 0 0 0) 0x7f69b8089bb0 con 0x7f69c80cdf90 2023-11-28T17:17:06.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.047+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69a806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T17:17:06.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.048+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69a806d790 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f69b0036000 tx=0x7f69b003c040 comp rx=0 tx=0).ready entity=mgr.4098 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T17:17:06.052 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.051+0000 7f69c57fa640 1 -- 172.21.15.116:0/64027060 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f69b80548e0 con 0x7f69c80cdf90 2023-11-28T17:17:06.134 INFO:tasks.ceph.osd.0:Sent signal 15 2023-11-28T17:17:06.134 INFO:tasks.ceph.osd.1:Sent signal 15 2023-11-28T17:17:06.135 INFO:tasks.ceph.osd.2:Sent signal 15 2023-11-28T17:17:06.135 INFO:tasks.ceph.osd.3:Sent signal 15 2023-11-28T17:17:06.135 INFO:tasks.ceph.osd.4:Sent signal 15 2023-11-28T17:17:06.136 INFO:tasks.ceph.osd.5:Sent signal 15 2023-11-28T17:17:06.136 INFO:tasks.ceph.osd.6:Sent signal 15 2023-11-28T17:17:06.136 INFO:tasks.ceph.osd.7:Sent signal 15 2023-11-28T17:17:06.136 INFO:tasks.ceph.mds.c:Sent signal 15 2023-11-28T17:17:06.137 INFO:tasks.ceph.mon.a:Sent signal 15 2023-11-28T17:17:06.137 INFO:tasks.ceph.mon.b:Sent signal 15 2023-11-28T17:17:06.137 INFO:tasks.ceph.mon.c:Sent signal 15 2023-11-28T17:17:06.138 INFO:tasks.ceph.mgr.y:Sent signal 15 2023-11-28T17:17:06.138 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-11-28T17:17:06.139 INFO:tasks.ceph.osd.4.smithi156.stderr:2023-11-28T17:17:06.136+0000 7f9b46f3d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 88863) UID: 0 2023-11-28T17:17:06.139 INFO:tasks.ceph.osd.4.smithi156.stderr:2023-11-28T17:17:06.136+0000 7f9b46f3d640 -1 osd.4 52 *** Got signal Terminated *** 2023-11-28T17:17:06.139 INFO:tasks.ceph.osd.4.smithi156.stderr:2023-11-28T17:17:06.136+0000 7f9b46f3d640 -1 osd.4 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:17:06.140 INFO:tasks.ceph.osd.5.smithi156.stderr:2023-11-28T17:17:06.136+0000 7fa6b9035640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 88865) UID: 0 2023-11-28T17:17:06.140 INFO:tasks.ceph.osd.5.smithi156.stderr:2023-11-28T17:17:06.136+0000 7fa6b9035640 -1 osd.5 52 *** Got signal Terminated *** 2023-11-28T17:17:06.140 INFO:tasks.ceph.osd.5.smithi156.stderr:2023-11-28T17:17:06.136+0000 7fa6b9035640 -1 osd.5 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:17:06.140 INFO:tasks.ceph.osd.6.smithi156.stderr:2023-11-28T17:17:06.136+0000 7f94fbfce640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 88867) UID: 0 2023-11-28T17:17:06.140 INFO:tasks.ceph.osd.6.smithi156.stderr:2023-11-28T17:17:06.136+0000 7f94fbfce640 -1 osd.6 52 *** Got signal Terminated *** 2023-11-28T17:17:06.141 INFO:tasks.ceph.osd.6.smithi156.stderr:2023-11-28T17:17:06.136+0000 7f94fbfce640 -1 osd.6 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:17:06.141 INFO:tasks.ceph.osd.7.smithi156.stderr:2023-11-28T17:17:06.137+0000 7fd1d1bf0640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 88868) UID: 0 2023-11-28T17:17:06.141 INFO:tasks.ceph.osd.7.smithi156.stderr:2023-11-28T17:17:06.137+0000 7fd1d1bf0640 -1 osd.7 52 *** Got signal Terminated *** 2023-11-28T17:17:06.141 INFO:tasks.ceph.osd.7.smithi156.stderr:2023-11-28T17:17:06.137+0000 7fd1d1bf0640 -1 osd.7 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:17:06.141 INFO:tasks.ceph.mon.b.smithi156.stderr:2023-11-28T17:17:06.137+0000 7f8cb8101640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 88294) UID: 0 2023-11-28T17:17:06.142 INFO:tasks.ceph.mon.b.smithi156.stderr:2023-11-28T17:17:06.137+0000 7f8cb8101640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-11-28T17:17:06.142 INFO:tasks.ceph.mon.c.smithi156.stderr:2023-11-28T17:17:06.138+0000 7f93eabe4640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 88295) UID: 0 2023-11-28T17:17:06.142 INFO:tasks.ceph.mon.c.smithi156.stderr:2023-11-28T17:17:06.138+0000 7f93eabe4640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-11-28T17:17:06.142 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-11-28T17:17:06.134+0000 7f4484f01640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 88916) UID: 0 2023-11-28T17:17:06.142 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-11-28T17:17:06.134+0000 7f4484f01640 -1 osd.0 52 *** Got signal Terminated *** 2023-11-28T17:17:06.143 INFO:tasks.ceph.osd.0.smithi116.stderr:2023-11-28T17:17:06.134+0000 7f4484f01640 -1 osd.0 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:17:06.143 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-11-28T17:17:06.135+0000 7f6ac39f2640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 88923) UID: 0 2023-11-28T17:17:06.143 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-11-28T17:17:06.135+0000 7f6ac39f2640 -1 osd.1 52 *** Got signal Terminated *** 2023-11-28T17:17:06.143 INFO:tasks.ceph.osd.1.smithi116.stderr:2023-11-28T17:17:06.135+0000 7f6ac39f2640 -1 osd.1 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:17:06.143 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-11-28T17:17:06.135+0000 7f8d6a9da640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 88925) UID: 0 2023-11-28T17:17:06.143 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-11-28T17:17:06.135+0000 7f8d6a9da640 -1 osd.2 52 *** Got signal Terminated *** 2023-11-28T17:17:06.143 INFO:tasks.ceph.osd.2.smithi116.stderr:2023-11-28T17:17:06.135+0000 7f8d6a9da640 -1 osd.2 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:17:06.144 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-11-28T17:17:06.135+0000 7fa7d825a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 88931) UID: 0 2023-11-28T17:17:06.144 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-11-28T17:17:06.135+0000 7fa7d825a640 -1 osd.3 52 *** Got signal Terminated *** 2023-11-28T17:17:06.144 INFO:tasks.ceph.osd.3.smithi116.stderr:2023-11-28T17:17:06.135+0000 7fa7d825a640 -1 osd.3 52 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T17:17:06.144 INFO:tasks.ceph.mds.c.smithi116.stderr:2023-11-28T17:17:06.137+0000 7fec26ac7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mds -f --cluster ceph -i c (PID: 105868) UID: 0 2023-11-28T17:17:06.144 INFO:tasks.ceph.mds.c.smithi116.stderr:2023-11-28T17:17:06.137+0000 7fec26ac7640 -1 mds.c *** got signal Terminated *** 2023-11-28T17:17:06.145 INFO:tasks.ceph.mon.a.smithi116.stderr:2023-11-28T17:17:06.137+0000 7f2317c23640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 88670) UID: 0 2023-11-28T17:17:06.145 INFO:tasks.ceph.mon.a.smithi116.stderr:2023-11-28T17:17:06.137+0000 7f2317c23640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-28T17:17:06.145 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:17:06.138+0000 7fb787e71640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i y (PID: 88673) UID: 0 2023-11-28T17:17:06.145 INFO:tasks.ceph.mgr.y.smithi116.stderr:2023-11-28T17:17:06.138+0000 7fb787e71640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-11-28T17:17:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.141+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cdf90 msgr2=0x7f69c8159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-11-28T17:17:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.141+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cdf90 msgr2=0x7f69c8159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:17:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.141+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cdf90 0x7f69c8159da0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f69b800c7f0 tx=0x7f69b800ccb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:17:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.141+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cdf90 0x7f69c8159da0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f69b800c7f0 tx=0x7f69b800ccb0 comp rx=0 tx=0).stop 2023-11-28T17:17:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.141+0000 7f69c57fa640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cdf90 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:17:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.141+0000 7f69c57fa640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69c80cdf90 0x7f69c8159da0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:06.146 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.141+0000 7f69c57fa640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a80756d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.141+0000 7f69c57fa640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a8077e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.141+0000 7f69c57fa640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a807ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.141+0000 7f69c57fa640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b730 con 0x7f69a80787a0 2023-11-28T17:17:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.141+0000 7f69c57fa640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b870 con 0x7f69c815ae20 2023-11-28T17:17:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.141+0000 7f69c57fa640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807ba30 con 0x7f69c80ceb90 2023-11-28T17:17:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.142+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a8077e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.142+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a8077e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.142+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a807ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.142+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a807ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.142+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a80756d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:06.147 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.142+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a80756d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:06.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.189+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69a806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2023-11-28T17:17:06.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.189+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69a806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T17:17:06.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.189+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69a806d790 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f69b0036000 tx=0x7f69b003c040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T17:17:06.190 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.189+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69a806d790 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f69b0036000 tx=0x7f69b003c040 comp rx=0 tx=0).stop 2023-11-28T17:17:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.189+0000 7f69c57fa640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69a806d790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-11-28T17:17:06.191 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.189+0000 7f69c57fa640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69a806d790 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:06.340 INFO:tasks.ceph.mgr.x.smithi156.stderr:daemon-helper: command crashed with signal 15 2023-11-28T17:17:06.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.343+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a8077e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:06.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.343+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a807ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:06.344 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.343+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a8077e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.343+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a807ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.344+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a80756d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:06.345 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.344+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a80756d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:06.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.744+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a807ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:06.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.744+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a807ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:06.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.744+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a8077e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:06.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.744+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a8077e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:06.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.745+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a80756d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:06.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:06.745+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a80756d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:07.048 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:07.047+0000 7f69beffd640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:07.663 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:07.047+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6832/1077400178 2023-11-28T17:17:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:07.047+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:07.248+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6832/1077400178 2023-11-28T17:17:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:07.248+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:07.546+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a807ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:07.546+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a807ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:07.546+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a8077e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:07.546+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a8077e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:07.546+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a80756d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:07.546+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a80756d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:07.649+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6832/1077400178 2023-11-28T17:17:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:07.649+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:08.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:08.450+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6832/1077400178 2023-11-28T17:17:08.452 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:08.450+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:09.044 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a80756d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:17:09.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a80756d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:09.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a8077e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:17:09.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a8077e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:09.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a807ac30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:17:09.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a807ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:09.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:09.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:09.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:09.045 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b730 con 0x7f69a80787a0 2023-11-28T17:17:09.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b870 con 0x7f69c815ae20 2023-11-28T17:17:09.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807ba30 con 0x7f69c80ceb90 2023-11-28T17:17:09.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:09.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:09.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:09.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400a350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:09.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:09.046 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.043+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.244+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:09.246 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.244+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.244+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.244+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.244+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400a350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:09.247 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.244+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:09.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.645+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:09.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.645+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:09.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.645+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:09.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.645+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:09.647 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.646+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400a350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:09.648 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:09.646+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:10.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:10.053+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6832/1077400178 2023-11-28T17:17:10.054 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:10.053+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:17:10.447 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:10.446+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:10.446+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:10.447+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:10.448 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:10.447+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:10.447+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400a350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:10.449 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:10.447+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:12.049 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:12.048+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:12.050 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:12.048+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:17:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:12.049+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:12.049+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400a350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:12.049+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:17:12.051 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:12.049+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:17:13.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:13.256+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6832/1077400178 2023-11-28T17:17:13.257 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:13.256+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:17:15.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.252+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:15.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.252+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:17:15.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.252+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400a350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:15.253 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.252+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:17:15.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.252+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:15.254 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.252+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:17:15.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:17:15.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:15.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400a350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:17:15.795 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400a350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:15.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:17:15.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:15.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:15.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:15.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:15.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b730 con 0x7f69a80787a0 2023-11-28T17:17:15.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b870 con 0x7f69c815ae20 2023-11-28T17:17:15.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807ba30 con 0x7f69c80ceb90 2023-11-28T17:17:15.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:15.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:15.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:15.796 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:15.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:15.797 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.794+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:15.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.995+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:15.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.995+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:15.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.995+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:15.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.995+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:15.996 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.995+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:15.997 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:15.995+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:16.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:16.396+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:16.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:16.396+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:16.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:16.396+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:16.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:16.396+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:16.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:16.396+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:16.397 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:16.396+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:17.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:17.197+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:17.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:17.197+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:17.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:17.197+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:17.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:17.197+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:17.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:17.197+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:17.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:17.197+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:18.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:18.798+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:18.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:18.798+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:17:18.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:18.798+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:18.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:18.798+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:18.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:18.798+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:17:18.800 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:18.798+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:17:19.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:19.663+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6832/1077400178 2023-11-28T17:17:19.664 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:19.663+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-11-28T17:17:22.004 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:22.002+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:22.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:22.002+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:17:22.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:22.002+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:22.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:22.002+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:17:22.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:22.002+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:22.005 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:22.002+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:17:25.206 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:17:25.215 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:17:25.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.918+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:17:25.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:25.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:17:25.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:25.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:17:25.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:25.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:25.921 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:25.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:25.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b730 con 0x7f69a80787a0 2023-11-28T17:17:25.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b870 con 0x7f69c815ae20 2023-11-28T17:17:25.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807ba30 con 0x7f69c80ceb90 2023-11-28T17:17:25.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:25.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:25.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:25.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:25.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:25.922 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:25.919+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:26.120+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:26.120+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:26.121+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:26.121+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:26.121+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:26.121+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:26.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:26.521+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:26.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:26.521+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:26.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:26.522+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:26.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:26.522+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:26.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:26.522+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:26.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:26.522+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:27.322+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:27.322+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:27.324 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:27.323+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:27.323+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:27.323+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:27.325 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:27.323+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:28.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:28.924+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:28.925 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:28.924+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:17:28.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:28.924+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:28.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:28.924+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:17:28.926 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:28.925+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:28.927 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:28.925+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:17:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:32.128+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:32.128+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:17:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:32.128+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:32.130 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:32.129+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:17:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:32.129+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:32.129+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:17:32.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:32.476+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6832/1077400178 2023-11-28T17:17:32.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:32.476+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-11-28T17:17:38.536 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:38.534+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:38.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:38.534+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-11-28T17:17:38.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:38.535+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:38.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:38.535+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-11-28T17:17:38.537 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:38.536+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:38.538 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:38.536+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-11-28T17:17:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:17:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:17:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:17:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:17:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:17:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b730 con 0x7f69a80787a0 2023-11-28T17:17:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b870 con 0x7f69c815ae20 2023-11-28T17:17:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807ba30 con 0x7f69c80ceb90 2023-11-28T17:17:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.107+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.108+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:41.110 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.108+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:17:41.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.308+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:41.309 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.308+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:41.310 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.309+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:41.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.309+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:41.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.309+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:41.311 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.309+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:17:41.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.709+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:41.710 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.709+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:41.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.710+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:41.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.710+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:41.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.710+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:41.711 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:41.710+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:17:42.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:42.510+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:42.512 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:42.510+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:42.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:42.511+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:42.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:42.511+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:42.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:42.511+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:42.513 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:42.511+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:17:44.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:44.112+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:44.113 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:44.112+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:17:44.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:44.112+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:44.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:44.112+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:17:44.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:44.113+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:44.114 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:44.113+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:17:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:47.316+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:47.316+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:17:47.318 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:47.316+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:47.317+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:17:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:47.317+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:47.317+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:17:47.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:47.491+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6832/1077400178 2023-11-28T17:17:47.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:47.491+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-11-28T17:17:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:53.722+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:17:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:53.722+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-11-28T17:17:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:53.723+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:17:53.724 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:53.723+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-11-28T17:17:53.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:53.724+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:17:53.725 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:17:53.724+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-11-28T17:17:55.346 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:17:55.355 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:18:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:02.507+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6832/1077400178 2023-11-28T17:18:02.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:02.507+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-11-28T17:18:03.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:18:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:18:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:18:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:18:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:18:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:18:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:18:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:18:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:18:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b730 con 0x7f69a80787a0 2023-11-28T17:18:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b870 con 0x7f69c815ae20 2023-11-28T17:18:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807ba30 con 0x7f69c80ceb90 2023-11-28T17:18:03.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:03.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:18:03.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:03.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:18:03.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.889+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:03.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:03.890+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:18:04.091 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:04.090+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:04.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:04.090+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:18:04.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:04.090+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:04.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:04.090+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:18:04.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:04.091+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:04.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:04.091+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:18:04.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:04.491+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:04.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:04.491+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:18:04.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:04.491+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:04.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:04.491+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:18:04.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:04.492+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:04.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:04.492+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:18:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:05.292+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:05.292+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:18:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:05.293+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:05.293+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:18:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:05.293+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:05.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:05.293+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:18:06.895 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:06.894+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:06.894+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:18:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:06.894+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:06.894+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:18:06.896 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:06.895+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:06.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:06.895+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:18:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:10.098+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:10.098+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:18:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:10.098+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:10.098+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:18:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:10.098+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:10.100 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:10.099+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:18:16.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:16.505+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:16.506 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:16.505+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-11-28T17:18:16.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:16.505+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:16.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:16.505+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-11-28T17:18:16.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:16.505+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:16.507 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:16.505+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-11-28T17:18:17.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:17.523+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6832/1077400178 2023-11-28T17:18:17.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:17.523+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-11-28T17:18:25.491 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:18:25.501 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:18:29.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:29.318+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:29.319 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:29.318+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-11-28T17:18:29.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:29.318+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:29.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:29.318+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:29.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:29.318+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-11-28T17:18:29.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:29.318+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-11-28T17:18:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:32.537+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6832/1077400178 2023-11-28T17:18:32.539 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:32.537+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-11-28T17:18:33.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.889+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:18:33.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.889+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:18:33.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.889+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:18:33.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.889+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:18:33.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.889+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:18:33.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.889+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:18:33.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.889+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:18:33.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.889+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:18:33.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.889+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:18:33.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.889+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b730 con 0x7f69a80787a0 2023-11-28T17:18:33.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.889+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b870 con 0x7f69c815ae20 2023-11-28T17:18:33.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.889+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807ba30 con 0x7f69c80ceb90 2023-11-28T17:18:33.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.890+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:33.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.890+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:33.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.890+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:18:33.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.890+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:18:33.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.890+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:33.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:33.890+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:18:34.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:34.091+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:34.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:34.091+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:18:34.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:34.091+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:34.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:34.091+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:18:34.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:34.091+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:34.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:34.091+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:18:34.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:34.492+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:34.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:34.492+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:34.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:34.492+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:18:34.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:34.492+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:18:34.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:34.493+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:34.495 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:34.493+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:18:35.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:35.293+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:35.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:35.293+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:18:35.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:35.293+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:35.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:35.293+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:18:35.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:35.294+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:35.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:35.294+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:18:36.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:36.895+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:36.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:36.895+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:18:36.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:36.895+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:36.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:36.895+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:36.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:36.895+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:18:36.897 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:36.895+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T17:18:40.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:40.099+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:40.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:40.099+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:18:40.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:40.099+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:40.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:40.099+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:18:40.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:40.099+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:40.101 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:40.099+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-11-28T17:18:46.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:46.506+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:46.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:46.506+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-11-28T17:18:46.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:46.506+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:46.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:46.506+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-11-28T17:18:46.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:46.507+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:46.508 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:46.507+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2023-11-28T17:18:47.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:47.553+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6832/1077400178 2023-11-28T17:18:47.554 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:47.553+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-11-28T17:18:55.646 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:18:55.657 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:18:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:59.319+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:18:59.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:59.319+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-11-28T17:18:59.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:59.319+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:18:59.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:59.319+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:18:59.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:59.319+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-11-28T17:18:59.321 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:18:59.319+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-11-28T17:19:02.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:02.568+0000 7f69bffff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 msgr2=0x7f69c8159da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6832/1077400178 2023-11-28T17:19:02.570 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:02.568+0000 7f69bffff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:6832/1077400178,v1:172.21.15.116:6833/1077400178] conn(0x7f69a806b2d0 0x7f69c8159da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2023-11-28T17:19:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:19:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:19:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T17:19:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:03.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69bf7fe640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b730 con 0x7f69a80787a0 2023-11-28T17:19:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807b870 con 0x7f69c815ae20 2023-11-28T17:19:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69bf7fe640 1 -- 172.21.15.116:0/64027060 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a807ba30 con 0x7f69c80ceb90 2023-11-28T17:19:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:03.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:03.890+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:04.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:04.091+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:04.092 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:04.091+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:04.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:04.091+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:04.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:04.092+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:04.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:04.092+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:04.093 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:04.092+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:04.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:04.492+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:04.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:04.492+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:04.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:04.493+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:04.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:04.493+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:04.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:04.493+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:04.494 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:04.493+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:05.294 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:05.293+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 msgr2=0x7f69a4005620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:05.293+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a80787a0 0x7f69a4005620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:05.294+0000 7f69c7fff640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 msgr2=0x7f69a4007b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:05.294+0000 7f69c7fff640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69c80ceb90 0x7f69a4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:05.295 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:05.294+0000 7f69cce66640 1 -- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 msgr2=0x7f69a400ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:05.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:05.294+0000 7f69cce66640 1 --2- 172.21.15.116:0/64027060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69c815ae20 0x7f69a400ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:05.941 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:19:05.943 INFO:tasks.cephfs_test_runner:Check that mds evicts blocklisted client ... ERROR 2023-11-28T17:19:05.944 INFO:tasks.cephfs_test_runner:ERROR 2023-11-28T17:19:05.944 INFO:tasks.cephfs_test_runner: 2023-11-28T17:19:05.945 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-28T17:19:05.945 INFO:tasks.cephfs_test_runner:ERROR: test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap) 2023-11-28T17:19:05.945 INFO:tasks.cephfs_test_runner:Check that mds evicts blocklisted client 2023-11-28T17:19:05.945 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-28T17:19:05.945 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-28T17:19:05.945 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr 2023-11-28T17:19:05.945 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2023-11-28T17:19:05.945 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read 2023-11-28T17:19:05.945 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2023-11-28T17:19:05.945 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2023-11-28T17:19:05.945 INFO:tasks.cephfs_test_runner: 2023-11-28T17:19:05.945 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2023-11-28T17:19:05.945 INFO:tasks.cephfs_test_runner: 2023-11-28T17:19:05.946 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-28T17:19:05.946 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/test_sessionmap.py", line 213, in test_session_evict_blocklisted 2023-11-28T17:19:05.946 INFO:tasks.cephfs_test_runner: self.mount_a.run_shell(["touch", "d1/f0"]) 2023-11-28T17:19:05.946 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/mount.py", line 784, in run_shell 2023-11-28T17:19:05.946 INFO:tasks.cephfs_test_runner: return self.client_remote.run(args=args, cwd=cwd, timeout=timeout, 2023-11-28T17:19:05.946 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run 2023-11-28T17:19:05.946 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2023-11-28T17:19:05.946 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run 2023-11-28T17:19:05.946 INFO:tasks.cephfs_test_runner: r.wait() 2023-11-28T17:19:05.946 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 148, in wait 2023-11-28T17:19:05.946 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2023-11-28T17:19:05.946 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2023-11-28T17:19:05.946 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2023-11-28T17:19:05.946 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise 2023-11-28T17:19:05.947 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2023-11-28T17:19:05.947 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2023-11-28T17:19:05.947 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 323, in copy_file_to 2023-11-28T17:19:05.947 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2023-11-28T17:19:05.947 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 276, in copy_to_log 2023-11-28T17:19:05.947 INFO:tasks.cephfs_test_runner: for line in f: 2023-11-28T17:19:05.947 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ 2023-11-28T17:19:05.947 INFO:tasks.cephfs_test_runner: line = self.readline() 2023-11-28T17:19:05.947 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline 2023-11-28T17:19:05.947 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2023-11-28T17:19:05.947 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1376, in _read 2023-11-28T17:19:05.947 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2023-11-28T17:19:05.947 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 749, in recv_stderr 2023-11-28T17:19:05.947 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2023-11-28T17:19:05.948 INFO:tasks.cephfs_test_runner:socket.timeout 2023-11-28T17:19:05.948 INFO:tasks.cephfs_test_runner: 2023-11-28T17:19:05.948 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-28T17:19:05.948 INFO:tasks.cephfs_test_runner:ERROR: test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap) 2023-11-28T17:19:05.948 INFO:tasks.cephfs_test_runner:Check that mds evicts blocklisted client 2023-11-28T17:19:05.948 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-28T17:19:05.948 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-28T17:19:05.948 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/cephfs_test_case.py", line 208, in tearDown 2023-11-28T17:19:05.948 INFO:tasks.cephfs_test_runner: self.mds_cluster.delete_all_filesystems() 2023-11-28T17:19:05.948 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/filesystem.py", line 494, in delete_all_filesystems 2023-11-28T17:19:05.948 INFO:tasks.cephfs_test_runner: Filesystem(ctx=self._ctx, fscid=fs['id']).destroy() 2023-11-28T17:19:05.948 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/filesystem.py", line 547, in __init__ 2023-11-28T17:19:05.948 INFO:tasks.cephfs_test_runner: self.getinfo(refresh = True) 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/filesystem.py", line 580, in getinfo 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: self.get_pool_names(status = status, refresh = refresh) 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/filesystem.py", line 942, in get_pool_names 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: osd_map = self.mon_manager.get_osd_dump_json() 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: out = self.raw_cluster_cmd('osd', 'dump', '--format=json') 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: return self.run_cluster_cmd(**kwargs).stdout.getvalue() 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: r.wait() 2023-11-28T17:19:05.949 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait 2023-11-28T17:19:05.950 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2023-11-28T17:19:05.950 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status 2023-11-28T17:19:05.950 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2023-11-28T17:19:05.950 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2023-11-28T17:19:05.950 INFO:tasks.cephfs_test_runner: 2023-11-28T17:19:05.950 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-28T17:19:05.950 INFO:tasks.cephfs_test_runner:Ran 2 tests in 566.663s 2023-11-28T17:19:05.950 INFO:tasks.cephfs_test_runner: 2023-11-28T17:19:05.950 INFO:tasks.cephfs_test_runner:FAILED (errors=2) 2023-11-28T17:19:05.950 INFO:tasks.cephfs_test_runner: 2023-11-28T17:19:05.950 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-28T17:19:05.950 INFO:tasks.cephfs_test_runner:ERROR: test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap) 2023-11-28T17:19:05.951 INFO:tasks.cephfs_test_runner:Check that mds evicts blocklisted client 2023-11-28T17:19:05.951 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-28T17:19:05.951 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-28T17:19:05.951 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr 2023-11-28T17:19:05.951 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2023-11-28T17:19:05.951 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read 2023-11-28T17:19:05.951 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2023-11-28T17:19:05.951 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2023-11-28T17:19:05.951 INFO:tasks.cephfs_test_runner: 2023-11-28T17:19:05.951 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2023-11-28T17:19:05.951 INFO:tasks.cephfs_test_runner: 2023-11-28T17:19:05.951 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-28T17:19:05.951 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/test_sessionmap.py", line 213, in test_session_evict_blocklisted 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: self.mount_a.run_shell(["touch", "d1/f0"]) 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/mount.py", line 784, in run_shell 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: return self.client_remote.run(args=args, cwd=cwd, timeout=timeout, 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: r.wait() 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 148, in wait 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2023-11-28T17:19:05.952 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 323, in copy_file_to 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 276, in copy_to_log 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner: for line in f: 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner: line = self.readline() 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1376, in _read 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 749, in recv_stderr 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner:socket.timeout 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner: 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-28T17:19:05.953 INFO:tasks.cephfs_test_runner:ERROR: test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap) 2023-11-28T17:19:05.954 INFO:tasks.cephfs_test_runner:Check that mds evicts blocklisted client 2023-11-28T17:19:05.954 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-28T17:19:05.954 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-28T17:19:05.954 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/cephfs_test_case.py", line 208, in tearDown 2023-11-28T17:19:05.954 INFO:tasks.cephfs_test_runner: self.mds_cluster.delete_all_filesystems() 2023-11-28T17:19:05.954 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/filesystem.py", line 494, in delete_all_filesystems 2023-11-28T17:19:05.954 INFO:tasks.cephfs_test_runner: Filesystem(ctx=self._ctx, fscid=fs['id']).destroy() 2023-11-28T17:19:05.954 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/filesystem.py", line 547, in __init__ 2023-11-28T17:19:05.954 INFO:tasks.cephfs_test_runner: self.getinfo(refresh = True) 2023-11-28T17:19:05.954 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/filesystem.py", line 580, in getinfo 2023-11-28T17:19:05.954 INFO:tasks.cephfs_test_runner: self.get_pool_names(status = status, refresh = refresh) 2023-11-28T17:19:05.954 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs/filesystem.py", line 942, in get_pool_names 2023-11-28T17:19:05.954 INFO:tasks.cephfs_test_runner: osd_map = self.mon_manager.get_osd_dump_json() 2023-11-28T17:19:05.954 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2537, in get_osd_dump_json 2023-11-28T17:19:05.955 INFO:tasks.cephfs_test_runner: out = self.raw_cluster_cmd('osd', 'dump', '--format=json') 2023-11-28T17:19:05.955 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd 2023-11-28T17:19:05.955 INFO:tasks.cephfs_test_runner: return self.run_cluster_cmd(**kwargs).stdout.getvalue() 2023-11-28T17:19:05.955 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd 2023-11-28T17:19:05.955 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2023-11-28T17:19:05.955 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run 2023-11-28T17:19:05.955 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2023-11-28T17:19:05.955 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run 2023-11-28T17:19:05.955 INFO:tasks.cephfs_test_runner: r.wait() 2023-11-28T17:19:05.955 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait 2023-11-28T17:19:05.955 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2023-11-28T17:19:05.955 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status 2023-11-28T17:19:05.955 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2023-11-28T17:19:05.955 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2023-11-28T17:19:05.956 INFO:tasks.cephfs_test_runner: 2023-11-28T17:19:05.956 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap), test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap) 2023-11-28T17:19:06.031 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=22f534bcf0f146c795677ca44fc2d08b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap), test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap) 2023-11-28T17:19:06.033 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-11-28T17:19:06.042 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-11-28T17:19:06.062 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-11-28T17:19:06.063 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:19:06.063 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:19:06.127 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054299,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T17:19:06.128 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-11-28T17:19:06.128 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:19:06.181 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054299,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249408k,nr_inodes=812352,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498824k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-11-28T17:19:06.182 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:19:06.182 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:19:06.242 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T17:19:06.242 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-11-28T17:19:06.242 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-28T17:19:06.299 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-11-28T17:19:06.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:19:06.299 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2023-11-28T17:19:06.352 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-28T17:19:06.352 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-28T17:19:06.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:19:06.354 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-28T17:19:06.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:19:06.450 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-28T17:19:06.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:19:06.541 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link delete ceph-brx 2023-11-28T17:19:06.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:19:06.691 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2023-11-28T17:19:06.751 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-28T17:19:06.751 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-28T17:19:06.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:19:06.752 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-28T17:19:06.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:19:06.854 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-28T17:19:06.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-28T17:19:06.946 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2023-11-28T17:19:07.065 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-28T17:19:07.079 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-28T17:19:07.079 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-28T17:19:07.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.269+0000 7f182bf4c640 1 Processor -- start 2023-11-28T17:19:07.271 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.270+0000 7f182bf4c640 1 -- start start 2023-11-28T17:19:07.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.271+0000 7f182bf4c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 0x7f182415f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:07.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.271+0000 7f182bf4c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18240703f0 0x7f1824071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:07.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.271+0000 7f182bf4c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f18240721b0 2023-11-28T17:19:07.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.271+0000 7f182bf4c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f18240703f0 2023-11-28T17:19:07.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.271+0000 7f182bf4c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f1824144430 2023-11-28T17:19:07.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.271+0000 7f182a4e9640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18240721b0 legacy=0x7f182406c250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:07.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.271+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18240703f0 msgr2=0x7f1824071c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:07.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.271+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18240703f0 0x7f1824071c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:07.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.271+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 msgr2=0x7f182415f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:07.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.271+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 0x7f182415f7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:07.473 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.472+0000 7f182a4e9640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18240721b0 legacy=0x7f182406c250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:07.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.472+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18240703f0 msgr2=0x7f1824071c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:07.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.472+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18240703f0 0x7f1824071c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:07.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.473+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 msgr2=0x7f182415f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:07.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.473+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 0x7f182415f7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:07.874 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.873+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18240703f0 msgr2=0x7f1824071c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:07.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.873+0000 7f182a4e9640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18240721b0 legacy=0x7f182406c250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:07.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.873+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18240703f0 0x7f1824071c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:07.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.874+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 msgr2=0x7f182415f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:07.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:07.874+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 0x7f182415f7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:08.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:08.674+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18240703f0 msgr2=0x7f1824071c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:08.675 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:08.674+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18240703f0 0x7f1824071c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:08.674+0000 7f182a4e9640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18240721b0 legacy=0x7f182406c250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:08.675+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 msgr2=0x7f182415f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:08.676 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:08.675+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 0x7f182415f7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:10.273 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.272+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 msgr2=0x7f182415f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:10.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.272+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 0x7f182415f7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:10.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.272+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18240703f0 msgr2=0x7f1824071c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:10.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.272+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18240703f0 0x7f1824071c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:10.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.272+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18240721b0 legacy=0x7f182406c250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:10.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.272+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f18080036c0 0x7f1808005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:10.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.272+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f1808006190 2023-11-28T17:19:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.272+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f18080036c0 2023-11-28T17:19:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.272+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f18080036c0 msgr2=0x7f1808005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.272+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006190 legacy=0x7f1808008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:10.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.272+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f18080036c0 0x7f1808005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:10.474 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.473+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f18080036c0 msgr2=0x7f1808005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:10.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.473+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f18080036c0 0x7f1808005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:10.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.474+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006190 legacy=0x7f1808008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:10.875 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.874+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f18080036c0 msgr2=0x7f1808005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:10.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.874+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f18080036c0 0x7f1808005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:10.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:10.875+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006190 legacy=0x7f1808008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:11.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:11.676+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f18080036c0 msgr2=0x7f1808005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:11.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:11.676+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f18080036c0 0x7f1808005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:11.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:11.676+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006190 legacy=0x7f1808008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:13.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.273+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f18080036c0 msgr2=0x7f1808005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.273+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f18080036c0 0x7f1808005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.273+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006190 legacy=0x7f1808008650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.273+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:19:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.273+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808009ef0 0x7f180800a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.273+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 0x7f180800a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.273+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f1808006190 2023-11-28T17:19:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.273+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808009ef0 2023-11-28T17:19:13.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.273+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f1824144430 2023-11-28T17:19:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.273+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808009ef0 msgr2=0x7f180800a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.273+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808009ef0 0x7f180800a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.273+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006190 legacy=0x7f1808008d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.273+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 msgr2=0x7f180800a910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:13.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.273+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 0x7f180800a910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.474+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808009ef0 msgr2=0x7f180800a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.474+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006190 legacy=0x7f1808008d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.474+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808009ef0 0x7f180800a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.474+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 msgr2=0x7f180800a910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.474+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 0x7f180800a910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:13.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.875+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 msgr2=0x7f180800a910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:13.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.875+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808009ef0 msgr2=0x7f180800a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:13.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.875+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006190 legacy=0x7f1808008d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:13.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.875+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808009ef0 0x7f180800a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:13.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:13.875+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 0x7f180800a910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:14.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:14.677+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808009ef0 msgr2=0x7f180800a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:14.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:14.677+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 msgr2=0x7f180800a910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:14.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:14.677+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006190 legacy=0x7f1808008d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:14.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:14.677+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808009ef0 0x7f180800a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:14.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:14.677+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 0x7f180800a910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:16.274 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.273+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 msgr2=0x7f180800a910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.273+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1824144430 0x7f180800a910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.273+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808009ef0 msgr2=0x7f180800a330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.273+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808009ef0 0x7f180800a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.273+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006190 legacy=0x7f1808008d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.273+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808009490 0x7f1808007640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.273+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f18080098a0 2023-11-28T17:19:16.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.273+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808009490 2023-11-28T17:19:16.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.273+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808009490 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:16.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.273+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808009490 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:16.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.273+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080098a0 legacy=0x7f1808007b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:16.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.474+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808009490 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:16.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.474+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808009490 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:16.475 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.474+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080098a0 legacy=0x7f1808007b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:16.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.875+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808009490 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:16.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.875+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808009490 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:16.876 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:16.875+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080098a0 legacy=0x7f1808007b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:17.676+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808009490 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:17.676+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808009490 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:17.676+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080098a0 legacy=0x7f1808007b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:19.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.274+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808009490 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:19.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.274+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808009490 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:19.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.274+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080098a0 legacy=0x7f1808007b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:19.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.274+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:19:19.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.274+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:19.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.274+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808007b80 0x7f1808007fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:19.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.274+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f180800a330 2023-11-28T17:19:19.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.274+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808007b80 2023-11-28T17:19:19.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.274+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f180800a910 2023-11-28T17:19:19.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.274+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a330 legacy=0x7f1808008d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:19.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.274+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808007b80 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:19.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.274+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:19.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.274+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808007b80 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:19.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.274+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:19.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.475+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a330 legacy=0x7f1808008d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:19.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.475+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808007b80 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:19.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.475+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808007b80 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:19.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.475+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:19.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.475+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:19.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.876+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808007b80 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:19.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.876+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a330 legacy=0x7f1808008d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:19.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.876+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808007b80 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:19.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.876+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:19.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:19.876+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:20.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:20.678+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808007b80 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:20.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:20.678+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a330 legacy=0x7f1808008d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:20.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:20.678+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:20.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:20.678+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808007b80 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:20.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:20.678+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:22.275 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.274+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:22.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.274+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:22.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.274+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808007b80 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:22.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.274+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808007b80 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:22.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.274+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a330 legacy=0x7f1808008d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:22.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.274+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080084f0 0x7f1808011080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:22.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.274+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f1808008930 2023-11-28T17:19:22.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.274+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f18080084f0 2023-11-28T17:19:22.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.274+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008930 legacy=0x7f18080135d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:22.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.274+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080084f0 msgr2=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:22.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.274+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080084f0 0x7f1808011080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.475+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080084f0 msgr2=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:22.476 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.475+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008930 legacy=0x7f18080135d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:22.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.475+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080084f0 0x7f1808011080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.876+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080084f0 msgr2=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.876+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008930 legacy=0x7f18080135d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:22.876+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080084f0 0x7f1808011080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:23.678 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:23.677+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080084f0 msgr2=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:23.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:23.677+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008930 legacy=0x7f18080135d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:23.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:23.677+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080084f0 0x7f1808011080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.274+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080084f0 msgr2=0x7f1808011080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.274+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080084f0 0x7f1808011080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.274+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008930 legacy=0x7f18080135d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.274+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.274+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:19:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.274+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.274+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f1808007b80 2023-11-28T17:19:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.274+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808008930 2023-11-28T17:19:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.274+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f180800a910 2023-11-28T17:19:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.274+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.274+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.274+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808007b80 legacy=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.275+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.275+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:25.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.476+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808007b80 legacy=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:25.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.476+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:25.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.476+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:25.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.476+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:25.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.476+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:25.820 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:19:25.831 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:19:25.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.877+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:25.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.877+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:25.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.877+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808007b80 legacy=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:25.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.877+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:25.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:25.877+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:26.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:26.678+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:26.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:26.678+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:26.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:26.678+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808007b80 legacy=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:26.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:26.678+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:26.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:26.678+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:28.276 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.274+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.274+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.274+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.274+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.274+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808007b80 legacy=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.274+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f180800f070 0x7f1808011460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.275+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f18080119a0 2023-11-28T17:19:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.275+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f180800f070 2023-11-28T17:19:28.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.275+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f180800f070 msgr2=0x7f1808011460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:28.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.275+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f180800f070 0x7f1808011460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:28.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.275+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080119a0 legacy=0x7f1808011db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:28.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.476+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080119a0 legacy=0x7f1808011db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:28.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.476+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f180800f070 msgr2=0x7f1808011460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:28.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.476+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f180800f070 0x7f1808011460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:28.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.877+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080119a0 legacy=0x7f1808011db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.877+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f180800f070 msgr2=0x7f1808011460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:28.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:28.877+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f180800f070 0x7f1808011460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:29.678+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080119a0 legacy=0x7f1808011db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:29.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:29.678+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f180800f070 msgr2=0x7f1808011460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:29.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:29.678+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f180800f070 0x7f1808011460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:31.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.275+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f180800f070 msgr2=0x7f1808011460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:31.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.275+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f180800f070 0x7f1808011460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:31.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.275+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080119a0 legacy=0x7f1808011db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:31.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.275+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:19:31.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.275+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:31.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.275+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:31.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.275+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f18080119a0 2023-11-28T17:19:31.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.275+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808008930 2023-11-28T17:19:31.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.275+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f180800a910 2023-11-28T17:19:31.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.275+0000 7f182a4e9640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080119a0 legacy=0x7f1808011db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:31.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.275+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:31.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.275+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:31.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.275+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:31.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.276+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:31.477 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.476+0000 7f182a4e9640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080119a0 legacy=0x7f1808011db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:31.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.476+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:31.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.476+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:31.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.477+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:31.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.477+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.877+0000 7f182a4e9640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080119a0 legacy=0x7f1808011db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.877+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.877+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.878+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:31.878+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:32.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:32.678+0000 7f182a4e9640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080119a0 legacy=0x7f1808011db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:32.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:32.678+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:32.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:32.678+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:32.680 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:32.679+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:32.679+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.275+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.275+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:34.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.275+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:34.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.275+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:34.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.275+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080119a0 legacy=0x7f1808011db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:34.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.275+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808007b80 0x7f1808008d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:34.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.275+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f18080134d0 2023-11-28T17:19:34.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.275+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808007b80 2023-11-28T17:19:34.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.276+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808007b80 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:34.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.276+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808007b80 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:34.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.276+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f18080134d0 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:34.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.477+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808007b80 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:34.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.477+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808007b80 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:34.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.477+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f18080134d0 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:34.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.878+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808007b80 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.878+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808007b80 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:34.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:34.878+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f18080134d0 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:35.679+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808007b80 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:35.679+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808007b80 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:35.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:35.679+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f18080134d0 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:37.277 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808007b80 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:37.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808007b80 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:37.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f18080134d0 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:37.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:19:37.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 0x7f1808008d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:37.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:37.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:37.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f18080138e0 2023-11-28T17:19:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f1808008930 2023-11-28T17:19:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1808006630 con 0x7f180800a910 2023-11-28T17:19:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.276+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:37.478 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.477+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:37.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.477+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:37.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.477+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:37.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.477+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:37.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.477+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.477+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:37.879 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.878+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:37.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.878+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:37.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.878+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:37.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.878+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:37.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.878+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:37.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:37.878+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:38.679+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:38.679+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:38.679+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:38.679+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:38.679+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:38.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:38.680+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:40.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:40.276+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:40.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:40.276+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:40.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:40.276+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:40.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:40.276+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008930 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:40.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:40.276+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:40.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:40.276+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:40.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:40.276+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f1808007260 2023-11-28T17:19:40.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:40.276+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f1808006770 2023-11-28T17:19:40.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:40.277+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:40.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:40.277+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:40.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:40.478+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:40.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:40.478+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:40.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:40.879+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:40.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:40.879+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:41.681 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:41.680+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:41.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:41.680+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:43.278 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.277+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:43.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.277+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:43.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.277+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:19:43.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.277+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 0x7f1808008d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:43.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.277+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:43.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.277+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18080134d0 con 0x7f1808006770 2023-11-28T17:19:43.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.277+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f18080138e0 2023-11-28T17:19:43.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.277+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f180800a910 2023-11-28T17:19:43.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.277+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:43.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.277+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:43.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.277+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:43.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.277+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:43.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.277+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.478+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.478+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.478+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.478+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.478+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:43.880 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.879+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.879+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.879+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.879+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:43.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:43.879+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:44.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:44.680+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:44.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:44.680+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:44.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:44.680+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:44.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:44.680+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:44.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:44.680+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.277+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.277+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.277+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.277+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080138e0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.277+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.277+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.277+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f1808007260 2023-11-28T17:19:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.277+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808008900 2023-11-28T17:19:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.278+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.278+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:46.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.278+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:46.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.479+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:46.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.479+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.479+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.880+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.880+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:46.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:46.880+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:47.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:47.681+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:47.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:47.681+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:47.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:47.681+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.278+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.278+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.278+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.278+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:19:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.278+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.278+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.278+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18080134d0 con 0x7f1808006770 2023-11-28T17:19:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.278+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808008900 2023-11-28T17:19:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.278+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f180800a910 2023-11-28T17:19:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.278+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.278+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.278+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.278+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.278+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:49.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.479+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:49.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.479+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:49.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.479+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:49.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.479+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:49.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.479+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:49.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.880+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:49.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.880+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:49.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.880+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:49.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.880+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:49.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:49.880+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:50.681+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:50.681+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:50.681+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:50.681+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:50.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:50.681+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:52.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.278+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:52.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.278+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:52.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.278+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:52.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.278+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:52.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.278+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:52.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.278+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:52.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.278+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f1808007260 2023-11-28T17:19:52.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.278+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f18080148f0 2023-11-28T17:19:52.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.278+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:52.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.278+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:52.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.278+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:52.480 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.479+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:52.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.479+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:52.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.479+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:52.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.880+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:52.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.880+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:52.881 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:52.880+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:53.682 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:53.681+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:53.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:53.681+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:53.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:53.681+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:55.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:55.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:19:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:19:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f18080148f0 2023-11-28T17:19:55.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f1808008900 2023-11-28T17:19:55.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1808006630 con 0x7f180800a910 2023-11-28T17:19:55.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:55.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:55.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:55.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:55.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:55.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.279+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:19:55.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.480+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:55.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.480+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:55.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.480+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:55.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.480+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:55.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.480+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:55.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.480+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:19:55.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.881+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:55.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.881+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:55.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.881+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:55.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.881+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:55.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.881+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:55.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:55.881+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:19:55.962 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:19:55.972 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:19:56.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:56.682+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:19:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:56.682+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:56.682+0000 7f182a4e9640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:19:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:56.682+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:19:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:56.682+0000 7f182a4e9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:56.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:56.682+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:19:58.280 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:58.279+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:58.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:58.279+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:58.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:58.279+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:58.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:58.279+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:58.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:58.279+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:19:58.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:58.279+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:19:58.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:58.279+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18080134d0 con 0x7f1808007260 2023-11-28T17:19:58.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:58.279+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f1808006770 2023-11-28T17:19:58.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:58.279+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:58.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:58.279+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:58.480+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:58.481 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:58.480+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:58.882 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:58.881+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:58.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:58.881+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:19:59.683 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:59.682+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:19:59.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:19:59.682+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:01.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.280+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:01.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.280+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.280+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:20:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.280+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.280+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.280+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f1808007260 2023-11-28T17:20:01.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.280+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f18080148f0 2023-11-28T17:20:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.280+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f180800a910 2023-11-28T17:20:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.280+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.280+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.280+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:01.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.280+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.280+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:01.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.481+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:01.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.481+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:01.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.481+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:01.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.481+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:01.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.481+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:01.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.882+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:01.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.882+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:01.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.882+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:01.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.882+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:01.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:01.882+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:02.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:02.683+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:02.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:02.683+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:02.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:02.683+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:02.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:02.683+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:02.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:02.683+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:04.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.280+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:04.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.280+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f180800a910 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:04.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.280+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:04.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.280+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:04.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.280+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:04.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.280+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:04.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.280+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18080134d0 con 0x7f1808006770 2023-11-28T17:20:04.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.280+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808008900 2023-11-28T17:20:04.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.280+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:04.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.280+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:04.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.280+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.481+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.481+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.481+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:04.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.882+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:04.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.882+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:04.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:04.882+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:05.683+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:05.683+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:05.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:05.683+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:07.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.280+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:07.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.280+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008900 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:07.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.280+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:07.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.280+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:20:07.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.280+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:07.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.280+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f1808007260 2023-11-28T17:20:07.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.280+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18080134d0 con 0x7f1808006770 2023-11-28T17:20:07.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.280+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f18080148f0 2023-11-28T17:20:07.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.280+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:07.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.280+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:07.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.281+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:07.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.281+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:07.482 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.481+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:07.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.482+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:07.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.482+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:07.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.482+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:07.883 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.882+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:07.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.883+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:07.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.883+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:07.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:07.883+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:08.684 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:08.683+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:08.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:08.684+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:08.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:08.684+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:08.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:08.684+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.281+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 msgr2=0x7f1808008d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.281+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080148f0 0x7f1808008d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.281+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.281+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808007260 legacy=0x7f1808011080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.281+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 0x7f1808007fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.281+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.281+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f180800a910 2023-11-28T17:20:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.281+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f18080088d0 2023-11-28T17:20:10.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.281+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:10.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.281+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:10.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.281+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:10.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.281+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:10.483 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.482+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:10.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.482+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:10.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.482+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:10.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.482+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:10.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.883+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:10.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.883+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:10.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.883+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:10.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:10.883+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:11.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:11.684+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:11.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:11.684+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:11.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:11.684+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:11.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:11.684+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:13.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.281+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:13.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.281+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:13.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.281+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.281+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.281+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:20:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.281+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.281+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f180800a910 0x7f1808012eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.281+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f1808006770 2023-11-28T17:20:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.281+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f180800a910 2023-11-28T17:20:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.281+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f18080088d0 2023-11-28T17:20:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.281+0000 7f182a4e9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f180800a910 msgr2=0x7f1808012eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.281+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.281+0000 7f182a4e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f180800a910 0x7f1808012eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:13.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.281+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:13.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.282+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:13.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.482+0000 7f182a4e9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f180800a910 msgr2=0x7f1808012eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:13.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.482+0000 7f182a4e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f180800a910 0x7f1808012eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:13.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.482+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:13.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.483+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:13.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.483+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:13.884 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.883+0000 7f182a4e9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f180800a910 msgr2=0x7f1808012eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:13.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.883+0000 7f182a4e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f180800a910 0x7f1808012eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:13.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.883+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:13.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.884+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:13.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:13.884+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:14.685 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:14.684+0000 7f182a4e9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f180800a910 msgr2=0x7f1808012eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:14.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:14.684+0000 7f182a4e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f180800a910 0x7f1808012eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:14.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:14.684+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:14.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:14.685+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:14.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:14.685+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.282+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.282+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.282+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f180800a910 msgr2=0x7f1808012eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.282+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f180800a910 0x7f1808012eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.282+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080127a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.282+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.282+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f18080147c0 2023-11-28T17:20:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.282+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808008d40 2023-11-28T17:20:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.282+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080147c0 legacy=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.282+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.282+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:16.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.483+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080147c0 legacy=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:16.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.483+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:16.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.483+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:16.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.884+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080147c0 legacy=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:16.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.884+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:16.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:16.884+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:17.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:17.685+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080147c0 legacy=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:17.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:17.685+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:17.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:17.685+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:19.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.282+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:19.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.282+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:19.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.282+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080147c0 legacy=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:19.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.282+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:20:19.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.282+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:19.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.282+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808008d40 0x7f18080147c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:19.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.282+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f1808006770 2023-11-28T17:20:19.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.282+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808008d40 2023-11-28T17:20:19.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.282+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f18080088d0 2023-11-28T17:20:19.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.282+0000 7f182a4e9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808008d40 msgr2=0x7f18080147c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:19.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.282+0000 7f182a4e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808008d40 0x7f18080147c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:19.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.282+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:19.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.282+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:19.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.282+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:19.484 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.483+0000 7f182a4e9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808008d40 msgr2=0x7f18080147c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.483+0000 7f182a4e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808008d40 0x7f18080147c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.483+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.484+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:19.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.484+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:19.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.884+0000 7f182a4e9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808008d40 msgr2=0x7f18080147c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:19.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.884+0000 7f182a4e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808008d40 0x7f18080147c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.884+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.885+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:19.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:19.885+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:20.686+0000 7f182a4e9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808008d40 msgr2=0x7f18080147c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:20.686+0000 7f182a4e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808008d40 0x7f18080147c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:20.686+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:20.686+0000 7f18294e7640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:20.686+0000 7f18294e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.283+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.283+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.283+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808008d40 msgr2=0x7f18080147c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:22.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.283+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808008d40 0x7f18080147c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.283+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006770 legacy=0x7f18080125d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.283+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808006b80 0x7f1808011ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.283+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f180800a910 2023-11-28T17:20:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.283+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808006b80 2023-11-28T17:20:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.283+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f180800a910 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:22.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.283+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808006b80 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:22.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.283+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808006b80 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:22.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.484+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f180800a910 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:22.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.484+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808006b80 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:22.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.484+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808006b80 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:22.885 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.885+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f180800a910 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:22.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.885+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808006b80 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:22.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:22.885+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808006b80 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:23.686 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:23.685+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f180800a910 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:23.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:23.685+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808006b80 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:23.687 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:23.685+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808006b80 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:25.284 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.282+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808006b80 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.282+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808006b80 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.282+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f180800a910 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.283+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:20:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.283+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.283+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808006f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.283+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f1808006b80 2023-11-28T17:20:25.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.283+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808006770 2023-11-28T17:20:25.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.283+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f18080088d0 2023-11-28T17:20:25.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.283+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006b80 legacy=0x7f18080125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:25.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.283+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:25.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.283+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808006f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:25.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.283+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:25.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.283+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808006f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:25.485 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.484+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808006f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:25.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.484+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006b80 legacy=0x7f18080125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:25.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.484+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:25.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.484+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808006f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:25.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.484+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:25.886 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.885+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808006f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:25.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.885+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:25.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.885+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808006f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:25.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.885+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006b80 legacy=0x7f18080125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:25.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:25.885+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:26.101 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:20:26.110 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:20:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:26.686+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808006f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:26.686+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808006f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:26.686+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:26.686+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006b80 legacy=0x7f18080125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:26.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:26.686+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.283+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808007fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.283+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808007fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.283+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808006f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:28.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.283+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808006f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:28.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.283+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f1808006b80 legacy=0x7f18080125d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:28.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.283+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 0x7f1808011ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:28.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.283+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f1808008d40 2023-11-28T17:20:28.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.283+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f180800a910 2023-11-28T17:20:28.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.283+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008d40 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:28.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.284+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:28.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.284+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.485+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008d40 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.485+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.485+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.886+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008d40 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:28.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.886+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:28.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:28.886+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:29.688 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:29.687+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008d40 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:29.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:29.687+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:29.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:29.687+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:31.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.283+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:31.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.283+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f180800a910 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:31.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.283+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008d40 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:31.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.284+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:20:31.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.284+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:31.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.284+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808011ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:31.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.284+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f1808008d40 2023-11-28T17:20:31.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.284+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808006770 2023-11-28T17:20:31.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.284+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f18080088d0 2023-11-28T17:20:31.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.284+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008d40 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:31.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.284+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:31.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.284+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:31.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.284+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:31.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.284+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:31.486 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.485+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008d40 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.485+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.485+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.485+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.485+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.886+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008d40 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:31.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.886+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:31.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.886+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:31.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.886+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:31.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:31.886+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:32.687+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:32.687+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:32.687+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008d40 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:32.687+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:32.689 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:32.687+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.284+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.284+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.284+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:34.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.284+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:34.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.284+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808008d40 legacy=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:34.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.284+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014740 0x7f1808006b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:34.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.284+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f180800a910 2023-11-28T17:20:34.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.284+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808014740 2023-11-28T17:20:34.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.285+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808012530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:34.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.285+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014740 msgr2=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:34.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.285+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014740 0x7f1808006b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:34.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.486+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808012530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:34.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.486+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014740 msgr2=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:34.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.486+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014740 0x7f1808006b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:34.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.887+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808012530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.887+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014740 msgr2=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:34.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:34.887+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014740 0x7f1808006b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:35.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:35.688+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808012530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:35.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:35.688+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014740 msgr2=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:35.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:35.688+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014740 0x7f1808006b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:37.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.284+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014740 msgr2=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:37.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.284+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014740 0x7f1808006b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:37.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.284+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808012530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:37.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.284+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:20:37.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.284+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:37.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.285+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808011ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:37.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.285+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f180800a910 2023-11-28T17:20:37.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.285+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808006770 2023-11-28T17:20:37.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.285+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f18080088d0 2023-11-28T17:20:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.285+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808012530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.285+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.285+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.285+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:37.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.285+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:37.487 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.486+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:37.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.486+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808012530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:37.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.486+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:37.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.486+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:37.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.486+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:37.888 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.887+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:37.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.887+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:37.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.887+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808012530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:37.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.887+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:37.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:37.887+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:38.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:38.688+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:38.688+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:38.688+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:38.688+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808012530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:38.688+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:40.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.285+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808011ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:40.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.285+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808011ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:40.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.285+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:40.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.285+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:40.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.285+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808012530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:40.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.285+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:40.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.285+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f1808014740 2023-11-28T17:20:40.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.285+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808008d40 2023-11-28T17:20:40.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.285+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:40.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.285+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808014740 legacy=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:40.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.286+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:40.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.487+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808014740 legacy=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:40.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.487+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:40.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.487+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:40.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.888+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808014740 legacy=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:40.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.888+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:40.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:40.888+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:41.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:41.689+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808014740 legacy=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:41.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:41.689+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:41.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:41.689+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:43.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.285+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:43.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.285+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:43.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.285+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808014740 legacy=0x7f1808006b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:43.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.285+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:43.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.285+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:20:43.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.285+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:43.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.286+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808006b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:43.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.286+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808006770 2023-11-28T17:20:43.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.286+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f1808008d40 2023-11-28T17:20:43.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.286+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1808006630 con 0x7f18080088d0 2023-11-28T17:20:43.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.286+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:43.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.286+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:43.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.286+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:43.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.286+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:43.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.286+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:43.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.286+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:43.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.487+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:43.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.487+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:43.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.487+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:43.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.487+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:43.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.487+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:43.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.487+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:43.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.888+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:43.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.888+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:43.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.888+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.888+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.888+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:43.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:43.888+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:44.689+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:44.689+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:44.689+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:44.689+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:44.689+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:44.689+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:46.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:46.286+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 msgr2=0x7f1808006b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:46.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:46.286+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18080088d0 0x7f1808006b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:46.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:46.286+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 msgr2=0x7f1808012ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:46.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:46.286+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808008d40 0x7f1808012ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:46.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:46.286+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:46.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:46.286+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1808006770 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:46.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:46.286+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f18080077a0 2023-11-28T17:20:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:46.286+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f180800a910 2023-11-28T17:20:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:46.286+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080077a0 legacy=0x7f1808010fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:46.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:46.286+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:46.488 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:46.487+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080077a0 legacy=0x7f1808010fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:46.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:46.487+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:46.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:46.888+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080077a0 legacy=0x7f1808010fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:46.889 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:46.888+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:47.690 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:47.689+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080077a0 legacy=0x7f1808010fe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:47.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:47.689+0000 7f1829ce8640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:49.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.287+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:49.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.287+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f18080077a0 legacy=0x7f1808010fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:49.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.287+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:20:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.287+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.287+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18080134d0 con 0x7f1808014670 2023-11-28T17:20:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.287+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824070bd0 con 0x7f180800a910 2023-11-28T17:20:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.287+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f18080088d0 2023-11-28T17:20:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.287+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808014670 legacy=0x7f1808018d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.287+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.287+0000 7f182a4e9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.287+0000 7f182a4e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:49.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.488+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808014670 legacy=0x7f1808018d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:49.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.488+0000 7f182a4e9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:49.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.488+0000 7f182a4e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:49.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.488+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:49.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.889+0000 7f182a4e9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:49.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.889+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808014670 legacy=0x7f1808018d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:49.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.889+0000 7f182a4e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:49.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:49.889+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:50.691 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:50.690+0000 7f182a4e9640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:50.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:50.690+0000 7f182a4e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:50.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:50.690+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808014670 legacy=0x7f1808018d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:50.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:50.690+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:52.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.287+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:52.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.287+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:52.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.287+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:52.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.287+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808014670 legacy=0x7f1808018d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:52.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.287+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:52.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.287+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808006cb0 0x7f180800efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:52.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.287+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808006cb0 2023-11-28T17:20:52.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.287+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f1808008d40 2023-11-28T17:20:52.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.287+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:52.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.287+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808006cb0 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:52.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.287+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808006cb0 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:52.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.287+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:52.489 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.488+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808006cb0 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:52.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.488+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808006cb0 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:52.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.489+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:52.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.489+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:52.890 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.889+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808006cb0 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:52.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.889+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808006cb0 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:52.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.890+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:52.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:52.890+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:53.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:53.691+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808006cb0 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:53.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:53.691+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808006cb0 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:53.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:53.691+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:53.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:53.691+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:55.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:55.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:55.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808006cb0 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808006cb0 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:20:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f180800a910 2023-11-28T17:20:55.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f18080088d0 2023-11-28T17:20:55.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f1808008d40 2023-11-28T17:20:55.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:55.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:55.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:55.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:55.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.288+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:55.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.489+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:55.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.489+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:55.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.489+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:55.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.489+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:55.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.489+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:55.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.890+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:55.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.890+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:55.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.890+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:55.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.890+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:55.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:55.890+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:56.265 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:20:56.274 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:20:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:56.691+0000 7f18294e7640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:20:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:56.691+0000 7f18294e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:56.691+0000 7f1829ce8640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:20:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:56.691+0000 7f182a4e9640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:20:56.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:56.691+0000 7f1829ce8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:20:58.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.288+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:58.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.288+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:58.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.288+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:58.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.288+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:20:58.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.288+0000 7f181b7fe640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f180800a910 legacy=0x7f1808007090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:20:58.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.288+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 0x7f180800efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:20:58.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.288+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18080134d0 con 0x7f1808006cb0 2023-11-28T17:20:58.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.288+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f1808014670 2023-11-28T17:20:58.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.288+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808006cb0 legacy=0x7f1808011520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:58.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.288+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:58.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.288+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:20:58.490 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.489+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808006cb0 legacy=0x7f1808011520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:58.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.489+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:58.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.489+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:20:58.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.890+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808006cb0 legacy=0x7f1808011520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:58.891 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.890+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:58.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:58.890+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:20:59.692 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:59.691+0000 7f18294e7640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808006cb0 legacy=0x7f1808011520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:20:59.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:59.691+0000 7f1829ce8640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:20:59.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:20:59.691+0000 7f1829ce8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:21:01.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:21:01.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:21:01.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f181b7fe640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f1808006cb0 legacy=0x7f1808011520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:21:01.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:21:01.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f182a4e9640 1 -- reap_dead start 2023-11-28T17:21:01.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 0x7f180800efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:21:01.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T17:21:01.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f181b7fe640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824144810 con 0x7f18080088d0 2023-11-28T17:21:01.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f181b7fe640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824070eb0 con 0x7f1808014670 2023-11-28T17:21:01.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f181b7fe640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1808006630 con 0x7f1808008d40 2023-11-28T17:21:01.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:21:01.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:21:01.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:21:01.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:21:01.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:21:01.293 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.289+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T17:21:01.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.490+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:21:01.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.490+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:21:01.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.490+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:21:01.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.490+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:21:01.492 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.490+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:21:01.493 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.490+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T17:21:01.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.891+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:21:01.892 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.891+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:21:01.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.891+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:21:01.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.891+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:21:01.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.891+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:21:01.893 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:01.891+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T17:21:02.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:02.692+0000 7f1829ce8640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.116:3300/0 2023-11-28T17:21:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:02.692+0000 7f1829ce8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:21:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:02.692+0000 7f18294e7640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3300/0 2023-11-28T17:21:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:02.692+0000 7f182a4e9640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.156:3301/0 2023-11-28T17:21:02.694 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:02.692+0000 7f18294e7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:21:02.695 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:02.692+0000 7f182a4e9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T17:21:04.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:04.289+0000 7f181b7fe640 1 -- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 msgr2=0x7f1808006770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:21:04.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:04.289+0000 7f181b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1808008d40 0x7f1808006770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:21:04.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:04.289+0000 7f181b7fe640 1 -- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 msgr2=0x7f180800efd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:21:04.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:04.289+0000 7f181b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1808014670 0x7f180800efd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:21:04.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:04.289+0000 7f181b7fe640 1 -- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 msgr2=0x7f1808007f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-28T17:21:04.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:04.289+0000 7f181b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18080088d0 0x7f1808007f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T17:21:04.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:04.289+0000 7f181b7fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824071190 con 0x7f180800a910 2023-11-28T17:21:04.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:04.289+0000 7f181b7fe640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18080134d0 con 0x7f18080073c0 2023-11-28T17:21:04.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:04.289+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f180800a910 legacy=0x7f1808006cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:21:04.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:04.289+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f18080073c0 legacy=0x7f1808011520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:21:04.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:04.490+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f180800a910 legacy=0x7f1808006cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:21:04.491 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:04.490+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f18080073c0 legacy=0x7f1808011520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:21:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:04.891+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f180800a910 legacy=0x7f1808006cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:21:04.894 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:04.891+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f18080073c0 legacy=0x7f1808011520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:21:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:05.692+0000 7f1829ce8640 1 -- >> v1:172.21.15.116:6789/0 conn(0x7f180800a910 legacy=0x7f1808006cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.116:6789/0 2023-11-28T17:21:05.693 INFO:teuthology.orchestra.run.smithi116.stderr:2023-11-28T17:21:05.692+0000 7f18294e7640 1 -- >> v1:172.21.15.156:6789/0 conn(0x7f18080073c0 legacy=0x7f1808011520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.156:6789/0 2023-11-28T17:21:07.174 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-28T17:21:07.175 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-28T17:21:07.186 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-28T17:21:07.186 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-28T17:21:07.186 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:21:07.189 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2023-11-28T17:21:07.199 INFO:tasks.ceph.mds.a:Stopped 2023-11-28T17:21:07.199 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-28T17:21:07.199 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.199 INFO:tasks.ceph.mds.c:Stopped 2023-11-28T17:21:07.200 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-28T17:21:07.200 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:21:07.200 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi156 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b' 2023-11-28T17:21:07.210 INFO:tasks.ceph.mds.b:Stopped 2023-11-28T17:21:07.210 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-28T17:21:07.210 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:21:07.210 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi156 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' 2023-11-28T17:21:07.220 INFO:tasks.ceph.mds.d:Stopped 2023-11-28T17:21:07.220 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-28T17:21:07.220 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-28T17:21:07.221 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.221 INFO:tasks.ceph.osd.0:Stopped 2023-11-28T17:21:07.221 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-28T17:21:07.221 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.221 INFO:tasks.ceph.osd.1:Stopped 2023-11-28T17:21:07.221 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-28T17:21:07.222 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.222 INFO:tasks.ceph.osd.2:Stopped 2023-11-28T17:21:07.222 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-28T17:21:07.222 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.222 INFO:tasks.ceph.osd.3:Stopped 2023-11-28T17:21:07.222 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-11-28T17:21:07.223 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.223 INFO:tasks.ceph.osd.4:Stopped 2023-11-28T17:21:07.223 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-11-28T17:21:07.223 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.223 INFO:tasks.ceph.osd.5:Stopped 2023-11-28T17:21:07.224 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-11-28T17:21:07.224 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.224 INFO:tasks.ceph.osd.6:Stopped 2023-11-28T17:21:07.224 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-11-28T17:21:07.224 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.224 INFO:tasks.ceph.osd.7:Stopped 2023-11-28T17:21:07.225 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-28T17:21:07.225 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-11-28T17:21:07.225 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.225 INFO:tasks.ceph.mgr.y:Stopped 2023-11-28T17:21:07.225 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-28T17:21:07.225 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:21:07.226 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi156 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' 2023-11-28T17:21:07.236 INFO:tasks.ceph.mgr.x:Stopped 2023-11-28T17:21:07.236 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-28T17:21:07.237 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-28T17:21:07.237 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.237 INFO:tasks.ceph.mon.a:Stopped 2023-11-28T17:21:07.237 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-28T17:21:07.237 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.237 INFO:tasks.ceph.mon.b:Stopped 2023-11-28T17:21:07.238 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-28T17:21:07.238 INFO:teuthology.orchestra.run:waiting for 300 2023-11-28T17:21:07.240 INFO:tasks.ceph.mon.c:Stopped 2023-11-28T17:21:07.241 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-28T17:21:07.241 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'client session with non-allowable root' | head -n 1 2023-11-28T17:21:07.364 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:21:07.364 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-28T17:21:07.567 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:21:07.568 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-28T17:21:07.761 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:21:07.762 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-28T17:21:07.912 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:21:07.913 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-11-28T17:21:08.108 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:21:08.108 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-11-28T17:21:08.317 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:21:08.317 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-11-28T17:21:08.474 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:21:08.474 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-11-28T17:21:08.657 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:21:08.657 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-11-28T17:21:08.829 INFO:tasks.ceph:Archiving mon data... 2023-11-28T17:21:08.830 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360/data/mon.a.tgz 2023-11-28T17:21:08.830 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2023-11-28T17:21:08.889 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.rkvmslPqGY 2023-11-28T17:21:08.890 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.rkvmslPqGY 2023-11-28T17:21:09.073 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.rkvmslPqGY 2023-11-28T17:21:09.224 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.rkvmslPqGY is 277KB 2023-11-28T17:21:09.332 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.rkvmslPqGY 2023-11-28T17:21:09.392 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360/data/mon.b.tgz 2023-11-28T17:21:09.392 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2023-11-28T17:21:09.452 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.Z9Z1nV0WQJ 2023-11-28T17:21:09.452 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Z9Z1nV0WQJ 2023-11-28T17:21:09.623 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.Z9Z1nV0WQJ 2023-11-28T17:21:09.771 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.Z9Z1nV0WQJ is 276KB 2023-11-28T17:21:09.877 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.Z9Z1nV0WQJ 2023-11-28T17:21:09.937 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360/data/mon.c.tgz 2023-11-28T17:21:09.938 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2023-11-28T17:21:09.990 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.h9zUuugL89 2023-11-28T17:21:09.990 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.h9zUuugL89 2023-11-28T17:21:10.149 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.h9zUuugL89 2023-11-28T17:21:10.302 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.h9zUuugL89 is 276KB 2023-11-28T17:21:10.410 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.h9zUuugL89 2023-11-28T17:21:10.469 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-28T17:21:10.469 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-28T17:21:10.478 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-28T17:21:10.582 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-28T17:21:10.582 DEBUG:teuthology.orchestra.run.smithi116:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-28T17:21:10.587 DEBUG:teuthology.orchestra.run.smithi156:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-28T17:21:10.685 INFO:teuthology.orchestra.run.smithi116.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-28T17:21:10.692 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-28T17:21:10.705 INFO:tasks.ceph:Archiving crash dumps... 2023-11-28T17:21:10.707 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360/remote/smithi116/crash 2023-11-28T17:21:10.708 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-28T17:21:10.803 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360/remote/smithi156/crash 2023-11-28T17:21:10.804 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-28T17:21:10.921 INFO:tasks.ceph:Shutting down logrotate 2023-11-28T17:21:10.921 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:21:10.930 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-28T17:21:11.094 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-28T17:21:11.196 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-28T17:21:11.299 INFO:tasks.ceph:Compressing logs... 2023-11-28T17:21:11.299 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-28T17:21:11.308 DEBUG:teuthology.orchestra.run.smithi156:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-28T17:21:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102980.log 2023-11-28T17:21:11.397 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94370.log 2023-11-28T17:21:11.398 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102980.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.102980.log.gz 2023-11-28T17:21:11.399 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.94244.log 2023-11-28T17:21:11.399 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107996.log 2023-11-28T17:21:11.400 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94370.log.gz 2023-11-28T17:21:11.401 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93945.log 2023-11-28T17:21:11.401 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.94244.log.gz 2023-11-28T17:21:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.107996.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-28T17:21:11.402 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.107996.log.gz 2023-11-28T17:21:11.403 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-11-28T17:21:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93945.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.93945.log.gz 2023-11-28T17:21:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96176.log 2023-11-28T17:21:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94538.log 2023-11-28T17:21:11.404 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.93638.log 2023-11-28T17:21:11.405 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.94394.log 2023-11-28T17:21:11.406 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109250.log 2023-11-28T17:21:11.406 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.96176.log.gz 2023-11-28T17:21:11.406 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100458.log 2023-11-28T17:21:11.406 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.94538.log.gz 2023-11-28T17:21:11.408 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.109250.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.109250.log.gz 2023-11-28T17:21:11.408 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101630.log 2023-11-28T17:21:11.408 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.3.93638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88492.log 2023-11-28T17:21:11.409 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.1.94394.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-28T17:21:11.409 INFO:teuthology.orchestra.run.smithi156.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.3.93638.log.gz 2023-11-28T17:21:11.409 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.88492.log: 89.3% -- replaced with /var/log/ceph/ceph-client.1.94394.log.gz 2023-11-28T17:21:11.409 INFO:teuthology.orchestra.run.smithi156.stderr:gzip 87.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.88492.log.gz -- 2023-11-28T17:21:11.409 INFO:teuthology.orchestra.run.smithi156.stderr: /var/log/ceph/ceph-mds.b.log 2023-11-28T17:21:11.410 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.93186.log 2023-11-28T17:21:11.411 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103452.log 2023-11-28T17:21:11.411 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100458.log.gz 2023-11-28T17:21:11.411 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96134.log 2023-11-28T17:21:11.411 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.101630.log.gz 2023-11-28T17:21:11.412 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103452.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.103452.log.gz 2023-11-28T17:21:11.412 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108722.log 2023-11-28T17:21:11.413 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88612.log 2023-11-28T17:21:11.414 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.1.93186.log: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-28T17:21:11.414 INFO:teuthology.orchestra.run.smithi156.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.1.93186.log.gz 2023-11-28T17:21:11.414 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-11-28T17:21:11.414 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109376.log 2023-11-28T17:21:11.414 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96134.log.gz 2023-11-28T17:21:11.414 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101046.log 2023-11-28T17:21:11.415 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108722.log.gz 2023-11-28T17:21:11.416 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.88612.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-11-28T17:21:11.416 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.4.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.88612.log.gz 2023-11-28T17:21:11.417 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-11-28T17:21:11.417 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.109376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107201.log 2023-11-28T17:21:11.417 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101046.log: 95.1% -- replaced with /var/log/ceph/ceph-client.admin.109376.log.gz 2023-11-28T17:21:11.418 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96302.log 2023-11-28T17:21:11.418 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.101046.log.gz 2023-11-28T17:21:11.418 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.107201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99347.log 2023-11-28T17:21:11.418 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.107201.log.gz 2023-11-28T17:21:11.419 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88732.log 2023-11-28T17:21:11.419 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.95090.log 2023-11-28T17:21:11.420 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96302.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.96302.log.gz 2023-11-28T17:21:11.420 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101418.log 2023-11-28T17:21:11.420 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103704.log 2023-11-28T17:21:11.421 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.99347.log.gz 2023-11-28T17:21:11.421 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106593.log 2023-11-28T17:21:11.421 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.101418.log.gz 2023-11-28T17:21:11.422 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.88732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88532.log 2023-11-28T17:21:11.422 INFO:teuthology.orchestra.run.smithi156.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88732.log.gz 2023-11-28T17:21:11.423 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98402.log 2023-11-28T17:21:11.423 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103704.log.gz 2023-11-28T17:21:11.423 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101126.log 2023-11-28T17:21:11.423 INFO:teuthology.orchestra.run.smithi116.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.106593.log.gz 2023-11-28T17:21:11.424 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.1.95090.log: /var/log/ceph/ceph-client.admin.88532.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88532.log.gz 2023-11-28T17:21:11.424 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88652.log 2023-11-28T17:21:11.425 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100374.log 2023-11-28T17:21:11.425 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.98402.log.gz 2023-11-28T17:21:11.425 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101126.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101126.log.gz 2023-11-28T17:21:11.426 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106785.log 2023-11-28T17:21:11.427 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88572.log 2023-11-28T17:21:11.427 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102124.log 2023-11-28T17:21:11.428 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100374.log.gz 2023-11-28T17:21:11.428 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108840.log 2023-11-28T17:21:11.428 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.106785.log.gz 2023-11-28T17:21:11.429 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.88652.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.88652.log.gz 2023-11-28T17:21:11.430 INFO:teuthology.orchestra.run.smithi156.stderr: 96.5% -- replaced with /var/log/ceph/ceph-client.1.95090.log.gz 2023-11-28T17:21:11.430 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.88572.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.88572.log.gz 2023-11-28T17:21:11.430 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88692.log 2023-11-28T17:21:11.430 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102124.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.102124.log.gz 2023-11-28T17:21:11.430 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106141.log 2023-11-28T17:21:11.430 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108165.log 2023-11-28T17:21:11.432 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-28T17:21:11.432 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108840.log.gz 2023-11-28T17:21:11.433 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99389.log 2023-11-28T17:21:11.433 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.106141.log.gz 2023-11-28T17:21:11.433 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108165.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108165.log.gz 2023-11-28T17:21:11.433 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109090.log 2023-11-28T17:21:11.434 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.88692.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88692.log.gz 2023-11-28T17:21:11.434 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-28T17:21:11.435 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97524.log 2023-11-28T17:21:11.435 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.99389.log.gz 2023-11-28T17:21:11.435 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.109090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97030.log 2023-11-28T17:21:11.435 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.109090.log.gz 2023-11-28T17:21:11.436 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-11-28T17:21:11.436 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97524.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97524.log.gz 2023-11-28T17:21:11.437 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101461.log 2023-11-28T17:21:11.437 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101588.log 2023-11-28T17:21:11.437 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97030.log.gz 2023-11-28T17:21:11.437 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97856.log 2023-11-28T17:21:11.438 INFO:teuthology.orchestra.run.smithi116.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101461.log.gz 2023-11-28T17:21:11.438 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-28T17:21:11.438 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-11-28T17:21:11.439 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101588.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.101588.log.gz 2023-11-28T17:21:11.439 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96488.log 2023-11-28T17:21:11.440 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104983.log 2023-11-28T17:21:11.441 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88772.log 2023-11-28T17:21:11.441 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97856.log.gz 2023-11-28T17:21:11.442 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.107062.log 2023-11-28T17:21:11.442 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96488.log.gz 2023-11-28T17:21:11.443 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-11-28T17:21:11.443 INFO:teuthology.orchestra.run.smithi156.stderr: 94.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-28T17:21:11.443 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95924.log 88.6% 2023-11-28T17:21:11.444 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.104983.log.gz 2023-11-28T17:21:11.444 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.107062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94328.log 2023-11-28T17:21:11.445 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103022.log 2023-11-28T17:21:11.445 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95924.log.gz 2023-11-28T17:21:11.446 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.88772.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88772.log.gz 2023-11-28T17:21:11.447 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94328.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.94328.log.gz 2023-11-28T17:21:11.447 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94664.log 2023-11-28T17:21:11.447 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98192.log 2023-11-28T17:21:11.447 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.103022.log.gz 2023-11-28T17:21:11.448 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104362.log 2023-11-28T17:21:11.448 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.94664.log.gz 2023-11-28T17:21:11.449 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95222.log 2023-11-28T17:21:11.449 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.98192.log.gz 2023-11-28T17:21:11.450 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95224.log 2023-11-28T17:21:11.451 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.104362.log.gz 2023-11-28T17:21:11.452 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109334.log 2023-11-28T17:21:11.452 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95222.log.gz 2023-11-28T17:21:11.453 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95224.log: 94.6% -- replaced with /var/log/ceph/ceph-client.0.107062.log.gz 2023-11-28T17:21:11.453 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100332.log 2023-11-28T17:21:11.453 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95224.log.gz 2023-11-28T17:21:11.454 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.109334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103368.log 2023-11-28T17:21:11.454 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109334.log.gz 2023-11-28T17:21:11.455 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mon.c.log: 90.5% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-11-28T17:21:11.455 INFO:teuthology.orchestra.run.smithi156.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-11-28T17:21:11.456 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103206.log 2023-11-28T17:21:11.456 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100332.log.gz 2023-11-28T17:21:11.456 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103368.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.103368.log.gz -5 --verbose 2023-11-28T17:21:11.456 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.94454.log 2023-11-28T17:21:11.457 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95549.log 2023-11-28T17:21:11.457 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.103206.log.gz 2023-11-28T17:21:11.458 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94580.log 2023-11-28T17:21:11.461 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.94454.log.gz 2023-11-28T17:21:11.464 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95549.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.95549.log.gz 2023-11-28T17:21:11.464 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95556.log 2023-11-28T17:21:11.464 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102854.log 2023-11-28T17:21:11.464 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.94580.log.gz 2023-11-28T17:21:11.464 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95556.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.95556.log.gz 2023-11-28T17:21:11.465 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103248.log 2023-11-28T17:21:11.465 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96386.log 2023-11-28T17:21:11.465 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.102854.log.gz 2023-11-28T17:21:11.465 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103248.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.103248.log.gz 2023-11-28T17:21:11.465 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100500.log 2023-11-28T17:21:11.466 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106099.log 2023-11-28T17:21:11.466 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100500.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.96386.log.gz 2023-11-28T17:21:11.466 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100500.log.gz 2023-11-28T17:21:11.466 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98975.log 2023-11-28T17:21:11.467 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98933.log 2023-11-28T17:21:11.467 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106099.log.gz 2023-11-28T17:21:11.468 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108658.log 2023-11-28T17:21:11.468 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.98975.log.gz 2023-11-28T17:21:11.469 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98933.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.98933.log.gz 2023-11-28T17:21:11.469 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105973.log 2023-11-28T17:21:11.470 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108658.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108658.log.gz 2023-11-28T17:21:11.470 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108930.log 2023-11-28T17:21:11.471 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105973.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105973.log.gz 2023-11-28T17:21:11.471 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102040.log 2023-11-28T17:21:11.473 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108930.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.108930.log.gz 2023-11-28T17:21:11.473 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99059.log 2023-11-28T17:21:11.473 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97687.log 2023-11-28T17:21:11.474 INFO:teuthology.orchestra.run.smithi116.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.102040.log.gz 2023-11-28T17:21:11.474 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107444.log 2023-11-28T17:21:11.475 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.99059.log.gz 2023-11-28T17:21:11.475 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97687.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97687.log.gz 2023-11-28T17:21:11.476 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100416.log 2023-11-28T17:21:11.476 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.107444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97940.log 2023-11-28T17:21:11.477 INFO:teuthology.orchestra.run.smithi116.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.107444.log.gz 2023-11-28T17:21:11.478 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109186.log 2023-11-28T17:21:11.478 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.100416.log.gz 2023-11-28T17:21:11.479 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96050.log 2023-11-28T17:21:11.479 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.97940.log.gz 2023-11-28T17:21:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.109186.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109186.log.gz 2023-11-28T17:21:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95555.log 2023-11-28T17:21:11.481 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96050.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.96050.log.gz 2023-11-28T17:21:11.481 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.101504.log 2023-11-28T17:21:11.482 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95555.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.95555.log.gz 2023-11-28T17:21:11.482 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105380.log 2023-11-28T17:21:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101504.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101504.log.gz 2023-11-28T17:21:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95216.log 2023-11-28T17:21:11.484 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98749.log 2023-11-28T17:21:11.484 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.105380.log.gz 2023-11-28T17:21:11.485 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95216.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.95216.log.gz 2023-11-28T17:21:11.485 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102208.log 2023-11-28T17:21:11.486 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97730.log 2023-11-28T17:21:11.486 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.98749.log.gz 2023-11-28T17:21:11.487 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103080.log 2023-11-28T17:21:11.488 INFO:teuthology.orchestra.run.smithi116.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.102208.log.gz 2023-11-28T17:21:11.488 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100247.log 2023-11-28T17:21:11.489 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97730.log.gz 2023-11-28T17:21:11.489 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95798.log 2023-11-28T17:21:11.489 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.103080.log.gz 2023-11-28T17:21:11.491 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105067.log 2023-11-28T17:21:11.491 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100247.log.gz 2023-11-28T17:21:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95798.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95798.log.gz 2023-11-28T17:21:11.492 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88675.log 2023-11-28T17:21:11.493 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105067.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.82602.log 2023-11-28T17:21:11.493 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105067.log.gz 2023-11-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.88675.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-28T17:21:11.494 INFO:teuthology.orchestra.run.smithi116.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.88675.log.gz 2023-11-28T17:21:11.495 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.tmp-client.admin.82602.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.82602.log.gz 2023-11-28T17:21:11.495 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108390.log 2023-11-28T17:21:11.496 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94875.log 2023-11-28T17:21:11.497 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99305.log 2023-11-28T17:21:11.497 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108390.log.gz 2023-11-28T17:21:11.497 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-28T17:21:11.498 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104302.log 2023-11-28T17:21:11.498 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.94875.log.gz 2023-11-28T17:21:11.499 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99305.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.99305.log.gz 2023-11-28T17:21:11.499 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103410.log 2023-11-28T17:21:11.500 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104899.log 2023-11-28T17:21:11.501 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104302.log.gz 2023-11-28T17:21:11.501 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94412.log 2023-11-28T17:21:11.502 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.103410.log.gz 2023-11-28T17:21:11.503 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103494.log 2023-11-28T17:21:11.503 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.104899.log.gz 2023-11-28T17:21:11.504 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94412.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.94412.log.gz 2023-11-28T17:21:11.504 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104857.log 2023-11-28T17:21:11.505 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100542.log 2023-11-28T17:21:11.505 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.103494.log.gz 2023-11-28T17:21:11.506 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98108.log 2023-11-28T17:21:11.506 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.104857.log.gz 2023-11-28T17:21:11.507 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107243.log 2023-11-28T17:21:11.507 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100542.log.gz 2023-11-28T17:21:11.508 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98108.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.98108.log.gz 2023-11-28T17:21:11.508 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99733.log 2023-11-28T17:21:11.509 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.107243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93658.log 2023-11-28T17:21:11.509 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.107243.log.gz 2023-11-28T17:21:11.510 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95500.log 2023-11-28T17:21:11.510 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.99733.log.gz 2023-11-28T17:21:11.511 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93658.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.93658.log.gz 2023-11-28T17:21:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107336.log 2023-11-28T17:21:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105632.log 2023-11-28T17:21:11.513 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95500.log.gz 2023-11-28T17:21:11.514 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.107336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97982.log 2023-11-28T17:21:11.514 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.107336.log.gz 2023-11-28T17:21:11.515 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93987.log 2023-11-28T17:21:11.515 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105632.log.gz 2023-11-28T17:21:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97982.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.97982.log.gz 2023-11-28T17:21:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104260.log 2023-11-28T17:21:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93987.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93987.log.gz -5 2023-11-28T17:21:11.518 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.108270.log 2023-11-28T17:21:11.519 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104260.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.104260.log.gz 2023-11-28T17:21:11.519 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108486.log 2023-11-28T17:21:11.522 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99875.log 2023-11-28T17:21:11.523 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.108270.log.gz 2023-11-28T17:21:11.523 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108486.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108486.log.gz 2023-11-28T17:21:11.523 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96008.log 2023-11-28T17:21:11.523 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103536.log 2023-11-28T17:21:11.523 INFO:teuthology.orchestra.run.smithi116.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.99875.log.gz 2023-11-28T17:21:11.524 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102274.log 2023-11-28T17:21:11.525 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.96008.log.gz 2023-11-28T17:21:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103536.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.103536.log.gz 2023-11-28T17:21:11.525 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99917.log 2023-11-28T17:21:11.526 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99590.log 2023-11-28T17:21:11.527 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.102274.log.gz 2023-11-28T17:21:11.527 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95212.log 2023-11-28T17:21:11.527 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.99917.log.gz 2023-11-28T17:21:11.529 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94883.log 2023-11-28T17:21:11.530 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.99590.log.gz 2023-11-28T17:21:11.532 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104588.log 2023-11-28T17:21:11.533 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.95212.log.gz 2023-11-28T17:21:11.533 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94883.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.94883.log.gz 2023-11-28T17:21:11.533 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.96908.log 2023-11-28T17:21:11.533 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105506.log 2023-11-28T17:21:11.533 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104588.log.gz 2023-11-28T17:21:11.533 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.96908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102743.log 2023-11-28T17:21:11.534 INFO:teuthology.orchestra.run.smithi116.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.0.96908.log.gz 2023-11-28T17:21:11.534 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105506.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.105506.log.gz 2023-11-28T17:21:11.535 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100836.log 2023-11-28T17:21:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96344.log 2023-11-28T17:21:11.536 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.102743.log.gz 2023-11-28T17:21:11.536 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100836.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100836.log.gz 2023-11-28T17:21:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108574.log 2023-11-28T17:21:11.537 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105109.log 2023-11-28T17:21:11.538 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96344.log.gz 2023-11-28T17:21:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108574.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.108574.log.gz 2023-11-28T17:21:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95553.log 2023-11-28T17:21:11.539 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98318.log 2023-11-28T17:21:11.540 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105109.log.gz 2023-11-28T17:21:11.540 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94200.log 2023-11-28T17:21:11.541 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95553.log.gz 2023-11-28T17:21:11.542 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105716.log 89.0% 2023-11-28T17:21:11.542 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.98318.log.gz 2023-11-28T17:21:11.543 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98570.log 2023-11-28T17:21:11.543 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.94200.log.gz 2023-11-28T17:21:11.545 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105716.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.105716.log.gz 2023-11-28T17:21:11.545 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94885.log 2023-11-28T17:21:11.545 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105193.log 2023-11-28T17:21:11.546 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.98570.log.gz 2023-11-28T17:21:11.546 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94885.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.94885.log.gz 2023-11-28T17:21:11.547 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104672.log 2023-11-28T17:21:11.547 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105193.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-28T17:21:11.548 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105193.log.gz 2023-11-28T17:21:11.549 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101004.log 2023-11-28T17:21:11.549 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.104672.log.gz 2023-11-28T17:21:11.550 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94873.log 2023-11-28T17:21:11.551 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101004.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101004.log.gz 2023-11-28T17:21:11.551 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95498.log 2023-11-28T17:21:11.552 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94873.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94873.log.gz 2023-11-28T17:21:11.552 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105422.log 2023-11-28T17:21:11.553 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95498.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95498.log.gz 2023-11-28T17:21:11.554 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99473.log 2023-11-28T17:21:11.555 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105422.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105422.log.gz 2023-11-28T17:21:11.555 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97566.log 2023-11-28T17:21:11.557 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103620.log 2023-11-28T17:21:11.557 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.99473.log.gz 2023-11-28T17:21:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97566.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97566.log.gz 2023-11-28T17:21:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97772.log 2023-11-28T17:21:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101810.log 2023-11-28T17:21:11.559 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.103620.log.gz 2023-11-28T17:21:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97772.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97772.log.gz 2023-11-28T17:21:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99137.log 2023-11-28T17:21:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101810.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.101810.log.gz 2023-11-28T17:21:11.561 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.104504.log 2023-11-28T17:21:11.562 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99137.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99137.log.gz 2023-11-28T17:21:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101998.log 2023-11-28T17:21:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94156.log 2023-11-28T17:21:11.565 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.104504.log.gz 2023-11-28T17:21:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101998.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.101998.log.gz 2023-11-28T17:21:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105811.log 2023-11-28T17:21:11.567 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94156.log: gzip -5 --verbose 89.1% -- -- replaced with /var/log/ceph/ceph-client.admin.94156.log.gz /var/log/ceph/ceph-client.admin.95882.log 2023-11-28T17:21:11.567 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-11-28T17:21:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105811.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.105811.log.gz 2023-11-28T17:21:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106339.log 2023-11-28T17:21:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95882.log: gzip -5 --verbose 88.4% -- -- replaced with /var/log/ceph/ceph-client.admin.95882.log.gz /var/log/ceph/ceph-client.admin.99017.log 2023-11-28T17:21:11.569 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-11-28T17:21:11.570 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106339.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.106339.log.gz 2023-11-28T17:21:11.570 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101768.log 2023-11-28T17:21:11.571 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99017.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.99017.log.gz 2023-11-28T17:21:11.571 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105931.log 2023-11-28T17:21:11.572 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101768.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.101768.log.gz 2023-11-28T17:21:11.572 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95213.log 2023-11-28T17:21:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105674.log 2023-11-28T17:21:11.573 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105931.log.gz 2023-11-28T17:21:11.574 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95213.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95213.log.gz 2023-11-28T17:21:11.575 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106425.log 2023-11-28T17:21:11.576 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105674.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.105674.log.gz 2023-11-28T17:21:11.576 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95223.log 2023-11-28T17:21:11.577 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106425.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.106425.log.gz 2023-11-28T17:21:11.577 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107936.log 2023-11-28T17:21:11.579 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95223.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.95223.log.gz 2023-11-28T17:21:11.579 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94286.log 2023-11-28T17:21:11.581 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.107936.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.107936.log.gz 2023-11-28T17:21:11.581 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100962.log 2023-11-28T17:21:11.581 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94286.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.94286.log.gz 2023-11-28T17:21:11.582 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.98276.log 2023-11-28T17:21:11.582 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100962.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100962.log.gz 2023-11-28T17:21:11.583 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99179.log 2023-11-28T17:21:11.584 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98276.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.98276.log.gz 2023-11-28T17:21:11.584 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101208.log 2023-11-28T17:21:11.585 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99179.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.99179.log.gz 2023-11-28T17:21:11.585 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101376.log 2023-11-28T17:21:11.586 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98833.log 2023-11-28T17:21:11.586 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101208.log.gz 2023-11-28T17:21:11.587 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101376.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.101376.log.gz 2023-11-28T17:21:11.587 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102166.log 2023-11-28T17:21:11.589 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98833.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98833.log.gz 2023-11-28T17:21:11.589 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106057.log 2023-11-28T17:21:11.590 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102166.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.102166.log.gz 2023-11-28T17:21:11.590 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98528.log 2023-11-28T17:21:11.592 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100752.log 2023-11-28T17:21:11.592 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106057.log.gz 2023-11-28T17:21:11.592 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98528.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.98528.log.gz 2023-11-28T17:21:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105277.log 2023-11-28T17:21:11.594 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100752.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.100752.log.gz 2023-11-28T17:21:11.594 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94872.log 2023-11-28T17:21:11.595 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105277.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105277.log.gz 2023-11-28T17:21:11.595 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106719.log 2023-11-28T17:21:11.596 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108643.log 2023-11-28T17:21:11.596 INFO:teuthology.orchestra.run.smithi116.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94872.log.gz 2023-11-28T17:21:11.597 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106297.log 2023-11-28T17:21:11.597 INFO:teuthology.orchestra.run.smithi116.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.106719.log.gz 2023-11-28T17:21:11.598 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107293.log 2023-11-28T17:21:11.599 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108643.log.gz 2023-11-28T17:21:11.600 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106297.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.106297.log.gz 2023-11-28T17:21:11.600 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103746.log 2023-11-28T17:21:11.601 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.107293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95840.log 2023-11-28T17:21:11.601 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.107293.log.gz 2023-11-28T17:21:11.602 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103746.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.103746.log.gz 2023-11-28T17:21:11.602 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99649.log 2023-11-28T17:21:11.603 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106635.log 2023-11-28T17:21:11.603 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95840.log.gz 2023-11-28T17:21:11.604 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99649.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99649.log.gz 2023-11-28T17:21:11.605 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99221.log 2023-11-28T17:21:11.606 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106635.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.106635.log.gz 2023-11-28T17:21:11.606 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-11-28T17:21:11.607 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99221.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99221.log.gz 2023-11-28T17:21:11.607 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100710.log 2023-11-28T17:21:11.608 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101546.log 2023-11-28T17:21:11.610 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100710.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100710.log.gz 2023-11-28T17:21:11.610 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101718.log 2023-11-28T17:21:11.611 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101546.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101546.log.gz 2023-11-28T17:21:11.611 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100001.log 2023-11-28T17:21:11.613 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.101718.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.107378.log 2023-11-28T17:21:11.613 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100001.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101718.log.gz 2023-11-28T17:21:11.614 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.100001.log.gz 2023-11-28T17:21:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95756.log 2023-11-28T17:21:11.616 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.107378.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.107378.log.gz 2023-11-28T17:21:11.616 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104546.log 2023-11-28T17:21:11.617 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100584.log 2023-11-28T17:21:11.618 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104546.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95756.log.gz 2023-11-28T17:21:11.618 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104546.log.gz 2023-11-28T17:21:11.619 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.97420.log 2023-11-28T17:21:11.620 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100584.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100584.log.gz 2023-11-28T17:21:11.620 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98360.log 2023-11-28T17:21:11.622 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.2.97420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108186.log 2023-11-28T17:21:11.623 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98360.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.98360.log.gz 2023-11-28T17:21:11.624 INFO:teuthology.orchestra.run.smithi116.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.2.97420.log.gz 2023-11-28T17:21:11.624 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94706.log 2023-11-28T17:21:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108186.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108186.log.gz 2023-11-28T17:21:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88940.log 2023-11-28T17:21:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94706.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.94706.log.gz 2023-11-28T17:21:11.626 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101896.log 2023-11-28T17:21:11.628 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.88940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99691.log 2023-11-28T17:21:11.628 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.88940.log.gz 2023-11-28T17:21:11.629 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101896.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101896.log.gz 2023-11-28T17:21:11.629 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89138.log 2023-11-28T17:21:11.631 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106677.log 2023-11-28T17:21:11.631 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.99691.log.gz 2023-11-28T17:21:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.89138.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.89138.log.gz 2023-11-28T17:21:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95560.log 2023-11-28T17:21:11.633 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109292.log 2023-11-28T17:21:11.634 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.106677.log.gz 2023-11-28T17:21:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95560.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95560.log.gz 2023-11-28T17:21:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109102.log 2023-11-28T17:21:11.637 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.109292.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109292.log.gz 2023-11-28T17:21:11.637 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100121.log 2023-11-28T17:21:11.638 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.109102.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109102.log.gz 2023-11-28T17:21:11.638 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94030.log 2023-11-28T17:21:11.639 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98665.log 2023-11-28T17:21:11.640 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100121.log.gz 2023-11-28T17:21:11.641 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94030.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.94030.log.gz 2023-11-28T17:21:11.641 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94622.log 2023-11-28T17:21:11.642 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94886.log 2023-11-28T17:21:11.643 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.98665.log.gz 2023-11-28T17:21:11.643 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94622.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.94622.log.gz 2023-11-28T17:21:11.644 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108998.log 2023-11-28T17:21:11.644 INFO:teuthology.orchestra.run.smithi156.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-11-28T17:21:11.645 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95219.log 2023-11-28T17:21:11.645 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.94886.log.gz 2023-11-28T17:21:11.647 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.104085.log 2023-11-28T17:21:11.647 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108998.log.gz 2023-11-28T17:21:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95219.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95219.log.gz 2023-11-28T17:21:11.648 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96260.log 2023-11-28T17:21:11.649 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.104085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103164.log 2023-11-28T17:21:11.650 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96260.log: 88.8% -- replaced with /var/log/ceph/ceph-client.0.104085.log.gz 2023-11-28T17:21:11.651 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.96260.log.gz 2023-11-28T17:21:11.651 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98486.log 2023-11-28T17:21:11.652 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100289.log 2023-11-28T17:21:11.653 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.103164.log.gz 2023-11-28T17:21:11.654 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98486.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.98486.log.gz 2023-11-28T17:21:11.654 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108859.log 2023-11-28T17:21:11.655 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106015.log 2023-11-28T17:21:11.656 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.100289.log.gz 2023-11-28T17:21:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108859.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108859.log.gz 2023-11-28T17:21:11.657 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98234.log 2023-11-28T17:21:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106015.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.106015.log.gz 2023-11-28T17:21:11.658 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102896.log 2023-11-28T17:21:11.659 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106381.log 2023-11-28T17:21:11.660 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98234.log.gz 2023-11-28T17:21:11.661 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102896.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.102896.log.gz 2023-11-28T17:21:11.661 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-11-28T17:21:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106381.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.106381.log.gz 2023-11-28T17:21:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101674.log 2023-11-28T17:21:11.666 INFO:teuthology.orchestra.run.smithi156.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-11-28T17:21:11.667 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109603.log 2023-11-28T17:21:11.667 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103662.log 2023-11-28T17:21:11.667 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101674.log.gz 2023-11-28T17:21:11.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.109603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95205.log 2023-11-28T17:21:11.669 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105590.log 2023-11-28T17:21:11.669 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103662.log.gz 2023-11-28T17:21:11.670 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95205.log: 95.8% -- replaced with /var/log/ceph/ceph-client.admin.109603.log.gz 2023-11-28T17:21:11.670 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105464.log 2023-11-28T17:21:11.670 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95205.log.gz 2023-11-28T17:21:11.672 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98150.log 2023-11-28T17:21:11.672 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105590.log.gz 2023-11-28T17:21:11.673 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105025.log 2023-11-28T17:21:11.674 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105464.log.gz 2023-11-28T17:21:11.675 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108250.log 2023-11-28T17:21:11.675 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.98150.log.gz 2023-11-28T17:21:11.676 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105025.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100878.log 2023-11-28T17:21:11.677 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.105025.log.gz 2023-11-28T17:21:11.678 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98891.log 2023-11-28T17:21:11.678 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.108250.log.gz 2023-11-28T17:21:11.680 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.102559.log 2023-11-28T17:21:11.680 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100878.log.gz 2023-11-28T17:21:11.680 INFO:teuthology.orchestra.run.smithi156.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-11-28T17:21:11.681 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93768.log 2023-11-28T17:21:11.681 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.98891.log.gz 2023-11-28T17:21:11.682 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.0.102559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94888.log 2023-11-28T17:21:11.683 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93768.log: 89.5% -- replaced with /var/log/ceph/ceph-client.0.102559.log.gz 2023-11-28T17:21:11.684 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101292.log 2023-11-28T17:21:11.684 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.93768.log.gz 2023-11-28T17:21:11.685 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98707.log 2023-11-28T17:21:11.685 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.94888.log.gz 2023-11-28T17:21:11.687 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100920.log 2023-11-28T17:21:11.687 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101292.log.gz 2023-11-28T17:21:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98707.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.98707.log.gz 2023-11-28T17:21:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108490.log 2023-11-28T17:21:11.689 INFO:teuthology.orchestra.run.smithi156.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-11-28T17:21:11.690 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97814.log 2023-11-28T17:21:11.690 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100920.log.gz 2023-11-28T17:21:11.692 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104446.log 2023-11-28T17:21:11.692 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108490.log.gz 2023-11-28T17:21:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103122.log 2023-11-28T17:21:11.693 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97814.log.gz 2023-11-28T17:21:11.694 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104446.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-28T17:21:11.694 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.104446.log.gz 2023-11-28T17:21:11.696 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100079.log 2023-11-28T17:21:11.696 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.103122.log.gz 2023-11-28T17:21:11.697 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105235.log 2023-11-28T17:21:11.699 INFO:teuthology.orchestra.run.smithi156.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-11-28T17:21:11.699 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106247.log 2023-11-28T17:21:11.699 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.100079.log.gz 2023-11-28T17:21:11.700 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105548.log 2023-11-28T17:21:11.701 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.105235.log.gz 2023-11-28T17:21:11.702 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100205.log 2023-11-28T17:21:11.702 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.106247.log.gz 2023-11-28T17:21:11.704 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99515.log 2023-11-28T17:21:11.704 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.105548.log.gz 2023-11-28T17:21:11.704 INFO:teuthology.orchestra.run.smithi156.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-11-28T17:21:11.705 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108028.log 2023-11-28T17:21:11.705 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100205.log.gz 2023-11-28T17:21:11.707 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103326.log 2023-11-28T17:21:11.707 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99515.log.gz 2023-11-28T17:21:11.708 INFO:teuthology.orchestra.run.smithi156.stderr: 2023-11-28T17:21:11.708 INFO:teuthology.orchestra.run.smithi156.stderr:real 0m0.346s 2023-11-28T17:21:11.708 INFO:teuthology.orchestra.run.smithi156.stderr:user 0m1.515s 2023-11-28T17:21:11.708 INFO:teuthology.orchestra.run.smithi156.stderr:sys 0m0.256s 2023-11-28T17:21:11.709 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97898.log 2023-11-28T17:21:11.709 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108028.log.gz 2023-11-28T17:21:11.710 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95966.log 2023-11-28T17:21:11.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.103326.log.gz 2023-11-28T17:21:11.711 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.97898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98024.log 2023-11-28T17:21:11.712 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.97898.log.gz 2023-11-28T17:21:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95966.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95966.log.gz 2023-11-28T17:21:11.713 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94496.log 2023-11-28T17:21:11.714 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98024.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98024.log.gz 2023-11-28T17:21:11.715 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101250.log 2023-11-28T17:21:11.716 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106509.log 2023-11-28T17:21:11.716 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.94496.log.gz 2023-11-28T17:21:11.718 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106467.log 2023-11-28T17:21:11.718 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.101250.log.gz 2023-11-28T17:21:11.719 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106509.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-28T17:21:11.719 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.106509.log.gz 2023-11-28T17:21:11.721 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104404.log 2023-11-28T17:21:11.721 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.106467.log.gz 2023-11-28T17:21:11.722 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101956.log 2023-11-28T17:21:11.724 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104404.log: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-28T17:21:11.724 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104404.log.gz 2023-11-28T17:21:11.725 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101956.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101956.log.gz 2023-11-28T17:21:11.726 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99263.log 2023-11-28T17:21:11.727 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102938.log 2023-11-28T17:21:11.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101334.log 2023-11-28T17:21:11.729 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.99263.log.gz 2023-11-28T17:21:11.730 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109167.log 2023-11-28T17:21:11.731 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.102938.log.gz 2023-11-28T17:21:11.732 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101334.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.101334.log.gz 2023-11-28T17:21:11.732 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-28T17:21:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.109167.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109167.log.gz 2023-11-28T17:21:11.734 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94881.log 2023-11-28T17:21:11.735 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108081.log 2023-11-28T17:21:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94881.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.94881.log.gz 2023-11-28T17:21:11.737 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94114.log 2023-11-28T17:21:11.738 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108081.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.108081.log.gz 2023-11-28T17:21:11.739 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95558.log 2023-11-28T17:21:11.740 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94114.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.94114.log.gz 2023-11-28T17:21:11.741 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-11-28T17:21:11.744 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.95558.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95558.log.gz 2023-11-28T17:21:11.744 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108554.log 2023-11-28T17:21:11.746 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.108554.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.108554.log.gz 2023-11-28T17:21:11.747 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105337.log 2023-11-28T17:21:11.750 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108334.log 2023-11-28T17:21:11.751 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105337.log.gz 2023-11-28T17:21:11.753 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98444.log 2023-11-28T17:21:11.756 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108334.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108334.log.gz 2023-11-28T17:21:11.758 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100794.log 2023-11-28T17:21:11.760 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% 91.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-11-28T17:21:11.760 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.98444.log.gz 2023-11-28T17:21:11.761 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99833.log 2023-11-28T17:21:11.762 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100794.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100794.log.gz 2023-11-28T17:21:11.763 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102796.log 2023-11-28T17:21:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99833.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.99833.log.gz 2023-11-28T17:21:11.767 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102796.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.102796.log.gz 2023-11-28T17:21:11.767 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98791.log 2023-11-28T17:21:11.769 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108103.log 2023-11-28T17:21:11.770 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98791.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.98791.log.gz 2023-11-28T17:21:11.771 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108103.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108103.log.gz 2023-11-28T17:21:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100668.log 2023-11-28T17:21:11.774 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104941.log 2023-11-28T17:21:11.775 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100668.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100668.log.gz 2023-11-28T17:21:11.777 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104941.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.104941.log.gz 2023-11-28T17:21:11.778 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98066.log 2023-11-28T17:21:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99431.log 2023-11-28T17:21:11.780 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.98066.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98066.log.gz 2023-11-28T17:21:11.782 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99431.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99431.log.gz 2023-11-28T17:21:11.783 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93864.log 2023-11-28T17:21:11.785 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109018.log 2023-11-28T17:21:11.785 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.93864.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.93864.log.gz 2023-11-28T17:21:11.787 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.109018.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109018.log.gz 2023-11-28T17:21:11.788 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104630.log 2023-11-28T17:21:11.789 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106551.log 2023-11-28T17:21:11.791 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.104630.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.104630.log.gz 2023-11-28T17:21:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106551.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.106551.log.gz 2023-11-28T17:21:11.793 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108742.log 2023-11-28T17:21:11.795 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99775.log 2023-11-28T17:21:11.796 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.108742.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108742.log.gz 2023-11-28T17:21:11.798 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99775.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.99775.log.gz 2023-11-28T17:21:11.798 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96092.log 2023-11-28T17:21:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94072.log 2023-11-28T17:21:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96092.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96092.log.gz 2023-11-28T17:21:11.803 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.94072.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.94072.log.gz 2023-11-28T17:21:11.803 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105151.log 2023-11-28T17:21:11.805 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100163.log 2023-11-28T17:21:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.105151.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.105151.log.gz 2023-11-28T17:21:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100163.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100163.log.gz 2023-11-28T17:21:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103578.log 2023-11-28T17:21:11.810 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-11-28T17:21:11.812 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103578.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.103578.log.gz 2023-11-28T17:21:11.815 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99959.log 2023-11-28T17:21:11.818 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100626.log 2023-11-28T17:21:11.818 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.99959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106185.log 2023-11-28T17:21:11.818 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.99959.log.gz 2023-11-28T17:21:11.820 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.100626.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100626.log.gz 2023-11-28T17:21:11.820 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101854.log 2023-11-28T17:21:11.821 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.106185.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.106185.log.gz 2023-11-28T17:21:11.824 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.101854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102082.log 2023-11-28T17:21:11.824 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.101854.log.gz 2023-11-28T17:21:11.826 INFO:teuthology.orchestra.run.smithi116.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-11-28T17:21:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103808.log 2023-11-28T17:21:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.102082.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.102082.log.gz 2023-11-28T17:21:11.830 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96218.log 2023-11-28T17:21:11.831 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.103808.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103808.log.gz 2023-11-28T17:21:11.833 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.96218.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.96218.log.gz 2023-11-28T17:21:11.834 INFO:teuthology.orchestra.run.smithi116.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-11-28T17:21:11.846 INFO:teuthology.orchestra.run.smithi116.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-28T17:21:11.873 INFO:teuthology.orchestra.run.smithi116.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-28T17:21:11.945 INFO:teuthology.orchestra.run.smithi116.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-11-28T17:21:11.997 INFO:teuthology.orchestra.run.smithi116.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-28T17:21:12.006 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-28T17:21:12.009 INFO:teuthology.orchestra.run.smithi116.stderr: 2023-11-28T17:21:12.010 INFO:teuthology.orchestra.run.smithi116.stderr:real 0m0.654s 2023-11-28T17:21:12.010 INFO:teuthology.orchestra.run.smithi116.stderr:user 0m1.866s 2023-11-28T17:21:12.010 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m0.983s 2023-11-28T17:21:12.011 INFO:tasks.ceph:Archiving logs... 2023-11-28T17:21:12.011 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360/remote/smithi116/log 2023-11-28T17:21:12.013 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-28T17:21:12.573 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360/remote/smithi156/log 2023-11-28T17:21:12.575 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-28T17:21:12.888 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-28T17:21:12.902 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-28T17:21:12.917 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-28T17:21:12.924 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-28T17:21:12.924 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-28T17:21:12.939 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-28T17:21:13.096 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-28T17:21:13.110 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-28T17:21:13.110 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T17:21:13.120 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T17:21:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2023-11-28T17:21:13.178 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2023-11-28T17:21:13.179 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T17:21:13.179 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2023-11-28T17:21:13.179 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +778us[ +778us] +/- 88ms 2023-11-28T17:21:13.179 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 6 377 10 -510us[ -498us] +/- 70ms 2023-11-28T17:21:13.180 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 -109us[ -98us] +/- 87ms 2023-11-28T17:21:13.180 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:21:13.188 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T17:21:13.188 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2023-11-28T17:21:13.189 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 10 +784us[ +784us] +/- 88ms 2023-11-28T17:21:13.189 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 -476us[ -472us] +/- 70ms 2023-11-28T17:21:13.189 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 -109us[ -105us] +/- 87ms 2023-11-28T17:21:13.189 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T17:21:13.190 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-28T17:21:13.200 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-28T17:21:13.201 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-28T17:21:13.210 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-28T17:21:13.480 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-28T17:21:13.736 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T17:21:13.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:21:13.849 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T17:21:13.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:21:13.959 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-28T17:21:13.969 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-28T17:21:13.979 INFO:teuthology.task.internal:Duration was 1522.646788 seconds 2023-11-28T17:21:13.979 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-28T17:21:13.990 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-28T17:21:13.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-28T17:21:13.999 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-28T17:21:14.157 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T17:21:14.171 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T17:21:14.561 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-28T17:21:14.562 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2023-11-28T17:21:14.562 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-28T17:21:14.635 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2023-11-28T17:21:14.635 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-28T17:21:14.714 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-28T17:21:14.715 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T17:21:14.723 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T17:21:14.817 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-28T17:21:14.817 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T17:21:14.824 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T17:21:15.010 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-28T17:21:15.021 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-28T17:21:15.022 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-28T17:21:15.116 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-28T17:21:15.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-28T17:21:15.223 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-28T17:21:15.232 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-28T17:21:15.339 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2023-11-28T17:21:15.344 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2023-11-28T17:21:15.419 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-28T17:21:15.470 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi116.front.sepia.ceph.com, flagging run as failed 2023-11-28T17:21:15.470 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-28T17:21:15.529 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi156.front.sepia.ceph.com, flagging run as failed 2023-11-28T17:21:15.530 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-28T17:21:15.541 INFO:teuthology.task.internal:Transferring archived files... 2023-11-28T17:21:15.542 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360/remote/smithi116 2023-11-28T17:21:15.542 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-28T17:21:23.052 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2023-11-28T17:21:23.062 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360/remote/smithi116/coredump/1701191508.105872.core.gz: gzip compressed data, was "1701191508.105872.core", last modified: Tue Nov 28 17:21:15 2023, max compression, original size modulo 2^32 234344448 2023-11-28T17:21:23.063 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2023-11-28T17:21:24.014 INFO:teuthology.task.internal: core looks like: /tmp/tmpn8cso_4p: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i a', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/bin/ceph-mds', platform: 'x86_64' 2023-11-28T17:21:24.039 INFO:teuthology.task.internal: dump_program: ceph-mds -f --cluster ceph -i a 2023-11-28T17:21:24.039 DEBUG:teuthology.orchestra.run.smithi116:> which 'ceph-mds -f --cluster ceph -i a' 2023-11-28T17:21:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/which: invalid option -- 'f' 2023-11-28T17:21:24.100 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/which: unrecognized option '--cluster' 2023-11-28T17:21:24.100 INFO:teuthology.orchestra.run.smithi116.stderr:/usr/bin/which: no a in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2023-11-28T17:21:24.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:21:24.102 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-mds 2023-11-28T17:21:24.102 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph 2023-11-28T17:21:24.103 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: "which 'ceph-mds -f --cluster ceph -i a'" 2023-11-28T17:21:24.123 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-28T17:21:24.136 INFO:teuthology.task.internal:Not uploading archives. 2023-11-28T17:21:24.136 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-28T17:21:24.156 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-28T17:21:24.157 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-28T17:21:24.175 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-28T17:21:24.225 INFO:teuthology.orchestra.run.smithi116.stdout: 524839 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 28 17:21 /home/ubuntu/cephtest 2023-11-28T17:21:24.225 INFO:teuthology.orchestra.run.smithi116.stdout: 524855 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Nov 28 17:21 /home/ubuntu/cephtest/archive 2023-11-28T17:21:24.225 INFO:teuthology.orchestra.run.smithi116.stdout: 524894 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 17:21 /home/ubuntu/cephtest/archive/syslog 2023-11-28T17:21:24.225 INFO:teuthology.orchestra.run.smithi116.stdout: 524413 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Nov 28 16:55 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-11-28T17:21:24.226 INFO:teuthology.orchestra.run.smithi116.stdout: 524421 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Nov 28 16:55 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-11-28T17:21:24.226 INFO:teuthology.orchestra.run.smithi116.stdout: 524422 116 -rw-r--r-- 1 ubuntu ubuntu 118259 Nov 28 17:21 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T17:21:24.226 INFO:teuthology.orchestra.run.smithi116.stdout: 656819 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 17:21 /home/ubuntu/cephtest/archive/audit 2023-11-28T17:21:24.226 INFO:teuthology.orchestra.run.smithi116.stdout: 657098 284 -rw------- 1 ubuntu ubuntu 290771 Nov 28 17:21 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-11-28T17:21:24.227 INFO:teuthology.orchestra.run.smithi116.stdout: 524869 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 17:11 /home/ubuntu/cephtest/archive/coredump 2023-11-28T17:21:24.227 INFO:teuthology.orchestra.run.smithi116.stdout: 526237 21924 -rw------- 1 root root 234344448 Nov 28 17:11 /home/ubuntu/cephtest/archive/coredump/1701191508.105872.core 2023-11-28T17:21:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T17:21:24.228 INFO:teuthology.orchestra.run.smithi116.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-28T17:21:24.229 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_47c1c547372ed5bff7984647dc200bac0e019caf/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-11-28T17:21:24.243 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-28T17:21:24.294 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-28T17:21:24.306 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi116 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-11-28T17:21:24.306 INFO:teuthology.nuke:Checking targets against current locks 2023-11-28T17:21:24.314 INFO:teuthology.orchestra.run.smithi156.stdout: 393907 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Nov 28 17:21 /home/ubuntu/cephtest 2023-11-28T17:21:24.314 INFO:teuthology.orchestra.run.smithi156.stdout: 393922 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Nov 28 17:21 /home/ubuntu/cephtest/archive 2023-11-28T17:21:24.314 INFO:teuthology.orchestra.run.smithi156.stdout: 393938 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 17:21 /home/ubuntu/cephtest/archive/syslog 2023-11-28T17:21:24.314 INFO:teuthology.orchestra.run.smithi156.stdout: 393341 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Nov 28 16:55 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-11-28T17:21:24.315 INFO:teuthology.orchestra.run.smithi156.stdout: 393348 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Nov 28 16:55 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-11-28T17:21:24.315 INFO:teuthology.orchestra.run.smithi156.stdout: 393349 112 -rw-r--r-- 1 ubuntu ubuntu 110641 Nov 28 17:21 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T17:21:24.315 INFO:teuthology.orchestra.run.smithi156.stdout: 525747 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 17:21 /home/ubuntu/cephtest/archive/audit 2023-11-28T17:21:24.315 INFO:teuthology.orchestra.run.smithi156.stdout: 526026 256 -rw------- 1 ubuntu ubuntu 259740 Nov 28 17:21 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-11-28T17:21:24.315 INFO:teuthology.orchestra.run.smithi156.stdout: 393928 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 17:12 /home/ubuntu/cephtest/archive/coredump 2023-11-28T17:21:24.315 INFO:teuthology.orchestra.run.smithi156.stdout: 393371 20856 -rw------- 1 root root 250941440 Nov 28 17:12 /home/ubuntu/cephtest/archive/coredump/1701191550.94798.core 2023-11-28T17:21:24.315 INFO:teuthology.orchestra.run.smithi156.stdout: 393370 20644 -rw------- 1 root root 250867712 Nov 28 17:12 /home/ubuntu/cephtest/archive/coredump/1701191530.94759.core 2023-11-28T17:21:24.316 INFO:teuthology.orchestra.run.smithi156.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-11-28T17:21:24.339 DEBUG:teuthology.nuke:shortname: smithi116 2023-11-28T17:21:24.340 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T17:21:24.366 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 16:46:47.808555', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAd/T2uVnzOHp9OpssfoR7Mv6f0LDVHPMou5adf1YT1rkfXwdtt2VxAB+RNPeCVBdlQuqDkrd4w7MPavNdXVc+w='} 2023-11-28T17:21:24.367 DEBUG:teuthology.nuke:shortname: smithi156 2023-11-28T17:21:24.368 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T17:21:24.393 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-28_06:23:42-fs-wip-vshankar-testing-20231127.102654-testing-default-smithi/7469360', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 16:46:47.806967', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHTMAA4YyLkkHqT3TQSZosSRhTfgBHoAWQmyxP5/lAyfF2GZ1OX97dufJBP9jzaMDtkuTglviynajI16ajnPc+k='} 2023-11-28T17:21:24.419 INFO:teuthology.orchestra.console.smithi116:Power off 2023-11-28T17:21:24.419 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-28T17:21:24.443 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2023-11-28T17:21:24.443 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T17:21:24.463 INFO:teuthology.orchestra.console.smithi156:Power off 2023-11-28T17:21:24.463 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-28T17:21:24.475 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2023-11-28T17:21:24.488 DEBUG:teuthology.orchestra.console.smithi156:power off output: Chassis Power Control: Down/Off 2023-11-28T17:21:24.488 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T17:21:24.513 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2023-11-28T17:21:28.476 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T17:21:28.515 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T17:21:28.592 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2023-11-28T17:21:28.627 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2023-11-28T17:21:32.597 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T17:21:32.628 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T17:21:32.709 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2023-11-28T17:21:32.739 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off 2023-11-28T17:21:32.811 INFO:teuthology.orchestra.console.smithi116:Power off completed 2023-11-28T17:21:32.841 INFO:teuthology.orchestra.console.smithi156:Power off completed 2023-11-28T17:21:32.946 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} duration: 1522.6467881202698 failure_reason: 'Test failure: test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap), test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap)' flavor: default mon.a-kernel-sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 mon.b-kernel-sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=22f534bcf0f146c795677ca44fc2d08b status: fail success: false 2023-11-28T17:21:32.946 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T17:21:33.024 INFO:teuthology.run:FAIL